INTERACT FORUM

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1]   Go Down

Author Topic: Ubuntu auto restart mediacenter on crash using cron  (Read 9976 times)

aoqw76

  • Galactic Citizen
  • ****
  • Posts: 257
Ubuntu auto restart mediacenter on crash using cron
« on: October 01, 2014, 03:30:49 pm »

Hi, so I've been fiddling for a while on and off to get this to work, and finally success.
I'm not great on linux so it's taken me a bit of googling to get this working.
Basically I have MC20 running in Ubuntu, with a couple of Windows computers running MC as clients, plus JRemote on an ipad.
Every now and then MC crashes on linux, and I have to manually restart it. Sometimes it's bothersome (when I'm at work and have to explain to the wife how to do it!) and I just want it to auto restart.

So I created the following script
Code: [Select]
log="/home/robert/scripts/checkmc20.log"
checkmc=`ls -ltr /var/crash/*mediacenter20* | wc -l`
if [ $checkmc -gt 0 ]
then
pkill apport
pkill mediacenter
now=$(date +%y%m%d%H%M)
cd /var/crash
for file in *mediacenter20*
do
  mv "$file" "$file.$now"
done
mv *mediacenter20* /home/robert/Documents
sudo restart apport >> $log
echo "restarted mc20 $(date)" >> $log
##sudo -u root /usr/bin/mediacenter20 --revised per later post
/usr/bin/mediacenter20
fi

then set up cron using command "crontab -e"
Code: [Select]
*/5 * * * * export DISPLAY=:0 && sudo /home/robert/scripts/checkmc20
in other words, run "checkmc20" every 5 minutes. the "export" bit is needed to allow the script to launch an application with a gui window.
(see http://www.jveweb.net/en/archives/2011/01/using-cronjobs-in-linux.html particularly the "launching a graphical program with cron" bit)

then, because the script contains "sudo" and i wanted it to run without needing a password, I added it to "sudoers"
see http://luiseth.wordpress.com/2012/04/15/in-a-nutshell-add-permissions-with-configuration-files-in-etcsudoers-d/
I created "checkmc20_sudo" in my home directory (apparently the file name must not contain a ~ or a .)
edit it and add something like the following
Code: [Select]
robert ALL=NOPASSWD: /home/robert/scripts/checkmc20In other words, allow this script to run without a password prompt
then chmod the file to 440 and chown it with root:root
then sudo mv to /etc/sudoers.d directory

Now if MC20 crashes, within 5 minutes the script will run and discover the crash log file within /var/crash
it will kill apport and mediacenter
it will append the date and time to the crash log file name, then move it into the documents folder (so it is available if needed)
then apport will be restarted - this is the crash report app in ubuntu, and mediacenter20 will be restarted. (and some logging along the way)

so far it seems to work beautifully.
I guess the principle can be applied to other distros with relative ease.
The stumbling blocks for me were working out what the pop up crash window was - it's called "apport", so I could add "pkill apport" to the script
and adding the "export" bit into crontab took a bit of googling.

hope this helps someone.
regards, Robert
Logged
xubuntu lts 14.04 32 bit, running mc22.0.36 as anything later doesn't work properly over vnc. using linux mc22 as media server to windows mc22 last version / jremote on ipad.
I am the owner / sole admin for www.cyrusunofficial.co.uk ("fan" site for Cyrus Audio hifi)

astromo

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 2239
Re: Ubuntu auto restart mediacenter on crash using cron
« Reply #1 on: October 01, 2014, 06:41:36 pm »

Neat idea. I like clever stuff like this.

There's a flip side question though, how are you feeding back the crash log data the devs? I'd hope the your experience is the exception not rule, so that the long your script not needed.

Logged
MC31, Win10 x64, HD-Plex H5 Gen2 Case, HD-Plex 400W Hi-Fi DC-ATX / AC-DC PSU, Gigabyte Z370 ULTRA Gaming 2.0 MoBo, Intel Core i7 8700 CPU, 4x8GB GSkill DDR4 RAM, Schiit Modi Multibit DAC, Freya Pre, Nelson Pass Aleph J DIY Clone, Ascension Timberwolf 8893BSRTL Speakers, BJC 5T00UP cables, DVB-T Tuner HDHR5-4DT

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5175
  • "Linux Merit Badge" Recipient
Re: Ubuntu auto restart mediacenter on crash using cron
« Reply #2 on: October 01, 2014, 06:54:42 pm »

I actually developed a similar script for different purposes here: http://yabb.jriver.com/interact/index.php?topic=91227.msg627239#msg627239

I was puzzling over why your script was so much more complex than mine, and then I realized that you must actually be getting a crash window in Ubuntu.  When MC for linux crashes for me in Gnome or XFCE on Arch, I just get a straight crash to desktop and mediacenter terminates itself; no explanatory window. There's usually a coredump and a trace in the journal of course, but that requires no effort on my part to log (systemd does it automagically).  

So my version of your script basically just checks if mediacenter is running, and if not, it exports the display, and restarts it.  I get a few crashes a week and it's handy not to have to manually intervene.

I look forward to needing to kill the crash popup window one day  ;D

P.S.- Is there some reason you're running MC as root?  I've never needed to, but maybe I'm missing out on something?
Logged

aoqw76

  • Galactic Citizen
  • ****
  • Posts: 257
Re: Ubuntu auto restart mediacenter on crash using cron
« Reply #3 on: October 02, 2014, 04:44:39 am »

Hi,

yes Ubuntu has a popup asking if you want to send a crash log back to .... whoever, I dont know where it goes? The couple of times I tried this it failed to send. It took a while to uncover that this was "apport", and that's how I ended up checking for the presence of a log file in /var/crash rather than something using "ps" and "grep" to check for an active mediacenter. That doesnt seem to work as mediacenter appears to be still active. (hence the need for pkill apport and pkill mediacenter)

ref running as root ("sudo -u root /usr/bin/mediacenter20") for some reason which slightly baffles me:
just putting "/usr/bin/mediacenter20" didnt do anything (actually I might need to backtrack and check it's not because I hadn't solved the need for "export DISPLAY=:0" in the crontab)
doing "sudo -u robert ..." launches MC successfully, but then tells me the library is read only, so I changed to "-u root" and it works as expected.
Though perversely, if I just do "nohup /usr/bin/mediacenter20 &" in a terminal, it runs and I can ctrl C to exit.

I guess there's the potential to auto send the crash log to JRiver, via ftp? I think mostly the crashes seem to related to playback issues with jremote; I get a stream error and then MC has crashed. Or maybe MC has crashed which manifests itself as a stream error in jremote.
Logged
xubuntu lts 14.04 32 bit, running mc22.0.36 as anything later doesn't work properly over vnc. using linux mc22 as media server to windows mc22 last version / jremote on ipad.
I am the owner / sole admin for www.cyrusunofficial.co.uk ("fan" site for Cyrus Audio hifi)

mwillems

  • MC Beta Team
  • Citizen of the Universe
  • *****
  • Posts: 5175
  • "Linux Merit Badge" Recipient
Re: Ubuntu auto restart mediacenter on crash using cron
« Reply #4 on: October 02, 2014, 10:20:11 am »

ref running as root ("sudo -u root /usr/bin/mediacenter20") for some reason which slightly baffles me:
just putting "/usr/bin/mediacenter20" didnt do anything (actually I might need to backtrack and check it's not because I hadn't solved the need for "export DISPLAY=:0" in the crontab)

The DISPLAY issue is probably what caused that issue at that point, when I've tried to run MC without any DISPLAY it just quietly segfaults.

Quote
doing "sudo -u robert ..." launches MC successfully, but then tells me the library is read only, so I changed to "-u root" and it works as expected.
Though perversely, if I just do "nohup /usr/bin/mediacenter20 &" in a terminal, it runs and I can ctrl C to exit.

That is bizarre; it only normally displays that error message when you launch an instance of mediacenter and one is already running somewhere; are you sure that wasn't the case when you tested?  Otherwise it might just be a permissions issue (your mc library may be owned by root because you've been running it as root?).  Generally, you shouldn't need to run it as root.

Also, you may already know, but if you need to do scripting that requires root permissions in the future, you can just run scripts from root's crontab rather than using sudo throughout.  This works even in distros where root login is disabled by default.  
Logged

aoqw76

  • Galactic Citizen
  • ****
  • Posts: 257
Re: Ubuntu auto restart mediacenter on crash using cron
« Reply #5 on: October 02, 2014, 12:55:44 pm »

Hi, removed the "sudo -u root" bit and still works correctly, so yes must have been in my attempts to get it to work prior to adding the "export display" bit.
My library files were at some point all owned by root - guess it was something I did when installing MC20? Once I sorted that and chowned the lot to me:me it was fine, but prior to that I would get a "read only database" issue.
With sudo -u robert it would open with a read only db, but not when substituting with root.
Logged
xubuntu lts 14.04 32 bit, running mc22.0.36 as anything later doesn't work properly over vnc. using linux mc22 as media server to windows mc22 last version / jremote on ipad.
I am the owner / sole admin for www.cyrusunofficial.co.uk ("fan" site for Cyrus Audio hifi)

aoqw76

  • Galactic Citizen
  • ****
  • Posts: 257
Re: Ubuntu auto restart mediacenter on crash using cron
« Reply #6 on: December 18, 2015, 06:01:39 am »

My current version of this script:
the "lok" file and the "restart" file logic is there as i have the script set up to run every 2 mins in cron, and on occasion the execution time ran longer than 2 mins. hence the script checks for the presence of "lok" file (script still running) and "restart" file and if either exists, then it simply exits 0. not particularly neat but it works.
i also have ssmtp set up so i get an email to notify me if the script restarts mc21, you can chop all that out if you dont want it. (replace "email@address" with your own valid one)
Code: [Select]
log="/home/robert/scripts/logs/checkmc21.log"
zzip="/home/robert/Documents/mc21crash.zip"
mc21email="/home/robert/scripts/mc21.email"
mc21lok="/home/robert/scripts/mc21.lok"
mc21restart="/home/robert/scripts/mc21.restart"
# if lok file exists then script is still running from previous cron start
if  [ -f "$mc21lok" ]
then
  exit 0
fi
# if restart file exists then mc21 is already on its way up again in which case just quit
# restart file can also be removed
if [ -f "$mc21restart" ]
then
  rm $mc21restart
  exit 0
fi
touch $mc21lok
touch /var/crash/dummy_file
# touching a dummy file means that the "ls" command at least finds something and doesnt report an error via cron
checkmc=`ls -ltr /var/crash/* | grep mediacenter21 | wc -l`
checkmcv2=`ps -aux | grep -v grep | grep mediacenter | wc -l`
now=$(date +%y%m%d%H%M)
# next section: if a crash file was found, then kill apport, kill mc, backup the crash log file to a zip file, send via email, and restart apport
if [ $checkmc -gt 0 ]
then
pkill apport
pkill mediacenter
cd /var/crash
for file in *mediacenter21*
do
  chown robert:robert "$file"
  mv "$file" "$file.$now"
done
mv *mediacenter21* /home/robert/Documents
rm $zzip
zip /home/robert/Documents/mc21crash /home/robert/Documents/$file.$now
echo "$(date) mc21 restarted, crash log attached." > $mc21email
dmesg | grep mediacenter | tail -1 >> $mc21email
cat $mc21email | mutt -a "$zzip" -s "mc21 crashed and has been restarted" -- email@address
sudo restart apport >> $log
fi
# next section, if a crash log file wasnt found, and mc21 simply wasnt active, send an email
if [ $checkmcv2 -eq 0 ] && [ $checkmc -eq 0 ]
then
echo "To: email@address" > $mc21email
echo "From: email@address" >> $mc21email
echo "Subject: mc21 restarted" >> $mc21email
echo "\n" >> $mc21email
echo "$(date) restarted mc21" >> $mc21email
dmesg | grep mediacenter | tail -1 >> $mc21email
ssmtp email@address < $mc21email
fi
# next section, restart mc21
if [ $checkmc -eq 0 ] | [ $checkmcv2 -eq 0 ]
then
echo "restarted mc21 $(date)" >> $log
rm $mc21lok
touch $mc21restart
/usr/bin/mediacenter21
fi
rm $mc21lok

bare minimum version, no logging, no email, no bells / whistles, no backup of crash log file. simply restart if not running or crashed.
Code: [Select]
touch /var/crash/dummy_file
checkmc=`ls -ltr /var/crash/* | grep mediacenter21 | wc -l`
checkmcv2=`ps -aux | grep -v grep | grep mediacenter | wc -l`
now=$(date +%y%m%d%H%M)
if [ $checkmc -gt 0 ]
then
  pkill apport
  pkill mediacenter
  cd /var/crash
  for file in *mediacenter21*
  do
    rm "$file"
  done
fi
if [ $checkmc -eq 0 ] | [ $checkmcv2 -eq 0 ]
then
  /usr/bin/mediacenter21
fi
Logged
xubuntu lts 14.04 32 bit, running mc22.0.36 as anything later doesn't work properly over vnc. using linux mc22 as media server to windows mc22 last version / jremote on ipad.
I am the owner / sole admin for www.cyrusunofficial.co.uk ("fan" site for Cyrus Audio hifi)
Pages: [1]   Go Up