Supervisor is a great python application that will make sure that all your long running programs on the server stay running. I was first introduced to this tool through the Laravel framework which requires Supervisor to keep its command line program running to process events that have been pushed on to a queue.
This combination was working great until one day we notice the queue was no longer being processed. Checking the server we found our program had entered the following status “FATAL Exited too quickly”. By default Supervisor will try to restart your program 3 times before deciding to permanently give up and assign a status of FATAL to that program.
You can set Supervisor to permantly to restart your program, but in our case if the program had failed 3 times it’s probably due to an issue that we need to be alerted to. Fortunately Supervisor allows you to add listeners to events like program status changes. Even more conveniently there is already a suit of listeners called Superlance which handle common tasks like email failed program alerts.
After installing Superlance via pip
We used the fatalmailbatch listener to email the team when a program entered a FATAL state. Adding a listener is just like adding any other program. I added a new config file at /etc/supervisor/conf.d/fatalmailbatch.conf
And then started the listener running with supervisorctl
All in all pretty simple and adds a bit more peace of mind.