Setting up Supervisord for Celery¶
When implementing celery on a production instance it may be preferable to delegate supervisord to manage celery workers and celery beats. The following is a guide for a linux-based OS; be advised you can change any of the file names, destinations, or permissions to suit your needs.
Supervisor can be installed using
pip
:pip install supervisor
.In the core arches repo, in arches/install/supervisor_celery_setup there exist example files for supervisor, celeryd, and celerybeat. We recommend copying them into the following directory structure:
/etc/supervisor/ |-- my_proj_name-supervisord.conf |-- conf.d/ | |-- my_proj_name-celeryd.conf | |-- my_proj_name-celerybeat.conf
In the content of the files as well as the filenames themselves, replace the values of the following placeholders:
/absolute/path/to/virtualenv/
- absolute path to your python3 virtualenv
[app]
- replace this with the value ofELASTICSEARCH_PREFIX
in your project’ssettings.py
file
/absolute/path/to/my_proj
- absolute path to your arches project
my_proj_name
- name of your project
Note that you can change the value for
user
in the-supervisord.conf
file to a designated user to run supervisord.Before proceeding, you will want to make sure that whichever user you designate to run supervisor has the appropriate permissions for the following files:
/var/log/supervisor/supervisord.log
/var/log/celery/worker.log
/var/log/celery/beat.log
/var/run/supervisord.pid
Download and install RabbitMQ: https://www.rabbitmq.com/download.html
Once successfully installed (and verified that it has been added to your PATH), start running it with the command
rabbitmq-server
. For a convenient option, this can be run in a screen. Note that rabbitmq should be run prior to running supervisord.Run
supervisord -c /etc/supervisor/my_proj_name-supervisord.conf
to start the supervisord which will start celery workers for your tasks.To stop your supervisord process, run
unlink /tmp/supervisor.sock
.
- More information:
Supervisord documentation: http://supervisord.org/
Celery sample files for supervisord: https://github.com/celery/celery/tree/master/extra/supervisord