Setting up Supervisord for Celery¶
When implementing celery on a production instance it may be preferable to delegate supervisord to manage celery workers and celery beats. The following is a guide for a linux-based OS; be advised you can change any of the file names, destinations, or permissions to suit your needs.
Supervisor can be installed using
pip install supervisor.
In the core arches repo, in arches/install/supervisor_celery_setup there exist example files for supervisor, celeryd, and celerybeat. We recommend copying them into the following directory structure:
/etc/supervisor/ |-- my_proj_name-supervisord.conf |-- conf.d/ | |-- my_proj_name-celeryd.conf | |-- my_proj_name-celerybeat.conf
In the content of the files as well as the filenames themselves, replace the values of the following placeholders:
/absolute/path/to/virtualenv/- absolute path to your python3 virtualenv
[app]- replace this with the value of
ELASTICSEARCH_PREFIXin your project’s
/absolute/path/to/my_proj- absolute path to your arches project
my_proj_name- name of your project
Note that you can change the value for
-supervisord.conffile to a designated user to run supervisord.
Before proceeding, you will want to make sure that whichever user you designate to run supervisor has the appropriate permissions for the following files:
Download and install RabbitMQ: https://www.rabbitmq.com/download.html
Once successfully installed (and verified that it has been added to your PATH), start running it with the command
rabbitmq-server. For a convenient option, this can be run in a screen. Note that rabbitmq should be run prior to running supervisord.
supervisord -c /etc/supervisor/my_proj_name-supervisord.confto start the supervisord which will start celery workers for your tasks.
To stop your supervisord process, run
- More information:
Supervisord documentation: http://supervisord.org/
Celery sample files for supervisord: https://github.com/celery/celery/tree/master/extra/supervisord