Setting up Supervisord for Celery

When implementing celery on a production instance it may be preferable to delegate supervisord to manage celery workers and celery beats. The following is a guide for a linux-based OS; be advised you can change any of the file names, destinations, or permissions to suit your needs.

  1. Supervisor can be installed using pip: pip install supervisor.

  2. In the core arches repo, in arches/install/supervisor_celery_setup there exist example files for supervisor, celeryd, and celerybeat. We recommend copying them into the following directory structure:

    /etc/supervisor/
|-- my_proj_name-supervisord.conf
|-- conf.d/
|    |-- my_proj_name-celeryd.conf
|    |-- my_proj_name-celerybeat.conf

  3. In the content of the files as well as the filenames themselves, replace the values of the following placeholders:

  • /absolute/path/to/virtualenv/ - absolute path to your python3 virtualenv

  • [app] - replace this with the value of ELASTICSEARCH_PREFIX in your project’s settings.py file

  • /absolute/path/to/my_proj - absolute path to your arches project

  • my_proj_name - name of your project

  1. Note that you can change the value for user in the -supervisord.conf file to a designated user to run supervisord.

  2. Before proceeding, you will want to make sure that whichever user you designate to run supervisor has the appropriate permissions for the following files:

  • /var/log/supervisor/supervisord.log

  • /var/log/celery/worker.log

  • /var/log/celery/beat.log

  • /var/run/supervisord.pid

  1. Download and install RabbitMQ: https://www.rabbitmq.com/download.html

  2. Once successfully installed (and verified that it has been added to your PATH), start running it with the command rabbitmq-server. For a convenient option, this can be run in a screen. Note that rabbitmq should be run prior to running supervisord.

  3. Run supervisord -c /etc/supervisor/my_proj_name-supervisord.conf to start the supervisord which will start celery workers for your tasks.

  4. To stop your supervisord process, run unlink /tmp/supervisor.sock.

More information: