Thursday, September 29, 2011

Django-Celery on Webfaction using RabbitMQ

This tutorial is meant to get you up and running from scratch with django-celery on Webfaction. Each of the steps is a bit of a hassle since you typically need to find different install steps for each individual part, so I just lumped up the whole experience in this guide.

Install Erlang

Erlang is needed for installing RabbitMQ which is the preferred message broker for Celery. Webfaction doesn't come with this installed, so you'll need to do it manually:
  • Go to the webfaction control panel and create a new app -> Custom App, Listening on Port
  • Download the latest version of Erlang. You can just use the command: wget http://www.erlang.org/download/otp_src_R14B03.tar.gz
  • Unzip it: gunzip -c otp_src_R14B03.tar.gz | tar xf -
  • cd into the directory
  • Configure the build: ./configure --prefix=/home/your_webfaction_username/
  • Make it: make
  • Install it: make install
  • Run it on the port given to you when you created the new Erlang app: epmd -port 12345 -daemon

Install RabbitMQ

Next, you need to change the file ~/lib/rabbitmq/rabbitmq-server. I found some information about this on the webfaction community forums. Open your text editor and change three lines to:
CONFIG_FILE=~/src/rabbitmq_server-2.6.1/sbin/
LOG_BASE=~/logs/user/rabbitmq
MNESIA_BASE=~/src/rabbitmq_server-2.6.1/sbin/
Added these lines to the rabbitmq-env file and use the ports you reserved for epmd and rabbitmq in your earlier steps:
export ERL_EPMD_PORT=12708
export RABBITMQ_NODE_PORT=35478
export ERL_INETRC=$HOME/.erl_inetrc
Added the file $HOME/hosts which looks like:
127.0.0.1 localhost.localdomain localhost
::1      localhost6.localdomain6 localhost6
127.0.0.1 web160 web160.webfaction.com
Added the file $HOME/.erl_inetrc which looks like:
{hosts_file, "/home/<your_user_name>/hosts"}.
{lookup, [file,native]}.
Run Rabbitmq and check that it is working:
./rabbitmq-server -detached
./rabbitmqctl status
Finally, add a new user and vhost, and configure it so only your app will have access to it.
./rabbitmqctl add_user <username> <password>
./rabbitmqctl set_user_tags <username> administrator
./rabbitmqctl add_vhost <vhostpath>
./rabbitmqctl set_permissions -p <vhostpath> <username> ".*" ".*" ".*"
./rabbitmqctl clear_permissions -p <vhostpath> guest

Install Celery and Django-Celery

pip install django-celery
In your settings file, you will then need to add the lines:
BROKER_HOST = "localhost"
BROKER_PORT = 36784
BROKER_USER = "username"
BROKER_PASSWORD = "password"
BROKER_VHOST = "vhostpath"
CELERYD_CONCURRENCY = 1
CELERYD_NODES="w1"
CELERY_RESULT_BACKEND="amqp"
The reason we set the concurrency so low is because Celery takes up a good amount of memory, and you are likely limited with your memory consumption on webfaction. The minimum amount of memory Celery can take will be however much it needs to run the main process (consuming messages, sending tasks to workers, etc), and a worker tasks that actually does stuff. Each of these will take up about 20-30MB of memory depending on the size of your Django app.
Add 'djcelery' to your installed apps.
Follow any other steps listed in their installation guide that are relevant to your app.  If you are using mod_wsgi, add the following to your .wsgi module:
import os
os.environ["CELERY_LOADER"] = "django"

Install a tool to create a Daemon

Celery does not daemonize itself, and thus you need to do this yourself. Creating a daemon is not exactly the same as simply running it in the background, so you should install a tool that can help you do this. Celery recommends a couple options. One of the easiest ways is to use a simple tool called django-supervisor. To install this, just type: 
pip install django-supervisor
Add the file supervisord.conf in the same directory as manage.py, and add the content:
[program:celeryd]
command={{ PYTHON }} {{ PROJECT_DIR }}/manage.py celeryd -l info
[program:autoreload]
exclude=true
[program:runserver]
exclude=true
[program:celerybeat]
exclude=true
Every time you restart your webserver, you can restart celery by issuing the following commands:
python manage.py supervisor --daemonize
python manage.py supervisor stop all
python manage.py supervisor start all
However, there is a downside with using django-supervisor in that it will run in the background and take up another 20-30MB of memory. A more memory efficient way would be to install a tool called daemonize. This page has very easy installation instructions. Once you install it, just add an alias to it in your .bashrc or .profile and then run:
daemonize /<full_path_to_django_directory>/manage.py celeryd
Everything should then be up and running. Good luck!