Project

General

Profile

Actions

RabbitMQ configuration and monitoring

Celery uses RabbitMQ (http://www.rabbitmq.com/) as a broker to dispatch jobs to processing nodes.

All commands need to be run as a super-user by using sudo su - or su.

Celery connection information

To connect with RabbitMQ, Celery workers nespecify:

  • Host name - name of host on which RabbitMQ is running. If local this can be localhost.
  • Port - by default RabbitMQ uses 5672. If you want worker nodes outside your firewall to use the RabbitMQ broker then you will need to open this port.
  • Virtual host name - maushost
  • Username - maus
  • Password - saum

Running the server

Start server

$ /sbin/service rabbitmq-server start
Starting rabbitmq-server: SUCCESS
rabbitmq-server.

Restart server

$ /sbin/service rabbitmq-server restart
Restarting rabbitmq-server: SUCCESS
rabbitmq-server.

Stop server

$ /sbin/service rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server

View server status

$ /sbin/service rabbitmq-server status
Status of all running nodes...
Node 'rabbit@maus' with Pid 24899: running
done

Note that maus is the name of the host on which RabbitMQ is running.

Server monitoring

rabbitmqctl is a super-user command that allows current message queues and connections to RabbitMQ to be viewed. For full information, see http://www.rabbitmq.com/man/rabbitmqctl.1.man.html.

List the current connections to RabbitMQ

You can show the Celery workers that have connected. This can be useful for checking that a Celery worker has successfully connected. By default this shows the IP address and connection port of the Celery worker and the state of the connection.

$ rabbitmqctl list_connections
Listing connections ...
maus    129.215.62.184  51465   running
maus    127.0.0.1       54750   running
...done.

The above shows two connections, one from a Celery worker running on the same host as RabbitMQ (address 127.0.0.1) and one running on a remote host (129.215.62.184).

You can request more connection information be shown e.g.

  • pid - RabbitMQ connection ID.
  • address - RabbitMQ address used by worker to connect.
  • port - RabbitMQ port used by worker to connect.
  • peer_address - address of worker.
  • peer_port - port of worker.
  • user - RabbitMQ username used by worker to connect.
  • vhost - RabbitMQ virtual host used by worker to connect.
  • state - current state of connection.
$ rabbitmqctl list_connections pid address port peer_address peer_port user vhost state
Listing connections ...
<rabbit@maus.1798.0>   129.215.62.218  5672    129.215.62.184  51465   maus   maushost running
<rabbit@maus.1547.0>    127.0.0.1       5672    127.0.0.1       54750   maus   maushost running
...done.

Both are connected to the MAUS virtual host, mausvhost, with username maus.

Browse the message queues

You can also browse the mesage queues. For these operations, rabbitmqctl takes a -p flag with the name of a virtual host (for MAUS this is typically called maushost).

List queue names and sum of messages ready to be delivered and unacknowledged messages:

$ rabbitmqctl list_queues -p maushost
Listing queues ...
celery  0
worker2.celeryd.pidbox  0
worker1.celeryd.pidbox  0
...done.

The above corresponds to two Celery workers having connected to RabbitMQ.

List queue names, number of messages ready to be delivered, number of unacknowledged messages and sum of these:

$ rabbitmqctl list_queues -p maushost name messages_ready messages_unacknowledged messages
Listing queues ...
celery  0       0       0
worker2.celeryd.pidbox  0       0       0
worker1.celeryd.pidbox  0       0       0
...done.

The celery queue is of interest as its messages_ready count corresponds to the number of requests from a client that are awaiting to be delivered to a Celery worker node. This can be useful to check if a client's requests are, at least, getting to RabbitMQ successfully. For example, after a client invokes 8 Celery worker tasks:

celery  8       0       8
worker1.celeryd.pidbox  0       0       0
worker2.celeryd.pidbox  0       0       0
cffbe4e5989a48c8aed2411ceb64dd70        0       0       0
b9714b4775ca4985b1bc8d9a8dd64d9b        0       0       0
70f8edfcf33949bab255fbd4b02c2220        0       0       0
ee240f398517497e8b1036ffb96756e5        0       0       0
bfba8aa743f746849e5e97ed5855f056        0       0       0
1fdace10ae8f452abb0d676a4cb28c28        0       0       0
aa95178d13dd4168a32ce539a7368158        0       0       0
147c362d2496497ba7990cf07a5f1e27        0       0       0

You will also see, as above, one queue entry per task submitted. These have auto-generated names. As the tasks are executed and results await collection, the states of these will change. The celery queue empties as the Celery worker's receive the entries. The auto-generated queues fill with the results, awaiting for collection by the clients.

celery  0       0       0
worker1.celeryd.pidbox  0       0       0
worker2.celeryd.pidbox  0       0       0
cffbe4e5989a48c8aed2411ceb64dd70        1       0       1
b9714b4775ca4985b1bc8d9a8dd64d9b        1       0       1
70f8edfcf33949bab255fbd4b02c2220        1       0       1
ee240f398517497e8b1036ffb96756e5        1       0       1
bfba8aa743f746849e5e97ed5855f056        1       0       1
1fdace10ae8f452abb0d676a4cb28c28        1       0       1
aa95178d13dd4168a32ce539a7368158        1       0       1
147c362d2496497ba7990cf07a5f1e27        1       0       1

And, after the client picks up the results,

celery  0       0       0
worker1.celeryd.pidbox  0       0       0
worker2.celeryd.pidbox  0       0       0
cffbe4e5989a48c8aed2411ceb64dd70        0       0       0
b9714b4775ca4985b1bc8d9a8dd64d9b        0       0       0
70f8edfcf33949bab255fbd4b02c2220        0       0       0
ee240f398517497e8b1036ffb96756e5        0       0       0
bfba8aa743f746849e5e97ed5855f056        0       0       0
1fdace10ae8f452abb0d676a4cb28c28        0       0       0
aa95178d13dd4168a32ce539a7368158        0       0       0
147c362d2496497ba7990cf07a5f1e27        0       0       0

Eventually, the task-specific queues will disappear.

Log files

RabbitMQ logs are in /var/log/rabbitmq/.

Web management plug-in

RabbitMQ server 2.7+ supports a web-based management plug-in that allow connections, queues and a wealth of other information to be explored. For full information, see http://www.rabbitmq.com/management.html.

Installation

  • Run:
    $ rabbitmq-plugins enable rabbitmq_management
    
    The following plugins have been enabled:
      webmachine
      rabbitmq_mochiweb
      amqp_client
      rabbitmq_management_agent
      rabbitmq_management
    
    Plugin configuration has changed. Restart RabbitMQ for changes to take effect.
    
  • Restart RabbitMQ:
    $ /sbin/service rabbitmq-server restart
    Restarting rabbitmq-server: SUCCESS
    rabbitmq-server.
    

Use

Updated by Jackson, Mike over 9 years ago ยท 10 revisions