8.7. Launch Celery

When web sites want to do something slow, or spontaneously, they should not do so within a user interaction. If the user says “build me a 1 Gb file for download”, the web site shouldn’t build it and then provide it for download; the user will think the site is unresponsive. Usually, it should reply immediately to say “OK, working on that”, do the work, and then indicate to the user somehow that the job is done. Similarly, web sites may need to act spontaneously (e.g. do things at midnight every day).

The general method of doing this is to have a task queue. CRATE uses Celery for its task queue. It creates Celery tasks when it wants to do something slow (like checking consent modes and sending a bunch of e-mails).

In turn, Celery puts tasks into a message queue system, passing messages via the AMQP protocol. By default, the message broker is RabbitMQ. It’s the job of the broker to receive messages, keep them safe (so that they’re not lost if the system is rebooted, for example), and to make them available to software that needs them.

When a task is run, Celery will call back into the CRATE code to get the job done, but using a different process from the one that responded to the user’s initial HTTP request.

8.7.1. crate_launch_celery

This launches the CRATE Celery system. You need this to be running for CRATE to work properly. (See also Windows service.)

Options:

USAGE: crate_launch_celery [-h] [--command COMMAND]
                           [--cleanup_timeout_s CLEANUP_TIMEOUT_S] [--debug]

Launch CRATE Celery processes. (Any leftover arguments will be passed to
Celery.)

OPTIONAL ARGUMENTS:
  -h, --help            show this help message and exit
  --command COMMAND     Celery command (default: worker)
  --cleanup_timeout_s CLEANUP_TIMEOUT_S
                        Time to wait when shutting down Celery via Ctrl-C
                        (default: 10.0)
  --debug               Ask Celery to be verbose (default: False)

8.7.2. crate_celery_status

This executes the command celery -A crate_anon.crateweb.consent status.

8.7.3. crate_launch_flower

This command has no options. It launches the Celery Flower tool, which is for monitoring Celery, and associates it with the CRATE NLP web server. It starts a local web server (by default on port 5555; see TCP/IP ports); if you browse to http://localhost:5555/ or http://127.0.0.1:5555/, you can monitor what’s happening.

8.7.4. See also

If Celery jobs are taken out of the queue and then crash inside CRATE, you may wish to resubmit unprocessed work. For this, use

crate_django_manage resubmit_unprocessed_tasks

See crate_django_manage.