8.7. Launch Celery
When web sites want to do something slow, or spontaneously, they should not do so within a user interaction. If the user says “build me a 1 Gb file for download”, the web site shouldn’t build it and then provide it for download; the user will think the site is unresponsive. Usually, it should reply immediately to say “OK, working on that”, do the work, and then indicate to the user somehow that the job is done. Similarly, web sites may need to act spontaneously (e.g. do things at midnight every day).
The general method of doing this is to have a task queue. CRATE uses Celery for its task queue. It creates Celery tasks when it wants to do something slow (like checking consent modes and sending a bunch of e-mails).
In turn, Celery puts tasks into a message queue system, passing messages via the AMQP protocol. By default, the message broker is RabbitMQ. It’s the job of the broker to receive messages, keep them safe (so that they’re not lost if the system is rebooted, for example), and to make them available to software that needs them.
When a task is run, Celery will call back into the CRATE code to get the job done, but using a different process from the one that responded to the user’s initial HTTP request.
8.7.1. crate_launch_celery
This launches the CRATE Celery system. You need this to be running for CRATE to work properly. (See also Windows service.)
Options:
USAGE: crate_launch_celery [-h] [--command COMMAND]
[--cleanup_timeout_s CLEANUP_TIMEOUT_S] [--debug]
Launch CRATE Celery processes. (Any leftover arguments will be passed to
Celery.)
OPTIONAL ARGUMENTS:
-h, --help show this help message and exit
--command COMMAND Celery command (default: worker)
--cleanup_timeout_s CLEANUP_TIMEOUT_S
Time to wait when shutting down Celery via Ctrl-C
(default: 10.0)
--debug Ask Celery to be verbose (default: False)
8.7.2. crate_celery_status
This executes the command celery -A crate_anon.crateweb.consent status
.
8.7.3. crate_launch_flower
This command has no options. It launches the Celery Flower tool, which is for monitoring Celery, and associates it with the CRATE NLP web server. It starts a local web server (by default on port 5555; see TCP/IP ports); if you browse to http://localhost:5555/ or http://127.0.0.1:5555/, you can monitor what’s happening.
8.7.4. See also
If Celery jobs are taken out of the queue and then crash inside CRATE, you may wish to resubmit unprocessed work. For this, use
crate_django_manage resubmit_unprocessed_tasks
See crate_django_manage.