Python Celery explained for beginners to Professionals(Part-3) — Workers, Pool and Concurrency configurations of Python Celery

Hello readers,

If you have not read my previous posts, Please read it.

We will be discussing few important points about Celery Workers, Pool and its concurrency configurations in this post.

Celery Worker is the one which is going to run the tasks.

celery -A tasks worker --pool=prefork --concurrency=1 --loglevel=info

Above is the command to start the worker.

We will explore those command line options.

-A, --app <app>

This expects our python module in which celery object is created.

worker

is an option used to start the celery worker.

--loglevel

It is for logging.

DEBUG|INFO|WARNING|ERROR|CRITICAL|FATAL can be used.

--pool=prefork

Now comes very important option “ — pool”.

Options

prefork|eventlet|gevent|solo can be used.

--concurrency=1

Concurrency specifies how many tasks you want to run concurrently with this worker.

But what is all about pool and concurrency?

Before that we need to know little about Python Multi-threading and Multiprocessing.

Multi-threading and Multiprocessing is a way of running same task concurrently.

Python Interpreter doesn’t actually execute threads concurrently in Multi-threading due to its limitation Global Interpreter Lock (GIL).

To achieve real concurrency. You have two options.

1. Disable GIL (Not recommended due to various reasons).
2. Go with Multiprocessing. Multiprocessing spawns processes and can take advantage of hardware CPUs power. But Use of Multiprocessing too depends on what type of tasks you want to run concurrently. Multiprocessing can be used when tasks are CPU bound.

So remember Multiprocessing is good when tasks are CPU bound like arithmetic operations, Multi-threading is good when tasks are I/O or network bound like reading files from disk, requesting data from an API (HTTP). This way of implementation you have to do yourself when you do in plain python.

Celery offers different ways of running tasks using “pool” option. We have to select the “pool” based on the type of task we are doing (CPU bound or I/O or Network bound).

We can run Celery with below “pool” settings.

1. solo
2. prefork
3. eventlet
4. gevent

NOTE : Below examples assume you are running only one worker with different pool and concurrency options.

solo

Solo creates only one thread and runs celery tasks using that thread. Concurrency number cannot be provided here.

celery -A tasks worker --pool=solo --loglevel=info

E.g.; You have defined one task which downloads a movie specified in the task.

Now you want to download 10 movies, you have submitted 10 tasks. As we have one worker running. Worker picks up tasks from the Queue and starts running in the thread. As we have only one thread here. It cannot pick another task until existing task is completed.

Assume on an average time taken to download one movie is one hour. Now this Worker takes 10 hours to complete all 10 tasks that is t download all 10 movies. So pool solo is not a good fit for this kind of tasks.

solo has it’s own use case based on the problem.

prefork are best pool option for CPU bound.

prefork

prefork uses Multiprocessing.

— concurrency option can be provided. (Recommended to provide the number of CPUs of the machine where Celery Worker is running)

celery -A tasks worker --pool=prefork --concurrency=4 --loglevel=info

E.g.; We have Celery Worker running on 4 CPUs machine. You have defined one task which does some complex mathematical calculation.

Now you want to run this calculation on 10 different datasets, you have submitted 10 tasks. As we have one worker running. Worker picks up tasks from the Queue and starts running in this processes. As we have 4 processes here, It can run 4 tasks concurrently.

eventlet and gevent are best pool option for I/O and Network.

eventlet

Eventlet is a concurrent networking library for Python that allows you to change how you run your code, not how you write it.

• It uses epoll or kqueue or libevent for highly scalable non-blocking I/O.
• Coroutines ensure that the developer uses a blocking style of programming that is similar to threading, but provide the benefits of non-blocking I/O.
• The event dispatch is implicit, which means you can easily use Eventlet from the Python interpreter, or as a small part of a larger application.

source : https://eventlet.net/

eventlet doesn’t create multiple threads with concurrency option. Instead what it does is it creates only one thread and handles the concurrency with one thread using concept called event loop.

celery -A tasks worker --pool=eventlet --concurrency=10 --loglevel=info

E.g.; You have defined one task which downloads a movie specified in the task.

Now you want to download 10 movies, you have submitted 10 tasks. As we have one worker running. Worker picks up tasks from the Queue and starts running in the thread. As we have eventlet running with concurrency as 10, all tasks starts downloading the respective movies.

Based on network speed, all 10 files will be downloaded very fast.

gevent

gevent is a coroutine -based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev or libuv event loop.

Features include:

• Fast event loop based on libev or libuv.
• Lightweight execution units based on greenlets.
• API that re-uses concepts from the Python standard library (for examples there are events and queues).
• Cooperative sockets with SSL support
• Cooperative DNS queries performed through a threadpool, dnspython, or c-ares.
• Monkey patching utility to get 3rd party modules to become cooperative
• TCP/UDP/HTTP servers
• Subprocess support (through gevent.subprocess)
• Thread pools

gevent is inspired by eventlet but features a more consistent API, simpler implementation and better performance. Read why others use gevent and check out the list of the open source projects based on gevent.

source : http://www.gevent.org/

gevent doesn’t create multiple threads with concurrency option. Instead what it does is it creates only one thread and handles the concurrency with one thread using concept called event loop.

celery -A tasks worker --pool=gevent --concurrency=10 --loglevel=info

E.g.; You have defined one task which downloads a movie specified in the task.

Now you want to download 10 movies, you have submitted 10 tasks. As we have one worker running. Worker picks up tasks from the Queue and starts running in the thread. As we have gevent running with concurrency as 10, all tasks starts downloading the respective movies.

Based on network speed, all 10 files will be downloaded very fast.

We will get into more in our next story. If you like the story, please follow me and provide suggestions.