Yes, Celery!

I’ve got a confession! Don’t judge me!

I am in love with Celery! It is very much a beauty!

A project I’ve been thinking of doing, had a very bad disadvantage. Usually, it is not easily scalable. You have to do everything synchronously. If you want to do things fat, run multiple instances. But how to orchestrate that? How to ensure they play safe?

Well, come Celery! Celery is simply a task queue which utilises some python magic, to be a good task queue. It is my very first introduction to a task queue, so.. what do I know?

Maybe I should say I am in love with task queues.

When I use the gevent pool with celery, simply by

celery -A myapp -P gevent -c 1000

It can scale very much! Thanks to the magic of gevent. That 1000 there? That’s how many green threads celery will use with gevent. gevent makes celery insanely scalable! As long as your tasks are io intensive.

Blissful is my life. For I have celery and gevent to accompany me.

1 comment