Celery task priority is not working as expected

CodingNow Source

I'm new to celery and I found that the task priority is quite confusing.

I write a very simple Python program to test the behavior of celery task priority but the result seems random.

My environment: Mac OS 10.13.5. Latest version of celery, rabbitMQ, python3(I use brew to install them just now).

Steps: My program: (tasks.py)

from __future__ import absolute_import, unicode_literals
from celery import Celery
from kombu import Queue, Exchange
import time

app = Celery('tasks', backend = 'rpc://', broker = 'pyamqp://')

app.conf.task_queues = [
    Queue('tasks', Exchange('tasks'), routing_key = 'tasks',
          queue_arguments={'x-max-priority': 10})
    ]

app.conf.worker_prefetch_multiplier = 1
app.conf.task_acks_late = True

@app.task
def example_task1():
    time.sleep(3)
    print('End1')

@app.task
def example_task2():
    time.sleep(3)
    print('End2')

@app.task
def example_task3():
    time.sleep(3)
    print('End3')

(add_tasks.py)

from tasks import example_task1, example_task2, example_task3

result1 = example_task1.apply_async(queue='tasks', priority = 1)
result2 = example_task2.apply_async(queue='tasks', priority = 3)
result3 = example_task3.apply_async(queue='tasks', priority = 2)

print(result1.get(timeout=1), result2.get(timeout=1), result3.get(timeout=1))

In command line:

celery worker -A tasks -Q tasks --loglevel=info

My expected result:

example_task2 runs before example_task3(because task 2 has higher priority than task 3).

Actual result(tried two rounds):

[2018-06-13 16:17:56,258: INFO/MainProcess] Received task: tasks.example_task1[580b916e-7df2-423a-8995-10a9f5401a52]
[2018-06-13 16:17:56,259: INFO/MainProcess] Received task: tasks.example_task2[fc4117a3-672f-4986-910d-104371aace13]
[2018-06-13 16:17:56,260: INFO/MainProcess] Received task: tasks.example_task3[751c419d-4d9f-4853-bd54-725ae60ead2e]
[2018-06-13 16:17:59,262: WARNING/ForkPoolWorker-1] End2
[2018-06-13 16:17:59,262: WARNING/ForkPoolWorker-8] End1
[2018-06-13 16:17:59,263: WARNING/ForkPoolWorker-3] End3

[2018-06-13 16:18:38,940: INFO/MainProcess] Received task: tasks.example_task1[bea8100b-afc7-43f6-9d40-e27f5c066b39]
[2018-06-13 16:18:38,941: INFO/MainProcess] Received task: tasks.example_task2[735c7ee1-4913-49cf-8093-2c7a074f98bb]
[2018-06-13 16:18:38,942: INFO/MainProcess] Received task: tasks.example_task3[c17be8cc-eca7-4a77-a3e7-80ddaa892f24]
[2018-06-13 16:18:41,944: WARNING/ForkPoolWorker-1] End3
[2018-06-13 16:18:41,943: WARNING/ForkPoolWorker-5] End1
[2018-06-13 16:18:41,945: INFO/ForkPoolWorker-1] Task tasks.example_task3[c17be8cc-eca7-4a77-a3e7-80ddaa892f24] succeeded in 3.001734145997034s: None
[2018-06-13 16:18:41,944: WARNING/ForkPoolWorker-7] End2
pythoncelerycelery-task

Answers

comments powered by Disqus