I have a queue and 3 consumers bind to the queue. Each consumer has a prefetch_count of 250(or say X) and manual acknowledgement is done prefetch_count(X)/2(i.e 125) messages - meaning consumer manually acknowledges 125 messages in a single go (which helps to reduce round-trip time and hence increases performance). Everything is working fine as expected but the only issue arises when there are no new messages in the queue and the consumers have some unacknowledged messages whose count is less 125.
As the acknowledgement is only sent when the count is 125, these unacknowledged messages keeps requeuing. How can I solve this ?
How can I know that my consumer has no new messages to process and can acknowledge all the remaining messages waiting to be acknowledged.python-3.xrabbitmq
If I understand your scenario correctly, it sounds as though you have a series of messages that get published all at once, and then you process them in batches of 250 at a time until you have none left. The problem is, that if you don't have a number of messages that is divisible by 125, then your final batch never gets acknowledged. Clearly this is a logical problem, but it sounds like you are wondering if there is an easy way to deal with it.
Your question "How can I know that my consumer has no new messages to process?" is based upon a premise which RabbitMQ does not support -- namely, the "end" of a sequence of messages. RabbitMQ consumers expect to continue to receive messages indefinitely, so from their perspective, there is no such thing as "done."
Thus, any such concept must be implemented elsewhere, higher up in your application logic. Here are some options for you to consider:
Caveat: I would argue that you have some deeper design problem going on that is leading down the path where it would ever be desirable to do this in the first place. Each message should be 100% independent of any other message. If that assumption is violated, you will have a very fragile system.