Failed publishing needs to be handled by the microservice with code patterns such as a circuit breaker, but failure to process consumed messages should be handled by RabbitMq.

If a microservice cannot process a message in some cases you may want to retry a few times(preferably with some time between the retries), and if it still fails after a certain amount of retries then escalate the message for manual investigation. Retries put more strain on the entire system because there are more messages on the queues and the microservices have to consume and process more messages. Therefore in order to conserve cpu, ram and network try reduce what you retry.

What could make a message fail?

First simple retry strategy.

  1. Reject the failed message with requeue true. (message will be queued at the front)
  2. The micro services will then pick up the message again and retry.
  3. To prevent a loop. check if the message has redelivered set to true, if so then do not requeue, rather reject without requeue.

pros

cons

  1. Reject failed message so that it gets sent to the dead letter exchange, which puts a time to live on the messages in its queue.
  2. The x-dead-letter-routing-key must be sent on the rejecting queue so that it changes the routing key to the name of the microservices inbound exchange.(for routing back)
  3. Messages get routed from the DLX to a wait-queue where it waits based on the ttl on the queue.
  4. Once a message expires it will be sent to the rerouter which is the wait-queues dead letter exchange.
  5. The rerouter will route the message back to the original exchange from the changed routing key.
  6. The application must check messages for the x-death header where the count will show the number of retries.
  7. If a message has retried too many times it must be published to an escalation queue and the original message must be Acked.

pros

cons

  1. Clone the message
  2. Add a header to the clone, retry = 3 (or -1 from an existing retry header, or reject if retry = 0)
  3. Change the clones routing key to match the microservice's inbound exchange (so that a generic rerouter can send it back here). Can add a number onto the rerouting key to select a wait queue.
  4. Ack the original message so that it is not sent to the dead letter exchange and not requeued. (the clone must be requeued)
  5. Publish the clone to a globally used RetryEx, which lands on a wait-queue. (can get fancy and route the message to a queue with the correct wait time eg waite-queue-3 or wait-queue-10)
  6. The message waits on a wait-queue due to the ttl on the queue.
  7. Once a message expires it will be sent to the rerouter which is the wait-queues dead letter exchange.
  8. The rerouter will route the message back to the original exchange from the changed routing key.

pros

cons

When implementing the above structures you can have many microservices use the same queues and exchanges by changing the routing key call so that the message can find its way home.

You can also have many different queues with different ttl's on them. Why"

Check out these links for more info:

My design and architecture repo