XL Deploy problem with RabbitMQ doing rollback of a deployment

Hello there,

I have deployed two XL Deploy instances (one master and one worker) and a RabbitMQ cluster with three instances. When I try to do a deployment there’s no problem, but when I try to do a rollback of any deployment XL Deploy queues the task as you can see in the image below:
image

I have no reached any conclusion because I cannot understand how this works fine to do the deployment but fails doing the rollback.

Are there any known issues about this?

Hello fleonfer, are your RabbitMQ instances configured for the Topic approach discussed here - https://docs.xebialabs.com/v.10.3/deploy/how-to/configure-task-queueing/#configure-your-jms-message-broker-setup

1 Like

First of all, thank you for your response.
I think I have referenced well RabbitMQ in XL Deploy and I guess the main problem comes from how XL Deploy do the rollback in a HA mode. I don’t know how XL Deploy execute the rollback but RabbitMQ seems to work fine.

Does anyone of you have info about how XL Deploy creates the queues in RabbitMQ?
As I can see there are created two queues of classic type and I think it would be better if they are of type quorum but I don’t know where to change this.

This is the error I’m getting in a loop:

Caused by: com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, │
│ xldeployha at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:258) │
│ xldeployha at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:341) │
│ xldeployha at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:282) │
│ xldeployha at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141) │
│ xldeployha at com.rabbitmq.client.impl.ChannelN.queueBind(ChannelN.java:1077) │
│ xldeployha at com.rabbitmq.client.impl.ChannelN.queueBind(ChannelN.java:52) │
│ xldeployha at com.rabbitmq.client.impl.recovery.RecordedQueueBinding.recover(RecordedQueueBinding.java:30) │
│ xldeployha at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.lambda$recoverBinding$6(AutorecoveringConnection.java:764) │
│ xldeployha at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.wrapRetryIfNecessary(AutorecoveringConnection.java:818) │
│ xldeployha at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverBinding(AutorecoveringConnection.java:763) │
│ xldeployha … 7 common frames omitted

the link above discusses rollback -

Note the following:

  • The Queue approach is used for tasks such as initial deployments, upgrades and undeployments.
  • The Topic approach is used for Rollback to send the Rollback task to the specific worker that was performing the deployment to be rolled back.
  • For RabbitMQ, the Topic exchange is disabled by default and should be enabled by Enabling the Topic Selector Plug-in.

Can you confirm the topic selector plugin is enabled?

Hi!

Yes, the selector plugin is enabled. But after reading this, what should happen if the worker that executed the deployment goes down after doing it and before the rollback?

Thank you for your response!

Hi!

I think I’ve found the problem. A RabbitMQ cluster with multiple instances selects one of them and name it master, if you delete the master instance it fails so the solution is to rebalance the queues.