ActiveMQ Moving Several Queues at Once - Example
(Update in 2022 for log4j)
There are times that a large number of queues need to be moved from one broker or another - or upload from/download to files.
One example is DLQs building up and taking too much storage space or something going wrong and needing to recover an ActiveMQ instance that had a load of messages on it. Here are a couple of scripts used to find all queues with messages without consumers and convert it into the Java Camel from-to statements in the code below. It uses the AMQ UI to get the data since using jolokia required more steps but it might be cleaner to use jolokia!
Get the queues:
curl -o queues http://0.0.0.0:8161/admin/xml/queues.jsp
tr -d "\n" < queues | sed "s/<queu/\n<queu/g" | grep -v size=\"0 | grep consumerCount=\"0 | sed 's/\"> /\"\/> /' | cut -d" " -f1,2 | cut -d\" -f2
Now, how to move all of the messages... (btw, there are other ways like adding bridges). Camel makes this fairly easy - all you need is a Java "wrapper" to run the camel config. Of course, there are two ways of doing this: camel config in its own file or camel used in Java files.
This example uses a camel example, along with some linux commands, to find and move messages from/to queues.
Setup a directory (such as TestCamel) with this directory under it (use IntelliJ or Eclipse to make it easier):
mkdir -p src/main/java
Add a Java file in that src/main/java directory, I've called my TestCamel.java:
package com.testcamel;
import javax.jms.ConnectionFactory;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.camel.CamelContext;
import org.apache.camel.component.jms.JmsComponent;
import org.apache.camel.impl.DefaultCamelContext;
public class TestCamel {
public static void main(String[] args) {
BasicRB rB = new BasicRB();
CamelContext ctx = new DefaultCamelContext();
ConnectionFactory connectionFactoryFrom = new ActiveMQConnectionFactory("tcp://0.0.0.0:61616");
ctx.addComponent("jmsFrom", JmsComponent.jmsComponentAutoAcknowledge(connectionFactoryFrom));
ConnectionFactory connectionFactoryTo = new ActiveMQConnectionFactory("tcp://10.10.0.38:61616");
ctx.addComponent("jmsTo", JmsComponent.jmsComponentAutoAcknowledge(connectionFactoryTo));
try {
ctx.addRoutes(rB);
ctx.start();
Thread.sleep(5000);//give it 5 seconds to work
ctx.stop();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
Add another Java file, I've called it BasicRB.java for basic camel route builder:
package com.testcamel;
import org.apache.camel.builder.RouteBuilder;
public class BasicRB extends RouteBuilder {
@Override
public void configure() throws Exception {
from("jmsFrom:queue:Activemq.DLQ").to("jmsTo:queue:reprocessing_application_10.queue");
}
}
Add this pom.xml to the top level directory (above src/):
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.testcamel</groupId>
<artifactId>camel-test-activemq</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<build>
<plugins>
<!-- any other plugins -->
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.14.2</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jms</artifactId>
<version>2.14.2</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
<version>5.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-broker</artifactId>
<version>5.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-client</artifactId>
<version>5.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-pool</artifactId>
<version>5.10.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/javax.jms/javax.jms-api -->
<dependency>
<groupId>javax.jms</groupId>
<artifactId>javax.jms-api</artifactId>
<version>2.0.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-slf4j-impl -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.18.0</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>2.0.0-alpha7</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Then you should be able to run:
mvn package
Which will kick off the compile and bundling of the code into a jar in the target/ directory
To run it is fairly straight forward - add the target to the classpath. It runs for 5 sec and then stops which might be enough time to move all of your messages.
java -cp camel-test-activemq-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.testcamel.TestCamel
As mentioned above, there are a few other options for moving messages temporarily - bridged queues from one broker to another or camel config in ActivMQ (will add that later) or pulling and pushing messages via the APIs or ActiveMQ cli consumer/producer. For longer term moving of messages and depending on your use case, virtualtopics and/or AMQ config via bridges or static routes is better.
Comments
Post a Comment