Topic Forwarder Python script

[sgsonter]

HI;

I'm trying to modify the topic forwarder python script you provided on the website under the title "Middleware plumbing".
I've attached the original code at the bottom of this e-mail for reference;

I'd like to forward from an existing queue to a newly created topic so that I can record and playback the messages.
This is slightly different from creating new queues and topics as in the code sample.

I realize that there's the possibility of "missing" messages which are consumed before Hermes polls the queue, but I'm hoping that it's "good enough" for the job requirement.

It's an existing production system with persistent messaging requirements, so I don't have flexibility to convert the existing JMS queue into a topic.

The idea is to retrieve messages from the queue, but not "commit" my reads. This should leave the messages free to be consumed by the intended system consumer. I then forward the retrieved messages to a topic which I record for test analysis and playback.

1. The python example code as provided doesn't execute in Hermes 1.10. It seem to not properly initialize the "browser" object. I'm hoping that adding browser = hermes.browser.HermesBrowser.getBrowser() and proper imports will do the trick.

2. I can't understand how the provided code creates the connection / session / destination managed objects I'm used to using in JMS. Are "wms" and "emq" javax.jmx.ConnectionFactory objects or JNDIConnectionFactory objects? something else?

3. How can I lookup an existing queue using the ?.?.?connnectionFactory classes?

4. How can I "rollback" my hermesIn.receive() or simply not commit it?

Because its typeless jython, a poor old java programmer like me is a bit lost. I can't quite figure out the class usages from javap, and the javadocs aren't included in the distro.

Any help gratefully appreciated;
Simon


from com.jidesoft.document import DocumentComponent
from com.jidesoft.document import DocumentComponentListener
from hermes.swing import SwingRunner
from javax.swing import JTextArea
from javax.swing import JScrollPane
from java.lang import Runnable
from java.lang import Thread
from javax.jms import MessageListener
from org.apache.log4j import Logger

area = JTextArea()

class Forwarder (Runnable, DocumentComponent, DocumentComponentListener):
def __init__(self):
DocumentComponent.__init__(self, JScrollPane(area), "Message Forwarder")
self.__running = 1
area.setColumns(80)
area.setLineWrap(1)
self.addDocumentComponentListener(self)
area.append("Ready\n")

def run(self):
hermesIn = browser.getContext().lookup("wmq")
hermesOut = browser.getContext().lookup("ems")
inbound = hermesIn.createQueue("MUREX.OUT")
outbound = hermesOut.createTopic("TRADE.CONFIRM")

while self.__running:
message = hermesIn.receive(inbound, 500)
if message != None:
hermesOut.send(outbound, message)
hermesOut.commit()
hermesIn.commit()
area.append(message.toString())

hermesIn.close()
hermesOut.close()

def stop(self):
self.__running = 0

def documentComponentOpened(self, event):
return

def documentComponentClosing(self, event):
return

def documentComponentClosed(self, event):
self.stop()
return

def documentComponentMoving(self, event):
return

def documentComponentMoved(self, event):
return

def documentComponentActivated(self, event):
return

def documentComponentDeactivated(self, event):
return


forwarder = Forwarder()
browser.addDocumentComponent(forwarder)
thread = Thread(forwarder)
thread.start()

[Colin]

Hi,

You're right the documentation on python support and the Hermes messaging model is a bit thin. I've updated a couple of pages:

http://hermesjms.com/confluence/display/HJMS/Python

...which now links to...

http://hermesjms.com/confluence/display ... ging+Model

Hopefully this will help a bit.

I've also put the latest javadoc online.

What provider are you using and do you use JNDI to reference the administered objects?

The following code publishes the content of a queue to a topic - notice how little setup code there is as Hermes does all the work for you. Note that if either of the sessions are created from JNDI then the createQueue or createTopic call is mapped to a JNDI Context.lookup() call and the destination is located there.

The duplicate message call is there as I've found that some providers have problems with messages from foreign providers. You may not need it.

Code: Select all

hermesIn = browser.getContext().lookup("queues")
hermesOut = browser.getContext().lookup("topics")

try:
   queue = hermesIn.createQueue("MyQueue")
   topic = hermesOut.createTopic("MyTopic")
   queueBrowser = hermesIn.createBrowser(queue)
   enum = queueBrowser.getEnumeration()

   while enum.hasMoreElements():
      message = enum.nextElement()
      messageOut = hermesOut.duplicate(message) # optional
      hermesOut.send(topic, messageOut)
finally:
   if queueBrowser != None:
      queueBrowser.close()   
   if hermesOut != None:
      hermesOut.commit()
   if hermesIn != None:
      hermesIn.close()
   if hermesOut != None:
      hermesOut.close()


Finally, if you're using Python is would better to use 1.13 rather than 1.10.

Regards,

Colin.

[sgsonter]

Hi Colin;

Thank you very much for your speedy and thoughtful reply. Thanks also for publishing the javadoc and additional documentation.
The provider I'm using is WebLogic 9.2.
I'm still using HermesJMS 1.10 because I can't get a Hermes 1.13 install to work.
The 1.1.3 install proceeds normally, but I when I run the bat file (after making changes to accomodate my JAVA_HOME and HERMES_HOME) I get a splash screen and then the JVM exits. Myenvironment is a vanilla Windows XP / JDK150_10 environment so I'm not sure what the problem might be.

There are a large number of JMS servers in the infrastructure here, so I've configured a large number of sessions and contexts in HermesJMS. I had assumed that opening a context or session in a hermes window would allow the open JNDI initial context to be shared by the jython script. Now that I understand the thread local per HermesInterface model, I don't think this is possible. I'm surprised that HermesBrowser doesn't have a getContext() signature which includes a url and/or initialContext class.

I've cobbled together some script for a daemon to do the processing task based on your initial Fowarder example and the sample you povided above. The script will "load" to ready status, but doesn't currently process messages (even when there are no active consumers in the real system).
Do you know how I can check that "browser.getContext()" is getting the particular JNDI context I actually want?

Many thanks;
Simon

Code: Select all

 from com.jidesoft.document import DocumentComponent
from com.jidesoft.document import DocumentComponentListener
from hermes.swing import SwingRunner
from hermes.browser import HermesBrowser
from javax.swing import JTextArea
from javax.swing import JScrollPane
from java.lang import Runnable
from java.lang import Thread
from javax.jms import MessageListener
from org.apache.log4j import Logger

area = JTextArea()

class Forwarder (Runnable, DocumentComponent, DocumentComponentListener):
   def __init__(self):
      DocumentComponent.__init__(self, JScrollPane(area), "Message Forwarder")      
      self.__running = 1
      area.setColumns(80)
      area.setLineWrap(1)
      self.addDocumentComponentListener(self)      
      area.append("Ready\n")

   def run(self):
      hermesIn = browser.getContext().lookup("queues")
      hermesOut = browser.getContext().lookup("topics")

      while self.__running:
         try:
            queue = hermesIn.createQueue("queue.Fabric.ProductEvent.HaY")
               topic = hermesOut.createTopic("MyTopic")
               queueBrowser = hermesIn.createBrowser(queue)
               enum = queueBrowser.getEnumeration()
               while enum.hasMoreElements():
                     message = enum.nextElement()
                     messageOut = hermesOut.duplicate(message) # optional
                     hermesOut.send(topic, messageOut)
                     
         finally:
               if queueBrowser != None:
                  queueBrowser.close()   
               if hermesOut != None:
                  hermesOut.commit()
               if hermesIn != None:
                  hermesIn.close()
               if hermesOut != None:
                  hermesOut.close()
   
   def stop(self):
      self.__running = 0

   def documentComponentOpened(self, event):      
      return

   def documentComponentClosing(self, event):
      return

   def documentComponentClosed(self, event):
      self.stop()
      return

   def documentComponentMoving(self, event):
      return
      
   def documentComponentMoved(self, event):
      return

   def documentComponentActivated(self, event):
      return

   def documentComponentDeactivated(self, event):
      return

forwarder = Forwarder()
browser = HermesBrowser.getBrowser()
browser.addDocumentComponent(forwarder)
thread = Thread(forwarder)
thread.start()


[Colin]

Hi,

1. Your 1.13 problem. In your hermes.bat script, at the end change:

Code: Select all

start "HermesJMS" "%JAVA_HOME%\bin\javaw" -Xmx1024m -Dhermes.home="%HERMES_HOME%" %HERMES_OPTS% -Dlog4j.configuration="file:%HERMES_HOME%\bin\log4j.props" -Dsun.java2d.noddraw=true -Dhermes="%HERMES_CONFIG%\hermes-config.xml" -Dhermes.libs="%HERMES_LIBS%" hermes.browser.HermesBrowser

to

Code: Select all

"%JAVA_HOME%\bin\java" -Xmx1024m -Dhermes.home="%HERMES_HOME%" %HERMES_OPTS% -Dlog4j.configuration="file:%HERMES_HOME%\bin\log4j.props" -Dsun.java2d.noddraw=true -Dhermes="%HERMES_CONFIG%\hermes-config.xml" -Dhermes.libs="%HERMES_LIBS%" hermes.browser.HermesBrowser

and then launch Hermes from the command line. Maybe something funny is going on and the error ending up on stdout.

2. Access to the context

You're right, its not easy to find. I guess this is because I don't assume JNDI is in use and so have coded the internals to be very JNDI neutral. It is however lurking just below the surface so was a 5 minute change to expose it:

http://hermesjms.org/jira/browse/HJMS-63

There is a build including this at http://hermesjms.com/patches. Assuming we get to the bottom of (1) that is.

3. Your script

Its an odd thing you're trying to do there - the script will constantly browse a queue and publish whatever it sees to a topic before disconnecting and then doing the same thing again creating some noticeable load on WLS and most likely a ton of duplicate messages being published.

You may want to consider using a more ESB type thing that reads from this queue and forwards the messages to another queue (that the real consumer reads from) and a topic - i.e a tee or wire-tap. I've done this before and if you do it in batches of a few hundred milliseconds using the same session and transaction it performs okay and is reliable.

Mule or ServiceMix or just some java code wired with Spring would do the job nicely.

Hope this helps,

Regards,

Colin.

[sgsonter]

Hi Colin;

Thanks again for taking the time and trouble to reply promptly.

1. For Hermes 1.1.3, I changed the script to try a foreground start yesterday without success. The splash screen displays, and a 30 MB Java process is started, but it consumes zero CPU and never goes further. Under JDK 1.6, I'm able to start the java console and the message;

Netscape security model is no longer supported.
Please migrate to the Java 2 security model instead.

appears in the java console about half a dozen times.

2. Thanks for the context info. The penny has dropped - the context we get from the HermesBrowser is a context which allows me to lookup a JNDI context identified by its name within Hermes! Now that I understand that, it all makes sense. Thanks for pointing me to the explanation.

3. I certainly appreciate the problems with the forwarding approach. Your absolutely correct - now that I've made the changes, there are duplicate and missed messages, and the approach isn't really usable in a distributed environment.

I'm actually instrumenting a production BEA AquaLogic Service Bus (ALSB) infrastructure, so I have ESB infrastructure at hand! The problem with using a "tee" is not technical. I could certainly develop and deploy a "proxy service" which tees the source queue into a new destination queue and a monitoring topic , but I don't have authority to change the production system.
Programmatically, ALSB also provides a "report" function which allows me to tee the message to a dedicated reporting log which I could post-process.

I guess I'll have to get authority to make the changes

Thanks for your assistance. I've been using Hermes JMS for basic Unit testing for over a year, and am reallly impressed. Congratulations and many thanks for your contribution to adding really useful tools to the growing FOSS toolkit.

Simon

[Colin]

Hi Simon,

Thanks again for taking the time and trouble to reply promptly.

No problem.

1. For Hermes 1.1.3, I changed the script to try a foreground start yesterday without success. The splash screen displays, and a 30 MB Java process is started, but it consumes zero CPU and never goes further. Under JDK 1.6, I'm able to start the java console and the message;

Netscape security model is no longer supported.
Please migrate to the Java 2 security model instead.

appears in the java console about half a dozen times.

When using Java 1.5, are there no errors anywhere - hermes.log as well as console? If so could you create an issue?

Are you using JRocket? Googling for that security error shows some interesting but not exactly related issues. Could the error be due to one of the BEA jars not being signed correctly?

I'm actually instrumenting a production BEA AquaLogic Service Bus (ALSB) infrastructure, so I have ESB infrastructure at hand! The problem with using a "tee" is not technical. I could certainly develop and deploy a "proxy service" which tees the source queue into a new destination queue and a monitoring topic , but I don't have authority to change the production system.

This requirement is so common and yet many message brokers don't let you create this kind of tee easily, meaning we have to resort to an external process to do the work for us. The recent Camel additions to ActiveMQ (where you can embedd ESB stuff in process with the broker) and the Tibco EMS "bridge" functionality help create a tee if you were using those products. Double check the WLS docs, maybe there is something new in 10 that may help you...

Thanks for your assistance. I've been using Hermes JMS for basic Unit testing for over a year, and am reallly impressed. Congratulations and many thanks for your contribution to adding really useful tools to the growing FOSS toolkit.

Thanks!

Regards,

Colin.