PoCo::MessageQueue version 0.2.0 released!

Submitted by David Snopek on April 25, 2008 - 6:35am

We reached a huge milestone this week, with the release of PoCo::MessageQueue version 0.2.0.  It is primarily a refactor engineered by Paul Driver.  Here is the change log:

  • Massive refactor:
    • Built on Moose.
    • Any storage engine can be a front-store.
    • Callback passing (versus old static method).
  • Storage::Complex:
    • Treat the front store like a cache.
    • Separate 'granularity' parameter.
  • Storage::Throttled:
    • Can claim and retrieve messages for improved performance.
  • Comprehensive unit-test suite.

For the future, Paul's already working on some interesting things:

  • Remote storage engines.
  • CouchDB store.
  • Clustered MQs.

Rock!

Comments

Hi, I'm experimenting with

Submitted by eskwayrd on May 27, 2008 - 3:18pm.

Hi,

I'm experimenting with PoCo::MessageQueue to see if it would help with a custom data processing framework implemented in Perl that currently exists, but is struggling with the current load.

Using minor variations of the sample producer and subscriber in the PoCo::MQ POD, I find that producers can run quite fast, sending 2K to 5K messages per second. However, using the defaults for mq.pl, a subscriber gets maybe 1-3 messages per second. Note that messages means 30 character strings.

If I set activemq.dispatchAsync to 0, and activemq.prefetchSize to 10, I can get the subscriber pulling messages at almost 10 per second. However, this is still quite slow.

Our current framework has some issues, but is able to handle up to 250 messages per second, with messages anywhere from 600 to 32K characters, before things start to fall over.

What kind of messaging performance are you getting from your subscribers with mq.pl?

@eskwayrd: If there are

Submitted by dsnopek on May 30, 2008 - 10:01am.

@eskwayrd: If there are subscribers available at the time the message is received, it will be delivered immediately, before it is even attempted to be stored. So, in theory, it should just be the network overhead of sending the message body from producer to MQ to subscriber.

PoCo::MQ doesn't support anything like activemq.prefetchSize, so if your subscriber is using an ack of "client", you will only be fed one message at a time, after each ack(). But if you set this instead to "auto", you don't have to ack() each message and new messages will be sent before you are done processing the last.

We get pretty good performance, however, the operations our subscribers perform generally take several seconds to complete anyway. But in your case, I suspect the problem is the large volume of incoming messages (you listed 2k to 5k messages per second). With that many messages coming in at once, the MQ is likely bogged down dealing with the traffic coming in and is unable to focus on getting messages back out.

There are a couple of things you can try: (1) ack => "auto", (2) sending fewer messages at one time (if possible, in production you could try having several MQs on seperate machines to share the work of marshalling all those messages).

If you've got anymore questions, please send them to the google group (pocomq@googlegroups.com) or come chat on irc.perl.org #poco_mq. Thanks!