Message Queueing - by an MQ noob

Message Queueing from an MQ noobs perspective

Preamble• I work for a global telco• We push telco events and messages around• All should be processed quickly and reliably• Some generate a response

Scenario• We send messages around in small text ﬁles, sometimes with RFC 2822 headers• Use the UNIX sendf/recﬁled mechanism

Problem• Its entirely intolerant to faults• ... and network latency

Our Solution• Add in our own "message queueing"• ... and some facilities to decouple delays

Shortcomings• Messages cannot be sent to multiple hosts• Target hosts are not easily reconﬁgurable• Its developed in-house and has issues

Scenario• multiple producers• a consumer and its backup• Systems separated by 100-400ms WAN • with occasional 100% packet loss

The Contenders

pymq• very hazy on the details• depends on MySQL? and Django?!?• ignored while looking at kombu / rabbitmq

ZeroMQ aka 0MQ"ZeroMQ is a message orientated IPC Library." - commenter on stackoverﬂow

AMQP• Is a standard for message queueing• Producers submit messages to brokers• Brokers consist of exchanges and queues• Exchanges route the messages• Queues store messages• Consumers pull messages out

RabbitMQ

RabbitMQProducer Broker Consumer

RabbitMQProducer Broker Exchange Queue Consumer

RabbitMQQueues

RabbitMQExchanges

RabbitMQ

RabbitMQExchanges

RabbitMQ Routing

RabbitMQExchange Types

RabbitMQDirect Exchanges

RabbitMQTopic Exchanges

RabbitMQFanout Exchanges

RabbitMQPersistence

RabbitMQBroker Persistence

RabbitMQMessage Persistence

RabbitMQ1. Mark the exchange “durable”.2. Mark the queue “durable”.3. Set the message’s “delivery mode” to a value of 2

RabbitMQPersistence Warnings

RabbitMQAcknowledging

My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer

My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer Consumer

My Desired Setup Remote Host Processing Server Publisher Consumer ebs Interw Queue

ShovelRemote Host Processing Server Publisher Queue bs Queue rwe Consumer Inte Shovel

=ERROR REPORT==== 7-Aug-2012::16:50:32 ===** Generic server <0.366.0> terminating** Last message in was {$gen_cast,init}** When Server state == {state,undefined,undefined,undefined,undefined, hello_shovel, {shovel, {endpoint, [{amqp_params_direct,<<"guest">>,<<"/">>, rabbit@localhost,none,[]}], []}, {endpoint, [{amqp_params_network,<<"guest">>,<<"guest">>, <<>>,"hardrock",undefined,0,0,0,infinity,none, [#Fun<amqp_auth_mechanisms.plain.3>, #Fun<amqp_auth_mechanisms.amqplain.3>], [],[]}], []}, 0,on_confirm,#Fun<rabbit_shovel_sup.16.91344538>, #Fun<rabbit_shovel_sup.16.91344538>,<<"hello">>,5}, undefined,undefined,undefined,undefined,undefined}** Reason for termination ==** {{badmatch,{error,access_refused}}, [{rabbit_shovel_worker,make_conn_and_chan,1,[]}, {rabbit_shovel_worker,handle_cast,2,[]}, {gen_server2,handle_msg,2,[]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

error in config file "/usr/local/etc/rabbitmq/rabbitmq.config" (37): syntax errorbefore: .

Shovel[ {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} ]} , {destinations, [ {brokers, ["amqp://server"]} ]} , {queue, <<"hello">>} ]} ]} ]}].

[ Shovel {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} , {declarations, [ {exchange.declare, [ {exchange, <<"foo">>} , {type, <<"direct">>} , durable ]} , {queue.declare, [ {queue, <<"hello">>} ]} , {queue.bind, [ {exchange, <<"foo">>} , {queue, <<"hello">>} ]} ]} ]} , {destinations, [ {brokers, ["amqp://hardrock"]} ]} , {queue, <<"hello">>} ]} ]} ]}].

Installation• 1. install RabbitMQ • (install homebrew; brew install rabbitmq; rabbitmq-server [-detached]) • (yum install rabbitmq-server.noarch) • rabbitmqctl or http://server:55672/• 2. pip install kombu, or pika, or puka, or ...

pika"a pure-Python implementation of the AMQP 0-9-1 protocol"

Publisherimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)channel.basic_publish(exchange=, routing_key=hello, body=Hello World!, properties=pika.BasicProperties(delivery_mode=2))print " [x] Sent Hello World!"connection.close() Publisher Queue

Consumerimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(localhost))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello, no_ack=True)channel.start_consuming() Queue Consumer

Remote Consumer Local Publisher

Remote queuesimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello, no_ack=True)channel.start_consuming() Host 1 Host 2 Publisher Queue Consumer Consumer

Remote Consumer Local Consumer Local Publisher

Acknowledgementimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,) raw_input(Press enter to ACK) ch.basic_ack(delivery_tag=method.delivery_tag)channel.basic_consume(callback, queue=hello, no_ack=False)channel.start_consuming()

Consumer Publisher

Issues• pikas BlockingConnection (my use case is simple enough) hard-codes the socket timeout and fails to cope with latency >1s• Fails to cope at all with packet loss

Deleting a Queueimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_delete(queue=hello)channel.close()connection.close()

Kombu"a messaging framework for Python"

Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"

Bizarro errorsamqplib.client_0_8.exceptions.AMQPChannelException: (406,u"PRECONDITION_FAILED - parameters for queue hello in vhost /not equivalent", (50, 10), Channel.queue_declare)

Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost//) as conn: with conn.SimpleQueue(hello, queue_opts=dict(durable=False)) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"

Puka"An opinionated RabbitMQ client"

Publisherimport pukaclient = puka.Client(amqp://localhost//)client.wait(client.connect())client.wait(client.queue_declare(queue=hello, durable=True))promise = client.basic_publish(exchange=, routing_key=hello, body=Hello World!)client.wait(promise)print " [x] Sent Hello World!"client.wait(client.close())

Other Plugins• Management (web interface, quite nice)• Federation (Shovel, but with MOAR)• STOMP protocol support

STOMPSimple (or Streaming) Text Orientated Messaging Protocol

STOMP "simple"

STOMP{rabbitmq_stomp, [ {tcp_listeners, [ {"127.0.0.1", 61613} , {"::1", 61613} ]} , {default_user, [ {login, "guest"} , {passcode, "guest"} ]}]}

stompclientfrom stompclient import PublishClientclient = PublishClient(localhost, 61613)client.connect()client.send(/queue/hello, Hello, world!)client.disconnect()

stompclientfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!" from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()

stompclient from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()from stompclient import PublishClientwith PublishClient(localhost, 61613) as client: client.send(/queue/hello, Hello, world!)

Celery"a synchronous or asynchronous task queue/job queue based on distributed message passing"

TcpCatcher• TCP, SOCKS, HTTP(S) proxy & monitor• Can introduce latency and transmission errors• Understands HTTP and images• Can debug/interfere/log SSL trafﬁc• Free: www.tcpcatcher.fr

Feb	MAR	Apr
	31
2012	2013	2014

Message Queueing - by an MQ noob

by Richard Jones on Oct 01, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Message Queueing - by an MQ noob Presentation Transcript