Bad Outputs moving from RabbitMQ to Rackspace Queueing

(DKlotz) #1

After running into some frustrations with RabbitMQ's stability, I am attempting to move to Rackspace's Queuing plugin.

I have 2 servers, one which populates the queue, one which pulls from the queue.

I have replaced the rules in my logstash instances with the equivalent in the rackspace plugin. However, now when my consumer server gets the messages, they have a bunch of backslashes and no longer are parsed cleanly into tags. I can see they're correct on the producer side.
In example:
"message" => "2015-06-14 21:48:38 PUT /www/ - 443 \r",
"@version" => "1",
"@timestamp" => "2015-06-15T20:43:40.021Z",
"host" => "",
"type" => "iis"

becomes this on the consumer side:
"message" => "{"@version"=>"1", "host"=>"", "type"=>"iis", "message"=>"2015-06-14 21:48:38 PUT /www/ - 443 r", "@timestamp"=>"2015-06-16T12:23:16.764Z"}",
"@version" => "1",
"@timestamp" => "2015-06-16T12:29:28.722Z",

It looks like it takes the message and then adds the timestamp it pulled it from the queue on, and does nothing with the message.

I have tried using "plain" or "json" for the codec with no luck. Not sure what else to try to get it to parse properly, or remove the backslashes.

(DKlotz) #2

Based on testing. I realized that it does not actually encode into JSON before being sent.. I manually changed line:

  @rackspace_queue.messages.create :body => event, :ttl => @ttl

@rackspace_queue.messages.create :body => event.to_json, :ttl => @ttl

in the plugin.

since I was having trouble getting the @codec.encode(event) to work and my ruby isn't that good.

(Mark Walkom) #3

Nice find!

(system) #4