Error: string not matched

Elastic Stack Logstash
1

Hi all
i have an error i cant understand
i have rabbitmq with a messege like this {"InternalMessageQueueSize":5,"ExceptionsNumber":300}
i like to take it with logstash

my input is :

input {
rabbitmq {
host => "Myhost"
port => 5672
ssl => false
user => "logstash"
password => "Mypassord"
queue => "Testq"
durable => true
codec => json
type => "app.metrics"
tags => "app.metrics"

    }

}

But in the logstash logs i see

{:timestamp=>"2015-09-20T12:55:46.339000+0300", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::RabbitMQ host=>"Myhost", user=>"logstash", queue=>"Testq", type=>"app.metrics", tags=>["app.metrics"], vhost=>"/", key=>"logstash">\n Error: string not matched", :level=>:error}

2

github.com/elastic/logstash issue #3641 looks similar and one thing you have in common with the reports in that issue is that you're all using the json codec on an input. Have you instead of using the json codec tried using the default codec and doing the JSON parsing via a json filter?

3

No because my application send it as json to rabbitmq
so you say to take it as plain text and parsing it filter?

4

Yes, at least as a way of figuring out what the problem might be. If doing so fixes the problem it's an indication that it's related to the json codec. I'd also crank up the log level by starting Logstash with --debug.

5

When i pull the messege from rabbitmq i see it with escape characters

2015-09-20T14:45:25.651Z %{host} "{"InternalMessageQueueSize":5,"ExceptionsNumber":300}"

so how can i filter it can you give me example

6

When i pull the messege from rabbitmq i see it with escape characters

2015-09-20T14:45:25.651Z %{host} "{"InternalMessageQueueSize":5,"ExceptionsNumber":300}"

And this is with the stdout output? For debugging I suggest you use the rubydebug codec so that you see the whole message and get unambiguous separation of all fields.

so how can i filter it can you give me example

filter {
  json {
    source => "message"
  }
}
7

Thanks i tryed to add the filter and debug now i get

←[33mTrouble parsing json {:source=>"message", :raw=>""{\"InternalMessageQueueSize\":5,\"ExceptionsNumber\":300}"", :exception=>#<TypeError: can't convert String into Hash>, :level=>:warn}←[0m
{
"message" => ""{\"InternalMessageQueueSize\":5,\"ExceptionsNumber\":300}"",
"@version" => "1",
"@timestamp" => "2015-09-21T07:42:28.411Z",
"tags" => [
[0] "_jsonparsefailure"
]
}

This is my conf

input {

    rabbitmq {
            host => "******"
            port => 5672
            ssl => false
            user => "logstash"
            password => "********"
            queue => "********"
            durable => true
			codec => "plain"
    }

}

filter {
json {
source => "message"
}
}

output {
stdout {
codec => rubydebug {
}
}
}

8

It looks like the messages posted to RabbitMQ isn't valid JSON; you're getting

"{\"InternalMessageQueueSize\":5,\"ExceptionsNumber\":300}"

instead of

{"InternalMessageQueueSize":5,"ExceptionsNumber":300}

Unless there's an additional filter that you're not showing us (a spurious file in /etc/logstash/conf.d perhaps?) I'd say the problem is with whatever software is posting messages to the queue.

9

Yes its looks like the post to rabbit but the guys says they use json library

any way i cat mutant the input to remove the "" and the start/end " and then set it as json?

10

Yes its looks like the post to rabbit but the guys says they use json library

You can easily use the RabbitMQ management interface to inspect exactly what's in the queue with no middle men.

any way i cat mutant the input to remove the "" and the start/end " and then set it as json?

You can use the mutate filter's gsub option to perform string substitution, but doing it correctly so that all input string possibilities are dealt with correctly isn't completely trivial.

11

when i get a messege from rabbitmq i can see it like this

"{"InternalMessageQueueSize":5,"ExceptionsNumber":300}"

So now i will try to remove the \ and " with gsub

12

Okay, but you're addressing the wrong problem. Whoever is posting messages to RabbitMQ should stop posting garbage.

13

Yes i know but we cant find it from his side