Using KV with JSON

Elastic Stack Logstash
1

My app generates the log entries in the following format. Logstash Forwarder ships the logs to Logstash Indexer which in turn pushes the logs into Elasticsearch. I'm using LSF 0.4.0, Logstash 1.5.3, ES 1.7.1

{"@version":"1","@timestamp":"2015-08-23T17:37:33.235Z","thread":"098658","level":"INFO","marker":{"name":"LOGGABLE"},"message":"someId=PMCtuW9.6sKDC,someMoreId=blabla"}

filter {
if [type] == "myapp" {
json {
source => "message"
kv {
source => "myapp.%{[message]}"
}
}
}

I was looking to get the fields someId and someMoreIds indexed and searchable, and such. However, I have not been successful yet. I am thinking that my field referencing is incorrect.

I've tried kv with source => "message", source => "[message][message]" without success

The json portion works fine.

Thx

2

source => "message" is what you should use. While I don't think it's causing your problems, you also need to set field_split => "," since your key/value pairs aren't separated by spaces.

3

That works. I am getting the good stuff now, maybe too much, but that's a different story. Thx

4

As a reference in case it helps someone else, I changed to source => "message" and added the field_split => ","

5

Btw, the json filter is referencing "message" and that means the log entry?

{"@version":"1","@timestamp":"2015-08-23T17:37:33.235Z","thread":"098658","level":"INFO","marker":{"name":"LOGGABLE"},"message":"someId=PMCtuW9.6sKDC,someMoreId=blabla"}

or does "message" refers to the field "message" within the log entry?

Similarly, for the kv filter, does "message" refer to the log entry or the field message with in the log entry?

Thanks...

6

There is no concept for referring to the whole log entry, so message refers to the message field in both cases.

7

Well, my message field is "message":"someId=PMCtuW9.6sKDC,someMoreId=blabla"

So, I figure your reference "message" is probably my log entry below? That's what in my log. It gets stuffed in a logstash "message" field, I guess.