Using KV with JSON


(coder lol) #1

My app generates the log entries in the following format. Logstash Forwarder ships the logs to Logstash Indexer which in turn pushes the logs into Elasticsearch. I'm using LSF 0.4.0, Logstash 1.5.3, ES 1.7.1

{"@version":"1","@timestamp":"2015-08-23T17:37:33.235Z","thread":"098658","level":"INFO","marker":{"name":"LOGGABLE"},"message":"someId=PMCtuW9.6sKDC,someMoreId=blabla"}

filter {
if [type] == "myapp" {
json {
source => "message"
kv {
source => "myapp.%{[message]}"
}
}
}

I was looking to get the fields someId and someMoreIds indexed and searchable, and such. However, I have not been successful yet. I am thinking that my field referencing is incorrect.

I've tried kv with source => "message", source => "[message][message]" without success

The json portion works fine.

Thx


(Magnus Bäck) #2

source => "message" is what you should use. While I don't think it's causing your problems, you also need to set field_split => "," since your key/value pairs aren't separated by spaces.


(coder lol) #3

That works. I am getting the good stuff now, maybe too much, but that's a different story. Thx


(coder lol) #4

As a reference in case it helps someone else, I changed to source => "message" and added the field_split => ","


(coder lol) #5

Btw, the json filter is referencing "message" and that means the log entry?

{"@version":"1","@timestamp":"2015-08-23T17:37:33.235Z","thread":"098658","level":"INFO","marker":{"name":"LOGGABLE"},"message":"someId=PMCtuW9.6sKDC,someMoreId=blabla"}

or does "message" refers to the field "message" within the log entry?

Similarly, for the kv filter, does "message" refer to the log entry or the field message with in the log entry?

Thanks...


(Magnus Bäck) #6

There is no concept for referring to the whole log entry, so message refers to the message field in both cases.


(coder lol) #7

Well, my message field is "message":"someId=PMCtuW9.6sKDC,someMoreId=blabla"

So, I figure your reference "message" is probably my log entry below? That's what in my log. It gets stuffed in a logstash "message" field, I guess.


(system) #8