Selective indexing of fields


(Arvind Sharma) #1

Hi All ,

I have a specific problem related to json parsing and extracting particular field for indexing

My sample json message is

{"log_time":"1/Sep/2015:12:35:50 +05:30","level":"INFO","log_message":"test message","description":"{"field1":"some data","field2":"some data "}","url":"some url","module":"some module","log_host":"localhost"}

description field is a string which itself is json
I want to extract only first 2 fields from description and add them as separate field in the even .

So my final output should be like

{
"message" => "{"log_time":"1/Sep/2015:12:35:50 +05:30","level":"INFO","log_message":"test message","description":"{"field2":"some data","field2":"some data "}","url":"some url","module":"some module","log_host":"localhost"}",
"@version" => "1",
"@timestamp" => "2015-09-03T11:20:43.881Z",
"host" => "localhost",
"log_time" => "1/Sep/2015:12:35:50 +05:30",
"level" => "INFO",
"log_message" => "test message",
"description" => ""description":"{"field2":"some data","field2":"some data"}"",
"url" => "some url",
"module" => "some module",
"log_host" => "localhost",
"field1" => "some data",
"field2" => "some data"
}

Note : description is having dynamic number of fields , it can be 2 or more .

Thanks & Regards
Arvind
Software Engg.
Shopclues.com


(Magnus B├Ąck) #2

Use the json filter to parse the description field into a subfield, move (rename) the fields you're interested in to wherever you want them, then delete the subfield.

json {
  source => "description"
  target => "description_json"
}
mutate {
  rename => {
    "[description_json][field1]" => "field1"
    "[description_json][field2]" => "field2"
  }
  remove_field => ["description_json"]
}

Not sure what you mean by "first two fields". You don't actually mean "first two" as in the first two fields listed in the string?


(Arvind Sharma) #3

Thanks @magnusbaeck , I was working on this from morning and you made my day :smile:

Thanks a lot


(system) #4