Logstash - split array into individual events

Elastic Stack Logstash
1

Hi!
One of our logs contains lines of arrays of json objects.... I want to take each one of the items in the array and make it a single even.
I read some posts on how to do it :
https://discuss.elastic.co/t/split-nested-json-array/147969
https://stackoverflow.com/questions/30558535/logstash-how-do-i-split-an-array-using-the-split-filter-without-a-target

But I was not able to make mine work....
Here is what a line looks like for me:
[{"timeStarted": "2019-05-13T09:03:34.995Z","operations": {"operation 1" { ... } "operation 2" { ... }} },{"timeStarted": "2019-05-13T09:03:35.000Z","operations": {"operation 1" { ... } "operation 2" { ... }} ]

and this is what I tried with my conf:

filter {	
		split {
			field => "message"
			target => "events"
			add_field => { 
				"operations" => "%{operations}" 
				"timeStarted"  => "%{timeStarted}" 
			}
		}
		
		mutate {
			add_field => { 
				"site" => "%{path}"
				#"operations" => "%{[events][operations]" 
				#"timeStarted"  => "%{[events][timeStarted]" 
			}
			
			remove_field => ["[message]"]
		}
	

		date {
			match => ["[timeStarted]", "yyyy-MM-dd'T'HH:mm:ss.SSS"]  
		}

}

I tried different options, but I could not get the fields out. I only get 1 event per line and get parse error for the fields...
I tried adding the fields both as part of the split and as part of a mutate, neither of them worked for me
any idea?
Thanks!!!

2

If that is the content of [message] then

input { generator { count => 1 lines => [ '[{"timeStarted": "2019-05-13T09:03:34.995Z","operations": {"operation 1": 1, "operation 2": 2 } },{"timeStarted": "2019-05-13T09:03:35.000Z","operations": {"operation 1": 1, "operation 2": 2 } } ]' ] } }
filter {
    json { source => "message" target => "someField" }
    split { field => "someField" }
}

will work. Note that since it is an array the target option on the json filter is required.

3

What the first line means (input generator)?
is this just to test instead of the file input we use?

4

Yes.

5

Thanks
can you explain - is this what you meant or a typo -

filter {
json { source => "message" target => "someField" }
split { field => "someField" }
}

In both the output is "someField" yet the split input is field which is undefined... did you mean that the json target is field or someField?

6

This tells the split filter to operate upon a field called "someField". That name has to match the target of the json filter.

7

Thanks! it worked!!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.