Logstash and JSON array split [SOLVED]

Elastic Stack Logstash
1

Hello,

I have the following input in JSON file:
{"jobs":[
{"name":"Onboarding_Build","builds":[{"duration":127431,"number":2480,"timestamp":1477608643723},{"duration":132275,"number":2479,"timestamp":1477605046524},{"duration":128311,"number":2478,"timestamp":1477603244388},{"duration":131157,"number":2477,"timestamp":1477601742760},{"duration":130942,"number":2476,"timestamp":1477594241970},{"duration":131681,"number":2475,"timestamp":1477592443221},{"duration":130358,"number":2474,"timestamp":1477591845485},{"duration":134905,"number":2473,"timestamp":1477590943481},{"duration":125540,"number":2472,"timestamp":1477583442267},{"duration":126236,"number":2471,"timestamp":1477581042801}]},

{"name":"Onboarding_Deploy_E1DEV4","builds":[{"duration":792391,"number":2255,"timestamp":1477608783769},
....

The actual config create one event in elasticsearch.

I am looking for creating multiple event based on this array:
"builds":[{"duration":792391,"number":2255,"timestamp":1477608783769},

How to split the JSON line in order to have one event per build AND use the timestamp value as the @datetime field ?

Thanks by advance.

BR

Emmanuel

2

you can use two split filters : https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html

first on the "jobs" field to create 1 event per job, then another split on builds to create 1 event per build (per job)

3

Thanks, I try to use split but no succeed for the moment.
There is only one job array per JSON file then couple name/build with build that is an array:

{"jobs":[
{"name":"Onboarding_Build","builds":[
{"duration":127431,"number":2480,"timestamp":1477608643723}, {"duration":132275,"number":2479,"timestamp":1477605046524},{"duration":128311,"number":2478,"timestamp":1477603244388},{"duration":131157,"number":2477,"timestamp":1477601742760},{"duration":130942,"number":2476,"timestamp":1477594241970},{"duration":131681,"number":2475,"timestamp":1477592443221},{"duration":130358,"number":2474,"timestamp":1477591845485},{"duration":134905,"number":2473,"timestamp":1477590943481},{"duration":125540,"number":2472,"timestamp":1477583442267},{"duration":126236,"number":2471,"timestamp":1477581042801}
]},

{"name":"Onboarding_Deploy_E1DEV4","builds":[{"duration":792391,"number":2255,"timestamp":1477608783769},{"duration":752825,"number":2254,"timestamp":1477605191572},{"duration":773871,"number":2253,"timestamp":1477603384440},{"duration":748282,"number":2252,"timestamp":1477601885590},{"duration":721893,"number":2251,"timestamp":1477594382020},{"duration":761717,"number":2250,"timestamp":147759

I try to have the following:

{"jobs":[
{"name":"Onboarding_Build","builds":[
{"duration":127431,"number":2480,"timestamp":1477608643723}
]}]}

{"jobs":[
{"name":"Onboarding_Build","builds":[
{"duration":127431,"number":2480,"timestamp":1477608643723}
]}]}

or just

{[
{"name":"Onboarding_Build","builds":[
{"duration":127431,"number":2480,"timestamp":1477608643723}
]}]}
...

And be able to assign the builds.timestamp value as @datetime

My filter config:

filter {
#json{
#source => "message"
#add_field =>
#{
# "kname" => "%{job.name}"
# "kbuilds" => "%{job[builds]}"
#}
#target => "kmsg"
#}
split {
field => "builds"
}
}

I did not create one event per job because there is only one but maybe I have to even if there is only one cause the the job target presence ?

4

The split parameteris not working. Even when using jobs or builds value.

5

you need:

filter {
  split { field => "[jobs]" }
  split { field => "[jobs][builds]" }
}
1 Like
6

Thanks,

I got this message unfortunately:

....
"type" => "json",
"tags" => [
[0] "_split_type_failure"
]
}

and no data injected

7

My full config file:

input {
file {
path => "/tmp/CSV/data.json"
type => "json"
#codec => "json"
sincedb_path => "/dev/null"
start_position => "beginning"
}

}

filter {
split { field => "jobs" }
split { field => "[jobs][builds]" }
#date {
# match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z” ]
# remove_field => "timestamp"
#}
}

output {
elasticsearch {
action => "index"
hosts => "localhost"
#index => "logstash-%{+YYYY.MM.dd}"
index => "json"
workers => 1
}
stdout {
codec => rubydebug
}
}

8

I see this in logstash-plain.log

[2016-11-05T07:47:35,539][WARN ][logstash.filters.split ] Only String and Array types are splittable. field:[jobs] is of type = NilClass
[2016-11-05T07:47:35,545][WARN ][logstash.filters.split ] Only String and Array types are splittable. field:[jobs][builds] is of type = NilClass

Emmanuel

1 Like
9

Ok now it seems to be better but still no data injected :

input {
file {
path => "/tmp/CSV/data.json"
type => "json"
codec => "json"
sincedb_path => "/dev/null"
start_position => "beginning"
}

}

filter {
split { field => "[jobs]" }
split { field => "[jobs][builds]" }

}

output {
elasticsearch {
action => "index"
hosts => "localhost"
#index => "logstash-%{+YYYY.MM.dd}"
index => "index_json"
workers => 1
}
stdout {
codec => rubydebug
}

Output

Sending Logstash logs to /var/log/logstash/ which is now configured via log4j2.properties.
{
"path" => "/tmp/CSV/data.json",
"@timestamp" => 2016-11-05T07:25:24.323Z,
"jobs" => {
"name" => "Onboarding_Build",
"builds" => {
"duration" => 127431,
"number" => 2480,
"timestamp" => 1477608643723
}
},
"@version" => "1",
"host" => "ks3309573.kimsufi.com",
"type" => "json"
}
{
"path" => "/tmp/CSV/data.json",
"@timestamp" => 2016-11-05T07:25:24.323Z,
"jobs" => {
"name" => "Onboarding_Build",
"builds" => {
"duration" => 132275,
"number" => 2479,
"timestamp" => 1477605046524
}
},
"@version" => "1",
"host" => "ks3309573.kimsufi.com",
"type" => "json"
}
{
"path" => "/tmp/CSV/data.json",
"@timestamp" => 2016-11-05T07:25:24.323Z,
"jobs" => {
"name" => "Onboarding_Build",
"builds" => {
"duration" => 128311,
"number" => 2478,
"timestamp" => 1477603244388
}
},
"@version" => "1",
"host" => "ks3309573.kimsufi.com",
"type" => "json"
}
{
"path" => "/tmp/CSV/data.json",
"@timestamp" => 2016-11-05T07:25:24.323Z,
"jobs" => {
"name" => "Onboarding_Build",
"builds" => {
"duration" => 131157,
"number" => 2477,
"timestamp" => 1477601742760
}
},
"@version" => "1",
"host" => "ks3309573.kimsufi.com",
"type" => "json"
}

Emmanuel

10

Ok I succeed ti make it working using the following config:

input {
file {
path => "/tmp/CSV/data.json"
type => "json"
codec => "json"
sincedb_path => "/dev/null"
start_position => "beginning"
}

}

filter {
split { field => "[jobs]" }
split { field => "[jobs][builds]" }

}

output {
elasticsearch {
action => "index"
hosts => "localhost"
#index => "logstash-%{+YYYY.MM.dd}"
index => "json"
workers => 1
}
stdout {
codec => rubydebug
}
}

Thanks Jose

11

I got it:

date {
match => ["[jobs][builds][timestamp]", "UNIX_MS"]
target => "@timestamp"
remove_field => "timestamp"
}