JSON parse - not seeing the newlines


(Vedran Maricevic) #1

Using ELK 5.0.0.
I am trying to parse simple JSON document:

{"timestamp":"2012-01-01 02:00:01", "severity":"ERROR", "messages":"Foo failed", "fieldone": "this if the value of a field one", "fieldtwo": "ttthis if the value of a field two"}
{"timestamp":"2013-01-01 02:04:02", "severity":"INFO", "messages":"Bar was successful", "fieldone": "this if the value of a field one", "fieldtwo": "this if the value of a field two"}
{"timestamp":"2017-01-01 02:10:12", "severity":"DEBUG", "messages":"Baz was notified", "fieldone": "this if the value of a field one", "fieldtwo": "this if the value of a field two"}

Inside the filter section I use this:

if [@metadata][type] == "jsonindex" {
json {
source => "message"
}
}

Problem is that I am getting only last JSON entry (Third line) in the ElasticSearch. Looks like it is not seeing the newlines properly.
Am I doing something wrong?


(Vedran Maricevic) #2

I have used
stdout { codec => rubydebug } to see what is going on. It seems that Logstash is emitting three events, but only the last one is written in the ES.

This is the input section on the Logstash:

input {
beats {
port => "5043"
codec => json
}
}

Ane here is the output from the Logstash:

{
"severity" => "DEBUG",
"offset" => 544,
"@uuid" => "a316bb67-98e5-4551-8243-f8538023cfd9",
"input_type" => "log",
"source" => "/Users/mar0004v/Downloads/elk/small/jsontest2.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"@timestamp" => 2017-05-08T11:25:41.586Z,
"@version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Baz was notified",
"timestamp" => "2017-01-01 02:10:12"
}
{
"severity" => "INFO",
"offset" => 361,
"@uuid" => "6d4b4401-a440-4894-b0de-84c97fc4eaf5",
"input_type" => "log",
"source" => "/Users/mar0004v/Downloads/elk/small/jsontest2.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"@timestamp" => 2017-05-08T11:25:41.586Z,
"@version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Bar was successful",
"timestamp" => "2013-01-01 02:04:02"
}
{
"severity" => "ERROR",
"offset" => 177,
"@uuid" => "d9bd0a0b-0021-48fd-8d9e-d6f82cd1e506",
"input_type" => "log",
"source" => "/Users/mar0004v/Downloads/elk/small/jsontest2.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"@timestamp" => 2017-05-08T11:25:41.586Z,
"@version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Foo failed",
"timestamp" => "2012-01-01 02:00:01"
}


(Vedran Maricevic) #3

What is interesting, it randomly writes to the ES. It is always one event, but it can be 1,2 or 3.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.