Comma destroys perfect events | file { codec => multiline { pattern

Elastic Stack Logstash
1

I want to index a json file. I use input file, with codec multiline. My filter is json filter, output is elasticsearch.
The structure of my json file looks like this:

[
 { 
   "Plugin": 92438,
   "Plugin Name": "WordPad History",
   "Family": "Windows",
   "Severity": "Info",
   "Total": 1
 },
 {
   "Plugin": 92439,
   "Plugin Name": "Explorer Search History",
   "Family": "Windows",
   "Severity": "Info",
   "Total": 1
 },
 {
   "Plugin": 53360,
   "Plugin Name": "SSL Server Accepts Weak Diffie-Hellman Keys",
   "Family": "General",
   "Severity": "Info",
   "Total": 1
 }
]

My logstash.conf:

input {
  file {
    path => "/home/vagrant/vuln.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "{"    
      negate => "true"
      what => "previous"
      auto_flush_interval => 1
    }
  }
}

filter {
  mutate {
    gsub => ["message", ",$",""]
   }
   json {
      source => "message"
    }
}

output {
  elasticsearch {
    index => "new-index"
    hosts => "127.0.0.1"
  }
}

My output is nearly perfect but just nearly. I dont understand the json filter....I get 3 events. First 2 events are different to the 3rd. The first 2 events write into the message field for example:
message:

 { 
   "Plugin": 92439, 
   "Plugin Name": "Explorer Search History", 
   "Family": "Windows", 
   "Severity": "Info", 
   "Total": 1 
 },

The last event has no comma at the end of the message and it indexes perfectly. It automatically makes new field, for example Plugin: 53360, Plugin Name: "SSL Server Accepts Weak Diffie-Hellman Keys", Severity: "Info". This is a valid json file, why can't it build new field also if i there are commas behind the message? I get _jsonparsefailure in the first 2 events, where no new fields are built, in the last event I don't get _jsonparsefailure...

message :

 {
   "Plugin": 53360,
   "Plugin Name": "SSL Server Accepts Weak Diffie-Hellman Keys",
   "Family": "General",
   "Severity": "Info",
   "Total": 1
 }

What i need to do, to get new fields if there are commas at the end of the message? Is it possible
mutate { gsub 0> and json { } don't work together?

Best regards

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.