Unable to send json to elastic

Elastic Stack Logstash
9

Hello @Magnuss

Given your snippet:

[{
  "id": 4000828499889,
  "link": "https://www.aliexpress.com/item/4000828499889.html?algo_pvid=d6d52e6d-b14b-4967-ade3-9f13a4f660cc&algo_expid=d6d52e6d-b14b-4967-ade3-9f13a4f660cc-17&btsid=0ab6f83115866312995255394e208c&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_",
  "title": "Disney Pixar Cars 2  38 Style Metallic Finish Silver Chrome Lightning McQueen 1:55 Diecast Metal Toy Car Kids Gift",
  "tradeAmount": "2 orders",
  "averageStar": "0.0",

There are 2 problems:

  • the JSON file contains an ARRAY of JSON objects (it starts with a [)
  • Except if you've pretty printed the JSON content, the JSON objects span over multiple lines

If we can make the assumption that every file starts with [{ and ends with }] (with an ending newline), you can use the following pipeline.

input {
	file {
		path => "..."
		start_position => "beginning"
		codec => multiline {
		  pattern => "^}?\]"
		  negate => true
		  what => next
		}
	}
}
filter { 
    json {
		source => "message"
		target => "json"
		remove_field => "message"
	}
	split {
		field => "json"
	}
}
output {
	 stdout { codec => rubydebug }
}

In any case this is highly inefficient as Logstash will buffer all the log lines prior to sending them to the filter and split the events.
The best would be to have a file containing JSON lines, one per line, without leading [ and ].

Similar asked question:

1 Like