Logstash Plugin es_bulk - Need a working exampleof bulk indexing


(Marc) #1

We are currently posting multiple events to Elastic Search using the /_bulk endpoint in the format:

{"index":{"_type" : "doc", "_id": "1"}}
{"category":"DATA","raw": "message 1"}
{"index":{"_type" : "doc", "_id": "2"}}
{"category":"DATA","raw": "message 2"}
...

We want to move to using LogStash so we looked at the es_bulk plugin but cannot get it working and cannot find any examples documented nor in this forum.

input {
	http {
		port => "5051"
		codec => "es_bulk"
	}
}
output {
	stdout {
		codec => rubydebug { metadata => false }
	}
}

POSTing the above to port 5051 Produces the output:
{
"index" => {
"_type" => "doc"
},
"headers": {...
...

So, obviously it's only picking up the first line of what we POST and not, as I would expect, treating each pair of lines as a single event with the actual message in every second line.

Has anyone got an example or better documentation than this?