Logstash Plugin es_bulk - Need a working exampleof bulk indexing

Elastic Stack Logstash
1

We are currently posting multiple events to Elastic Search using the /_bulk endpoint in the format:

{"index":{"_type" : "doc", "_id": "1"}}
{"category":"DATA","raw": "message 1"}
{"index":{"_type" : "doc", "_id": "2"}}
{"category":"DATA","raw": "message 2"}
...

We want to move to using LogStash so we looked at the es_bulk plugin but cannot get it working and cannot find any examples documented nor in this forum.

input {
	http {
		port => "5051"
		codec => "es_bulk"
	}
}
output {
	stdout {
		codec => rubydebug { metadata => false }
	}
}

POSTing the above to port 5051 Produces the output:

{
    "index" => {
        "_type" => "doc"
    },
    "headers": {
        ...
    }
}

So, only one event (which doesn't contain the fields category or raw) is generated thus it's only picking up the first line of what we POST and not, as I would expect, treating each pair of lines as a single event with the actual message in every second line.

Has anyone got an example or better documentation than this?

2

Bump

3

What kind of processing are you looking to perform? Where are you sending data? Have you considered using an ingest node pipeline instead?

4

OK, the solution seems to be adding the following header:

Content-Type: application/x-ndjson

This results in the content being properly split into different events:

{
	"headers" => {
	...
	},
	"raw" => "message 1",
	"@timestamp" => 2018-12-17T06:53:22.912Z,
	"category" => "DATA"
}
{
	"headers" => {
	...
	},
   "raw" => "message 2",
	"@timestamp" => 2018-12-17T06:53:22.912Z,
	"category" => "DATA"
}

I got this from the ES _bulk documentation. Interestingly ES doesn't care about the Content-Type header but Logstash does.

5

Which version of the stack are you using? Content type verification has been tightened in recent versions...

6

As I said, until now we have been directly submitting documents to ES and want to move to LogStash so all input comes via LogStash.

Do you have any idea for how we can submit changes/corrections to the ES docs?

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-es_bulk.html

7

We're on 6.5 across the board and we're seeing the _bulk/ API accepts Content-Type: application/json.

8

We're also seeing that ES won't accept application/x-ndjson; charset=utf-8 as a Content-Type header though it does accept application/x-ndjson.

{"error":"Content-Type header [application/x-ndjson; charset=utf-8] is not supported","status":406}

The issue has already been reported.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.