Logstash csv output filter

(Charlie) #1

I am querying my ES using Logstash.
I want to output the id of the record from ES into my csv file.

"_index": "data",
"_type": "default",
"_id": "vANfNGYB9XD0VZRJUFfy",
"_version": 1,
"_score": null,
"_source": {

output { csv {
fields => [ "field1", "field2", "field3", "_id"]
path => "/tmp/export.%{+YYYY-MM-dd-hh}.csv"

but the _id is never logged to my csv. All other fields are there as expected, but the special metadata _id field is not there.
How can I get it there? I do not want to edit 60+ pipelines I have just to clone one field into another.

(Charlie) #2

Found solution:

When we query ES we have to enable docinfo => true. By default it is false.

input {
 elasticsearch {
  hosts => [ your hosts ]
  index => "ti"
  query => '{your query}'
  size => 1000
  scroll => "1s"
  docinfo => true
  schedule => "14 * * * *"

We can than access the metadata with syntax:

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.