Logstash csv output filter


(Charlie) #1

Hello.
I am querying my ES using Logstash.
I want to output the id of the record from ES into my csv file.

"_index": "data",
"_type": "default",
"_id": "vANfNGYB9XD0VZRJUFfy",
"_version": 1,
"_score": null,
"_source": {
....

output { csv {
fields => [ "field1", "field2", "field3", "_id"]
path => "/tmp/export.%{+YYYY-MM-dd-hh}.csv"
}
}

but the _id is never logged to my csv. All other fields are there as expected, but the special metadata _id field is not there.
How can I get it there? I do not want to edit 60+ pipelines I have just to clone one field into another.


(Charlie) #2

Found solution:

When we query ES we have to enable docinfo => true. By default it is false.

input {
 elasticsearch {
  hosts => [ your hosts ]
  index => "ti"
  query => '{your query}'
  size => 1000
  scroll => "1s"
  docinfo => true
  schedule => "14 * * * *"
 }
}

We can than access the metadata with syntax:
"[@metadata][_id]",


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.