How to set the id in elasticsearch

Elastic Stack Elasticsearch
1

helo,

How you set the id when sending files to elasticsearch? I'm using filebeat to send the logs, then to logstash then to elasticsearch. I'm using Kibana and the fields for _id show up but, I'm not sure how to change this. I'm also trying to use nifi to sends file to elasticsearch and it requires to know the id before it can successfully send to elasticsearch.

https://nifi.apache.org/docs.html the putelasticsearch processor. Identifier Attribute The name of the attribute containing the identifier for each FlowFile which is where the id part goes in.

2

You cannot do this with filebeat.
You would need to do it with Logstash and then set https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id yourself.

3

What if i'm using something else as the transport of data and not logstash? How do I do that?

4

Then pass in an _id when you send the doc.

5

how I do that? I been trying to google some info but, not easy to find.

6

That depends entirely on whatever you are pushing the data to ES with.

7

your document should have a top level '_id' field with the value you want.

8

Set the "Identifier Attribute" under Procces details for PutElasticsearchHttp. Please note that this should be flowfile attribute. Custom Flow file attributes can be added using Process UpdateAttribute.