How to set the id in elasticsearch


#1

helo,

How you set the id when sending files to elasticsearch? I'm using filebeat to send the logs, then to logstash then to elasticsearch. I'm using Kibana and the fields for _id show up but, I'm not sure how to change this. I'm also trying to use nifi to sends file to elasticsearch and it requires to know the id before it can successfully send to elasticsearch.

https://nifi.apache.org/docs.html the putelasticsearch processor. Identifier Attribute The name of the attribute containing the identifier for each FlowFile which is where the id part goes in.


(Mark Walkom) #2

You cannot do this with filebeat.
You would need to do it with Logstash and then set https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id yourself.


#3

What if i'm using something else as the transport of data and not logstash? How do I do that?


(Mark Walkom) #4

Then pass in an _id when you send the doc.


#5

how I do that? I been trying to google some info but, not easy to find.


(Mark Walkom) #6

That depends entirely on whatever you are pushing the data to ES with.


#7

your document should have a top level '_id' field with the value you want.


(Subodh Chettri) #8

Set the "Identifier Attribute" under Procces details for PutElasticsearchHttp. Please note that this should be flowfile attribute. Custom Flow file attributes can be added using Process UpdateAttribute.


(system) #9