Ingest Node Index According to Timestamp

Elastic Stack Elasticsearch
1

I just upgraded to version 5. Currently i am using Filebeat to send loglines into logstash which ill in turn filter the data and pass the document into elasticsearch. Now, I am hoping to use the elasticsearch ingest node in place of logstash (i.e. filebeat -> ingest node).

I understand that by using the ingest nodes, i will have to specify the rules and patterns in the pipeline processors. However, i cant seem to find any documentation regarding what index will my document(s) be sent to.

Is it possible for me to define which index to store the document? I want to define it according to the value from a field (i.e. timestamp)

2

So filebeat will send data to filebeat-YYYY-MM-DD.

To do that it will use a ?pipeline=YOURPIPELINE http parameter.

The pipeline will index the document in the index this document is sent to. Here in filebeat-YYYY-MM-DD. Note that in a pipeline, you can change _index value which by default contains the index name where the document is supposed to be sent to.

Makes sense?

3

I don't understand what you mean exactly.

The thing is that the logfile can contain messages from different days (i.e. different timestamp). Previously in logstash i could name my index dynamically depending on the value of the fields it receives. Is that possible in this case?

4

I did not try it but I believe you can use https://www.elastic.co/guide/en/elasticsearch/reference/current/date-index-name-processor.html

1 Like
5

I just tried that out and that is exactly what i was looking for. Thank you very much!!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.