Hello,
Is there a way to check in logstash pipeline if the exists. I am trying to use elasticsearch filter but I am getting an error on the first document because the index is not yet created.
Regards,
Peter
What exactly are you trying to do? Examples would help.
Assuming the defaults, if the elasticsearch filter fails to lookup something, it adds a tag_on_failure. The filter stage is sequential, so just put in a conditional after that filter, if the tag exists, then create the index or do something else.... Hard to help with the limited info.
I have some data coming to logstash but instead of being daily it is a an updated data like previous day + current day and increasing every day. In order to get the curent date value I was thinking to actually search elasticsearch for the previous day data. But that does not work on the first document since there is no index at that time yet. And logstash is failing because of that reason. I was thinking to run that code only if the index exists. But I guess I can check for the new field and if not there I skip it.
I'm not sure I'm following fully. Perhaps you can be specific on what is being indexed? Unless its highly confidential...
What I'm understanding is, if we start today, you will have one day's worth of data (today's). Tomorrow, you'll have two days worth of data. Day after you'll have three days worth of data..... and so on until what point? If your data grows at that rate it becomes hard to manage...?
Also, I am not sure why you are saying "logsstash is failing". I can't imagine Logstash is actually failing. You are meaning the filter plugin fails and tags the documents as such?
This is all too vague to really be able to help you, try spending some time to explain in some detail...
Have you looked at setting the document _id and indexing into the same index (using ilm/rollovers).
ElasticSearch is trying to do away with separate indexes by day. You can create an index with an alias and then automatically rollover the indexes automatically. Check out Index Lifecycle Management. I think that's your best bet.
Nevermind checking if field exists worked. So I have a workaround.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.