Overwrite an index in logstash

Elastic Stack Logstash
1

Hi All,

I am collecting alarms status from an oracle database every 5 minutes using logstash and load them into an index, the thing is that the remote DB will contain alarms only if the alarms are still open and I would like to have same behavior in my index. I was thinking to have a solution that will recrate the index each time I am collecting the data so that my index will contain only alams collected during the last 5 minutes.
Is there any idea how to do this in logstash ?

Thanks & Regards,

Hamid Allaoui

2

Yes with post process... with elastic input

3

Hi Joao,

Thank you for your reply, could you please share the logstash config for this.
Also I would like to know if this is the best way to fulfill my requirement which is to have in my index only the active alarms (the ones collected since last 5 minutes).

Thanks for your support.

Hamid

4

Hello Hamid,

I would like to propose another way of action if the table contains an ID:
You could create a history table in the database to reflect all changes done to your table(create trigger on source table for insert, update, delete and write changes to the new table including the action name).
Then you could rework your pipeline to read from the history table instead of the original one.
In the elasticsearch output there is a field called action where you can insert your action name:

  • insert -> create
  • update -> update
  • delete -> delete

Best regards
Wolfram

5

Hi Wolfram,

Thanks so much for your suggestion.
For info I did not need to have the history of alarms. I have an application reading Elasticsearch index and this application should display alarms whenever there are entries in that index, otherwise no alarm. This index should be the copy of the remote Oracle table which should contain data only if there are alarms otherwise the table will be empty.
So my solution was to schedule logstash collection every 5 min and if there are data in remote table meaning that alarms are there, the data will be indexed to my local index and for next run the logstash will empty the index before indexing new data if there is.
In conclusion we dont need the history data but only the active alarms.
I was thinking to another solution which to collect data every 5 minutes without empty my index then create an alias on top of this index using a filter to only show newest data got during last 5 minutes then the application should read this alias.
I would like to get opinions from community to see if there is better way to fulfill this requirement.

Thanks & Regards,

Hamid

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.