Indexing for only updated data using logstash

Elastic Stack Logstash
1

Hello,

Using jdbc input plugin in logstash, I have fetched all the data which I have wanted the most from database tables.

Now I want only updated data from all that db tables. Every time I don’t want to index whole data, rather I want to index only updated data from db tables and add that indexes to old index.

Is this possible using elastic search? And If yes kindly guide me how we can achieve this?

Thanks,
Priyanka

2

Not in elasticsearch but in logstash if you have a last modified timestamp in your data you can use that to just get the newest data.

1 Like
3

Hello @dadoonet ,

Thanks for quick reply!!!

I have last modified timestamp column in our database for some tables.

So currently I have fetched data from more than 5 tables and for each sql query there is a separate SQL file. So every time I have to go each and every query and need to change last modified date?And what if my data does not have any column for last modified date? Not all tables has timestamp column in my data.

Thanks & Regards,
Priyanka Yerunkar.

4

Not sure I understood. But that may be a question to ask in #logstash instead?

If your goal is to sync a SQL based application with elasticsearch, I'd recommend reading this blog post where I shared most of my thoughts.

Basically, I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.

5

Hi,

Thanks @dadoonet
I have changed #elasticsearch to #logstash. What else needs to be done so I can get reply on this?

Regards,
Priyanka

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.