How to insert only newly added rows in csv in logstash in old index

Elastic Stack Logstash
1

Hi community,

If I have a csv with 5 rows and I have shipped it in logstash. In future, the csv gets updated with 5(old) + 2(new) =7 rows. I want to ship the updated csv in the same index. Can I ship only the updated 2(new) rows of the csv file with the 5(old) rows already preexisting in the index? If yes, please help.

I want to visualise the old and new rows in kibana with the help of the timestamp

2

Hello,
you can add 7 columns from the beginning of the index or you can also add field after creation of index also by adding field in index pattern.

image
image1324×341 20.6 KB

3

Cant I simply ingest the updated csv and it only adds the new rows in the index.
Also can you please elaborate on your solution.
Thanks a lot

4

it will create duplicate rows... so you need to delete your index each time and again you have to create index.... normally you can create your updated csv for every day and ingest it.
No. of rows update automatically but you must maintain the field = columns of the table.

5

OK noted. Can you please confirm that my desired test case be done or not with the solution that I have proposed.

Thank you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.