Document deleted after logstash execution

Elastic Stack Logstash
1

Hello,

I am using the latest version of ELK (7.11.2) on ubuntu 18.4.
I am getting data from postgre using JDBC

My logstash.conf file look lile : input { jdbc { jdbc_connection_string =>"jdbc:postgresql://123.0.0.1:5432/elk" jdbc_user => "elk" jdbc_password => "asco" jdbc_driver_class => "org.postgresql.Driver" jdbc_driver_library => "/etc/logstash/postgresql-42.2.20.jar" statement => "SELECT * from elk.w7_log" jdbc_default_timezone => "Europe/Paris" schedule => "*/5 * * * * *" } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "plwelk" #document_id => "users_%{userid}" document_id => "%{id}" #doc_as_upsert => true #action => "update" #user => "elk" } }

In my index "plwelk" I can see maximum 2 documents. When I execute

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/elk.conf its deleting 3rd document.

I dont want to delete any documents and dont know how to prevent.

I the attached 2 screenshot you can see that I have document with Apr 29, 2021 and Apr 22, 2021 but after execution of logstash (/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/elk.conf) on Apr 30, 2021 I lost Apr 28, 2021

ELK1.PNG
ELK1.PNG1217×765 191 KB
ELK2
ELK21839×898 182 KB

can you please tell me why?

2

What is the _id value on each document? You are setting the document_id option on the elasticsearch output, so if there are two events with the same value of %{id} the second will overwrite the first.

3

Hello Badger,

Thanks for reply.
I don't know what I can put to get unique id. In my database table (w7_log) I dont have any column which is unique.
Can you please suggest me how I can generate unique id.? Also is it mandatory to provide id? Can't I remove #document_id => "users_%{userid}" from the output of my conf file?

4

The document_id option on the elasticsearch output is optional. If you do not supply it then a unique id will be generated.

5

Ok, thanks Badger, I will remove and execute logstash again tomorrow.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.