Logstash/Elasticsearch process overwriting doc version each time?



I'm very new to the elastic world, but I was able to get logstash setup and successfully moving data from a SQL Server into my index in elasticsearch using the JDBC driver setup. As of now, my ETL process is currently scheduled to run every minute as my site will be pretty transactional in nature.

When I query (via REST) my elasticsearch index by the document Id ("__id"), I can see the "version" key is incrementing by 1 every time . I'm assuming what's happening is the document is being overwritten despite no changes to the data.

Is there a way I can only have the "__version" key increment by 1 if the record/document has actually changed?

Here is the way my output config file looks, not sure if I need to modify this in any way to achieve my result.

output {
    elasticsearch { 
     action => "update"
     hosts => ["localhost:9200"] 
     index => "company"
     document_type => "article"
     document_id => "%{article_id}"
     doc_as_upsert => true
   stdout { codec => json_lines  }

(Mark Walkom) #2

No, there's not. It's a simplistic retrieve and index.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.