Logstash Insert or Update if exists

Elastic Stack Logstash
1

hello guys , so lately i've trying to insert some data from txt file into elasticsearch index , but i'm having a problem in updating my index , when i run the logstach conf file it's not updating the data or inserting the new ones , it is adding the data again, here is my logstash conf :
input {

file {
path => "C:/Elastic/Rapport_finale.txt"
start_position => "beginning"

}

}
 filter {
    
      csv {
	  separator => ";"
		 columns =>["ID","DATE_SITUATION","HEURE","COMPANY","NOM_COMPANY","DATE_SIT","NBRE_OPERATION_TOTAL","OPERATION_CAISSE","LR_LATITUDE","LR_LONGITUDE","SOMME","LIB_REGION","LIB_ZONE","LIB_AGENCE","DATE_UNIQ"]
		 
      } 
	  	
	  
	  date {
  match => [ "DATE_SIT", "ISO8601", "YYYY-MM-dd HH:mm:ss", "YYYY-MM-dd HH:mm:ss.ZZZ" ]
  target => "DATE"
  locale => "en"
}
	  mutate {
    convert => {"NBRE_OPERATION_TOTAL" => "integer"}
    convert => {"OPERATION_CAISSE" => "integer"}
	convert => {"SOMME" => "integer"}
   }
	
	  mutate {
   add_field => { "[location][lat]" => "%{LR_LONGITUDE}" }
   add_field => { "[location][lon]" => "%{LR_LATITUDE}" }
  }
   mutate {
    convert => {"[location][lat]" => "float"}
    convert => {"[location][lon]" => "float"}
   }
}
 

output{

  elasticsearch {
  hosts => "*.*.*.*:9200"
  index => "rapport_index_dg_finale_test"
  #document_type => "my_type"
  action => "update"
  doc_as_upsert => true
  document_id => "%{ID}"
  user => "logstash_admin"
  password => "*******"
 }


 }

any idea what am i doing wrong ?

thanks.

2

Remove this :slight_smile:

3

@AquaX Hi , i tried removing doc_as_upsert but i got this error
Could not index event to Elasticsearch. {:status=>404, :action=>["update", {:_id=>"2019010218604", :_index=>"rapport_index_dg_finale_test", :_type=>"_doc", :routing=>nil, :retry_on_conflict=>1}, #<LogStash::Event:0x3606e081>], :response=>{"update"=>{"_index"=>"rapport_index_dg_finale_test", "_type"=>"_doc", "_id"=>"2019010218604", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[_doc][2019010218604]: document missing", "index_uuid"=>"V6toDwWOSyG8YJNXrpZJtA", "shard"=>"0", "index"=>"rapport_index_dg_finale_test"}}}}
any idea what's wrong ?

4

Yep! Now change action to “index”. Sorry didn’t see that you had “update”

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.