CouchDB plugin set dynamic type for Elasticsearch

Elastic Stack Logstash
12

Thank you very mutch! I would never have found this solution since im very new in the elastic ecosystem.
Querying the type from elasticdb works like a charm for now. I hope this wont slow the system too much down when we are in a stage with many millions of data.
My final config looks like this:

input { 
  couchdb_changes {
      db => "mydbname"
      host => "127.0.0.1"
      port => 5984 	
      initial_sequence => 0 #this is only required for the an initial indexing
  }
}
filter {
  mutate {
    add_field => { "action" => "%{[@metadata][action]}" }
  }
  if [action] == 'delete' {
    elasticsearch {
      hosts => ["127.0.0.1"]
      query => "_id:%{[@metadata][_id]}"
      fields => ["type", "type"]
      sort => ""
    }
  } else {
    mutate {
      add_field => { "type" => "%{[doc][type]}" }
    }
  }
}
output {
  elasticsearch { 
    action => "%{[@metadata][action]}"
    document_id => "%{[@metadata][_id]}"
    host => "127.0.0.1"
    index => "myindexname"
    protocol => "http"
    port => 9200   	  	
  }  
  #stdout { codec => rubydebug } #enable this option for debugging purpose
}

Is there something suboptimal in the config? Something that could slow us down unnecessary?

1 Like