Can i prevent duplicate entry in logstash template?

Elastic Stack Logstash
1

I am using the template to insert data to ES. But some duplicate value is inserting, I want to prevent them baseed on reviewId in my example

sample data

[
  {
    "reviewId": "1",
    "displayName": "JOHN"
  },
  {
    "reviewId": "2",
    "displayName": "MAYA"
  },
  {
    "reviewId": "1",
    "displayName": "JOHN"
  }
]

my template is

{
  "index_patterns": ["data_my_*"],
  "settings": {
    "index": {
      "number_of_shards": "2",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "doc":{
      "dynamic": "false",
      "properties": {
        "reviewId": {
          "type": "text"
        }, 
        "displayName": {
          "type": "text"
        }
      }
    }
  },
  "aliases": {}
}

Logstash conf file

input {
  file {
    path            => "/home/templetes/dt5.txt"
    start_position  => "beginning"
    sincedb_path    => "/dev/null"
    codec  => "json"        
  }
}

filter { 
  json {
    source  => "message"
  } 
}

output {
  if [reviewId] != "" {
    stdout { codec => rubydebug }
    elasticsearch {
      hosts => ["http://127.0.0.1:9200"] 
      index => "data_my_1"
      template =>"/home/templetes/datamobi.json"
      template_name=>"datamobi"
      template_overwrite => true
    }
  }
}

thanks, !!!

2

You should use document_id in the output.
Here's an article on duplicates:

1 Like
3

thanks @Mojster ... thumbs up

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.