# Using Logstash for deduplication of Elasticsearch documents

**URL:** <https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548>\
**Category:** Logstash\
**Created:** [April 30, 2020, 12:34pm UTC](https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548 "2020-04-30T12:34:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Youssef\_SBAI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/youssef_sbai/32/64242_2.png) [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\
**Post date:** [April 30, 2020, 12:34pm UTC](https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548/1 "2020-04-30T12:34:31Z")

</div>

```
input {
  elasticsearch {
    "http://localhost:9200"
    index => "stocks"
    query => '{ "sort": ["_doc"] }'
  }
}

filter {
    fingerprint {
        key => "1234ABCD"
        method => "SHA256"
        source => ["chaine", "job", "date_plan", "statut", "date_debut", "date_fin", "serveur", "numero_passage", "application", "sous_application"]
        target => "[@metadata][generated_id]"
        concatenate_sources => true 
    }
}
output {
    stdout { codec => dots }
    elasticsearch {
        index => "stocks_after_fingerprint"
        document_id => "%{[@metadata][generated_id]}"
    }
}
```

---

<div class="post-metadata">

**Author:** ![Youssef\_SBAI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/youssef_sbai/32/64242_2.png) [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\
**Post date:** [April 30, 2020, 12:35pm UTC](https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548/2 "2020-04-30T12:35:21Z")

</div>

i have this error

Thread.exclusive is deprecated, use Thread::Mutex  
Sending Logstash logs to C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logs which is now configured via log4j2.properties  
[2020-04-30T14:31:41,471][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2020-04-30T14:31:41,671][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.5.2"}  
[2020-04-30T14:31:43,199][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 1)", :backtrace=\>["C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:156:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "C:/Users/h83710/Desktop/elastic/logstash-7.5.2/logstash-core/lib/logstash/agent.rb:326:in `block in converge\_state'"]}  
[2020-04-30T14:31:43,683][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2020-04-30T14:31:48,629][INFO][logstash.runner] Logstash shut down.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 30, 2020, 2:32pm UTC](https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548/3 "2020-04-30T14:32:17Z")

</div>

> [@Youssef\_SBAI](#):
>
> at line 1, column 1 (byte 1)"

The very first byte of your configuration file is a problem. I suggest you check (using 'od -ha') to see if there is a BOM or anything else at the start of your file.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2020, 2:32pm UTC](https://discuss.elastic.co/t/using-logstash-for-deduplication-of-elasticsearch-documents/230548/4 "2020-05-28T14:32:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.