# Save only unique events that are not yet stored in ES based on field value
**URL:** https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466
**Category:** Logstash
**Tags:** docker
**Created:** [April 4, 2022, 7:57am UTC](https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466 "2022-04-04T07:57:36Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Lupul\_Dacic](https://discuss.elastic.co/u/Lupul_Dacic)
#### Post date: [April 4, 2022, 7:57am UTC](https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466/1 "2022-04-04T07:57:36Z")
Hi all,
I am using Logstash with a http\_poller. Basically I am querying an API and save the data in ELK. All is fine, but the situation is that I am saving the same events and I need unique ones.
I would like to save only those events that are not already in the DB. If the value from data.attributes.cveID is not already saved in the ELK, then save the objects, else drop the event.
I.e if there is no CVE-2022-21698 in ES, then save it else drop the event.
How I can do this?
PFA my logstash.conf
```auto
input {
http_poller {
urls => {
test2 => {
method => get
url => "http://localhost:1337/api/webs/"
}
}
request_timeout => 60
schedule => { cron => "* * * * * UTC"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
filter {
split { field => "[data]" }
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "logstash_internal"
password => "${LOGSTASH_INTERNAL_PASSWORD}"
}
}
```
If you could help me, I will owe you a lot!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 4, 2022, 6:22pm UTC](https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466/2 "2022-04-04T18:22:59Z")
> [@Lupul\_Dacic](#):
>
> If the value from data.attributes.cveID is not already saved in the ELK, then save the objects, else drop the event.
You can do this by overwriting the document each time.
Use `document_id => %{[data][attributes][cveID]}` in the elasticsearch output.
If you really want to drop the document then you could use an elasticsearch filter to do a lookup and see if it exists. However, in my [very limited experience](https://discuss.elastic.co/t/elasticsearch-filter-giving-typeerror-no-implicit-conversion-of-nil-into-string/123791) that is fragile and does not work as well as you would expect (in that case possibly due to the way logstash processes batches).
---
### Author:  [@Lupul\_Dacic](https://discuss.elastic.co/u/Lupul_Dacic)
#### Post date: [April 4, 2022, 7:43pm UTC](https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466/3 "2022-04-04T19:43:06Z")
> [@Badger](#):
>
> document\_id =\> %{[data][attributes][cveID]}
PFA the output from logstash.conf
```auto
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
user => "logstash_internal"
password => "${LOGSTASH_INTERNAL_PASSWORD}"
document_id => %{[attributes][cveID]}
}
}
```
I got an error:
```auto
[logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", [A-Za-z_], \"-\", [0-9], \"[\", \"{\" at line 36, column 18 (byte 725) after output {\n\telasticsearch {\n\t\thosts => [\"http://elasticsearch:9200\"]\n\t\tuser => \"logstash_internal\"\n\t\tpassword => \"${LOGSTASH_INTERNAL_PASSWORD}\"\n\t\tdocument_id => ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:376:in `block in converge_state'"]}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 4, 2022, 7:45pm UTC](https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466/4 "2022-04-04T19:45:09Z")
> [@Lupul\_Dacic](#):
>
> `document_id => %{[attributes][cveID]}`
Add double quotes around the value: `"%{[attributes][cveID]}"`.
---
### Author:  [@Lupul\_Dacic](https://discuss.elastic.co/u/Lupul_Dacic)
#### Post date: [April 4, 2022, 7:57pm UTC](https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466/5 "2022-04-04T19:57:44Z")
it works like a charm! Thank you!
You're the best!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 2, 2022, 7:58pm UTC](https://discuss.elastic.co/t/save-only-unique-events-that-are-not-yet-stored-in-es-based-on-field-value/301466/6 "2022-05-02T19:58:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.