# Catch records and erros that don't go into the elasticsearch cluster in another bucket **URL:** \ **Category:** Logstash\ **Created:** [July 1, 2019, 6:45pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389 "2019-07-01T18:45:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Mohit\_Ruke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohit_ruke/32/46323_2.png) [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\ **Post date:** [July 1, 2019, 6:45pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389/1 "2019-07-01T18:45:43Z")
Hi All, Trying to transfer data from S3 bucket to ES. Using Logstash for that. Everytime a new file is deposited, it will be transferred to ES. The follow is the config file. ``` input { s3 { bucket => "logging-services-ods-dev" access_key_id => "*" secret_access_key => "*" region => "us-east-1" prefix => "Engineering_Indexes/platform_dump/" codec => "json" type => "s3" } } filter{ mutate{ add_field =>{ "file" => "%{[@metadata][s3][key]}" } } mutate { gsub => ["file", ".{34}", ""] } mutate { gsub => ["file", ".{3}$", "", "file", "([0-9]{4}-[0-9]{2})-[0-9]+", "\1" ] } } output { amazon_es { hosts => ["endpoint"] region => "us-east-1" aws_access_key_id => '*' aws_secret_access_key => '*' index => "%{file}" template_name => "sqe_template1" template_overwrite => "true" codec => "json" } } ``` However if can records are rejected from a file, how can I catch those rejected records either in a new s3 bucket or a by writing to a file. Is it possible to catch the rejected or error records? Thank You, Mohit Ruke ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 1, 2019, 7:41pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389/2 "2019-07-01T19:41:07Z")
It sounds like you want a [DLQ](https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html), but that is currently only supported for an elasticsearch output. ---
**Author:** ![Mohit\_Ruke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohit_ruke/32/46323_2.png) [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\ **Post date:** [July 1, 2019, 9:02pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389/3 "2019-07-01T21:02:22Z")
Yes that is what I need for the output , can you help put how to integrate it with my current output. Like the error files or records should be written somewhere Thank You ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 1, 2019, 10:04pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389/4 "2019-07-01T22:04:22Z")
If you look at the [code](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/e7cc0c16d349e597d425b720815e407ed2b2c687/lib/logstash/outputs/elasticsearch/common.rb#L251) for the elasticsearch output, there is actually very little needed to support a DLQ. It basically says "If you get a 400 or 404 back from ES, then call execution\_context.dlq\_writer". execution\_context.dlq\_writer is core logstash functionality, available to any plugin. So it should be pretty straightforward to modify the amazon\_es output to make the same call in the same circumstances. ---
**Author:** ![Mohit\_Ruke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohit_ruke/32/46323_2.png) [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\ **Post date:** [July 3, 2019, 7:34pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389/5 "2019-07-03T19:34:36Z")
Hey badger I tired implementing the logstash DLQ but I am getting this error: ``` input { dead_letter_queue { commit_offsets => true path => "/var/lib/logstash/dead_letter_queue" pipeline_id => "main" } } output { amazon_es { hosts => ["https://vpc-log-service-legacy-data-lm4aq2wmxzh4xxj2uukkucykfi.us-east-1.es.amazonaws.com"] region => "us-east-1" aws_access_key_id => '*' aws_secret_access_key => '*' index => "dead_letter_queue-error" } } ``` Error: [ERROR] 2019-07-03 18:56:01.122 [[main]-pipeline-manager] pipeline - Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"\"main", path=\>"/var/lib/logstash/dead\_letter\_queue", id=\>"87551508e23c487ab5776be5570a2ce4189f7ffff5083e2ead7b74203deaf8d7", commit\_offsets=\>true, enable\_metric=\>true, codec=\>\"plain\_ebb78360-e392-4b37-be0d-ef39476d811c", enable\_metric=\>true, charset=\>"UTF-8"\>\>", :error=\>"/var/lib/logstash/dead\_letter\_queue/main", :thread=\>"#\"} [ERROR] 2019-07-03 18:56:01.201 [[main]-pipeline-manager] pipeline - Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>java.nio.file.NoSuchFileException: /var/lib/logstash/dead\_letter\_queue/main, :backtrace=\>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:86)", "sun.nio.fs.UnixException.asIOException(sun/nio/fs/UnixException.java:111)", "sun.nio.fs.LinuxWatchService$Poller.implRegister(sun/nio/fs/LinuxWatchService.java:246)", "sun.nio.fs.AbstractPoller.processRequests(sun/nio/fs/AbstractPoller.java:260)", "sun.nio.fs.LinuxWatchService$Poller.run(sun/nio/fs/LinuxWatchService.java:364)", "java.lang.Thread.run(java/lang/Thread.java:748)"], :thread=\>"#\"} [ERROR] 2019-07-03 18:56:01.221 [Converge PipelineAction::Create] agent - Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create, action\_result: false", :backtrace=\>nil} ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 3, 2019, 9:05pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389/6 "2019-07-03T21:05:47Z")
I think you have misunderstood. There are two sides to the DLQ. It is written to by the elasticsearch output when it gets a 400 or 404 from elasticsearch. I was suggesting that you could modify the amazon\_es output to support the same functionality. If you did that then amazon\_es would write to the DLQ which you would be able to consume with a dead\_letter\_queue input. However, until something writes to the DLQ the file will not exist and the dead\_letter\_queue input will log that error. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 31, 2019, 9:05pm UTC](https://discuss.elastic.co/t/catch-records-and-erros-that-dont-go-into-the-elasticsearch-cluster-in-another-bucket/188389/7 "2019-07-31T21:05:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.