# Logstash pipeline DLQ issue **URL:** \ **Category:** Logstash\ **Created:** [July 18, 2022, 3:15pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920 "2022-07-18T15:15:36Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Harper\_S1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harper_s1/32/103430_2.png) [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Post date:** [July 18, 2022, 3:15pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/1 "2022-07-18T15:15:36Z")
Hi, I am upgrading my logstash from 7.x.x to 8.2.0. Problem is, logstash is not able to run the pipeline, showing the following error. `[2022-07-18T11:08:52,606][ERROR][org.logstash.common.io.DeadLetterQueueWriter][main][447b305b91484edbed17421a6a1732821599fce21ed1ba05c87e58275ef6e875] cannot write event to DLQ(path: /app/logstash/failed/queue/main): reached maxQueueSize of 2147483648 {"index"=>{"_index"=>"index_name", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}` Logstash filters the data from cloud foundry and send the events to elastic and create an index. This thing was working fine in 7.x.x version but after the upgrade its failing. I tried to add the following line in logstash.yml to make pipeline 7.x.x to compatible with 8.x.x but the same result ` ecs_compatibility => disabled` It seems elastic 8.x.x version is not able to process the \_doc. Do anyone know the workaround, It is really frustrating. Thanks! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 18, 2022, 4:27pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/2 "2022-07-18T16:27:14Z")
In 8.0 the value of the [data\_stream](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data_stream) option on the elasticsearch output changed from "false" to "auto", which I think means logstash will check whether the ES instance it is talking to supports so, and if it does it will use them. You could try setting that option on your elasticsearch output ``` data_stream => false ``` ---
**Author:** ![Harper\_S1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harper_s1/32/103430_2.png) [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Post date:** [July 18, 2022, 5:19pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/3 "2022-07-18T17:19:08Z")
Thanks, I tried as per your say but same error. ```auto 022-07-18T13:17:24,228][ERROR][org.logstash.common.io.DeadLetterQueueWriter][main][8af87a010e4f7ef4ee89a5594dca85e0d43e11832c10e9e2655609ced3c0844b] cannot write event to DLQ(path: /app/logstash/failed/queue/main): reached maxQueueSize of 2147483648 ``` here is my output.conf file ```auto filter { mutate { add_field => { "[@metadata][index_name]" => "%{[@metadata][type]}" } } if [appCode] { mutate { lowercase => ["appCode"] } mutate { replace => { "[@metadata][index_name]" => "%{[@metadata][type]}-%{[appCode]}" } } } } output { elasticsearch { user => username password => password hosts => ['https://host1:9200','https://host2:9200','https://host3:9200'] ssl_certificate_verification => false data_stream => false manage_template => false index => "%{[@metadata][index_name]}-8-%{+YYYY.MM.dd}" } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 18, 2022, 5:31pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/4 "2022-07-18T17:31:44Z")
> [@Harper\_S1](#): > > same error. > > ```auto > ... reached maxQueueSize of 2147483648 > > ``` That is a completely different error! My guess is that you ran 2 GB of messages through that got the data stream error, and doing that filled up the DLQ. You now need to empty the DLQ before you can send any more data through your main pipeline. If the only error that you had for the messages in the DLQ is the data stream error, then you can create a pipeline with a [DLQ](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-dead_letter_queue.html) input that sends them to the elasticsearch output. If they were getting other errors then you will need to write a pipeline that modifies the events in the DLQ so that they no longer get those errors. Alternately, if you don't care about the data in the DLQ (which would be odd, because why would you use a DLQ if you don't care about the contents) you could stop logstash and delete the DLQ files. ---
**Author:** ![Harper\_S1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harper_s1/32/103430_2.png) [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Post date:** [July 18, 2022, 5:39pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/5 "2022-07-18T17:39:50Z")
earlier DLQ space was 1 gb, I increased it to 2gb. I tried to delete the dlq messages so many times, but it fill up again and shows error. Same pipelines works fine with 7.16.0 version. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 18, 2022, 5:53pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/6 "2022-07-18T17:53:15Z")
Then you need to read and address the error messages that cause the events to be sent to the DLQ. ---
**Author:** ![Harper\_S1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harper_s1/32/103430_2.png) [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Post date:** [July 18, 2022, 6:44pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/7 "2022-07-18T18:44:17Z")
Yeah, right. Found the problem but not sure about the fix. `response: {"index"=>{"_index"=>"Index-name-abe", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 18, 2022, 6:48pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/8 "2022-07-18T18:48:51Z")
As I said, set `data_stream => false` on the output. You may need to delete the existing index for this to work. ---
**Author:** ![Harper\_S1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harper_s1/32/103430_2.png) [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Post date:** [July 18, 2022, 7:10pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/9 "2022-07-18T19:10:05Z")
Tried the following and deleted the index as well. `data_stream => false` on the output but same dlq issue and data stream is getting created not index, thats strange! ---
**Author:** ![Harper\_S1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harper_s1/32/103430_2.png) [@Harper\_S1](https://discuss.elastic.co/u/Harper_S1)\ **Post date:** [July 18, 2022, 7:31pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/10 "2022-07-18T19:31:37Z")
@Badger I found the problem, template in kibana was created in data view. I just deleted it and created the template in legacy mode and it worked. Thanks for your inputs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 15, 2022, 7:31pm UTC](https://discuss.elastic.co/t/logstash-pipeline-dlq-issue/309920/11 "2022-08-15T19:31:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.