# Index Permission blocking ingestion of the whole pipeline **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [June 10, 2024, 6:58pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189 "2024-06-10T18:58:12Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Perry\_Lam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/perry_lam/32/122745_2.png) [@Perry\_Lam](https://discuss.elastic.co/u/Perry_Lam)\ **Post date:** [June 10, 2024, 6:58pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/1 "2024-06-10T18:58:12Z")
Hi, My Logstash containers stopped their ingestion when one index pattern is not permitted in ESS. Code (simplified): ```auto output { opensearch { id => "oss_output_cluster-unknown" hosts => ["${OPENSEARCH_ENDPOINT}"] auth_type => { type => 'basic' user => '${OPENSEARCH_USER}' password => '${OPENSEARCH_PASSWORD}' } index => "%{[fields][indexName]}-ds" action => "create" ssl_certificate_verification => false } } ``` Error: `[2024-06-07T12:35:42,574][INFO][logstash.outputs.opensearch][output_oss][oss_output_cluster] Retrying failed action {:status=>403, :action=>[\"create\", {:_id=>nil, :_index=>\"commercialcorp-cclstaging-17507-ds\", :routing=>nil}...}}}], :error=>{\"type\"=>\"security_exception\", \"reason\"=>\"no permissions for [indices:admin/create] and User [name=logstash_user, backend_roles=[], requestedTenant=null]\"}}` My assumption is that this particular document cannot be sent to ESS, but that's not the case. Nothing can come through because of this error. I finally fixed it by adding this index pattern into the role that Logstash is assuming, but my question is that why failure in ingestion for 1 index will affect the ingestion for the whole pipeline? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 10, 2024, 6:58pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/2 "2024-06-10T18:58:12Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See [What is OpenSearch and the OpenSearch Dashboard? | Elastic](https://www.elastic.co/elasticsearch/opensearch) for more details. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 10, 2024, 7:14pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/3 "2024-06-10T19:14:04Z")
Hello and welcome, Are you using multiple pipelines or one pipeline with multiple outputs? You didn't share neither your `pipelines.yml` nor your entire logstash pipeline, so it is not possible to know. If you are using one pipeline with multiple outputs, then this is expected. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 19, 2024, 2:41pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/5 "2024-06-19T14:41:10Z")
> [@rtwolfe94022](#): > > ```auto > output { > if "_grokparsefailure" in [tags] { > dead_letter_queue { > path => "/var/log/logstash/dead_letter_queue" > } > > ``` This is wrong, there is no `dead_letter_queue` output, this does not exist. I would recommend that you validate those AI generated answers as they can be misleading and just invent some things that does not exist. ---
**Author:** ![rtwolfe94022](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtwolfe94022/32/132253_2.png) [@rtwolfe94022](https://discuss.elastic.co/u/rtwolfe94022)\ **Post date:** [June 19, 2024, 2:54pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/6 "2024-06-19T14:54:37Z")
Its a RAG with your support documentation which is current. There is no making anything up ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 19, 2024, 3:03pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/7 "2024-06-19T15:03:15Z")
I do not work for Elastic, but this answer is wrong. In the `output` block it is saying to use a plugin named `dead_letter_queue`, but this plugin does not exists for the `output`, just for the `input`. You can validate that it does not exist just by looking on the output documentation page [here](https://www.elastic.co/guide/en/logstash/current/output-plugins.html). So it is mixing up things and making up stuff that does not exist. > For example, you can use the `dead_letter_queue` output plugin which can store failed events for further inspection without stalling the whole pipeline. This is entirely made up because there is no `dead_letter_queue` output plugin, only the input plugin for `dead_letter_queue` exists. The DLQ is configured on `logstash.yml` or `pipelines.yml` as mentioned in the correct [documentation](https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html). ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 19, 2024, 3:11pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/8 "2024-06-19T15:11:36Z")
The main issue with those AI generated answers as that they can just make up things, we have a couple of examples of this in the forum. Depending on the case this can mislead the users and even cause problems for them. I'm not sure what is the Elastic policy for AI generated answers on this forum, but for example for their contributor program, AI generated content is not allowed. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 19, 2024, 3:30pm UTC](https://discuss.elastic.co/t/index-permission-blocking-ingestion-of-the-whole-pipeline/361189/10 "2024-06-19T15:30:15Z")
> [@rtwolfe94022](#): > > We use a RAG sitting on Llama3 in this case had both Elastic and Docker current documentaion. I chain the RAG.. So in this case the user specified Docker + Elastic the two were chained. This isnt ChatGPT This doesn´t change the fact that at least this specific answer is wrong by telling the end user to use something that does not exist, that is what I'm saying. There is no `logstash-output-dead_letter_queue` plugin, so you could never have a configuration like this: ```auto output { dead_letter_queue { path => "/any/path"} } ``` If you start logstash with a configuration like that, you will got an error like this: > [2024-06-19T12:23:23,928][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"Java::JavaLang::IllegalStateException", :message=\>"Unable to configure plugins: (PluginLoadingError) **Couldn't find any output plugin named 'dead\_letter\_queue'. Are you sure this is correct?** Trying to load the dead\_letter\_queue output plugin resulted in this error: **Unable to load the requested plugin named dead\_letter\_queue of type output. The plugin is not installed.**" If you then try to install a plugin named `logstash-output-dead_letter_queue` you get this another error: > sudo ./logstash-plugin install logstash-output-dead\_letter\_queue > Using bundled JDK: /usr/share/logstash/jdk > Validating logstash-output-dead\_letter\_queue > **Plugin logstash-output-dead\_letter\_queue does not exist** > ERROR: Installation aborted, verification failed for logstash-output-dead\_letter\_queue So, as mentioned the tool you used generated an answer with some step that does not exist and would confuse the users. If this is wrong about it, what else it is wrong about? That's why there is a need to validate the answers being generated. You may use this tool to generate answers, but you should also try to validate these answers as they can provide wrong information.