Index Permission blocking ingestion of the whole pipeline

Elastic Stack Logstash
1

Hi,

My Logstash containers stopped their ingestion when one index pattern is not permitted in ESS.

Code (simplified):

output {
        opensearch {
            id => "oss_output_cluster-unknown"
            hosts => ["${OPENSEARCH_ENDPOINT}"]
            auth_type => {
              type => 'basic'
              user => '${OPENSEARCH_USER}'
              password => '${OPENSEARCH_PASSWORD}'
            }
            index => "%{[fields][indexName]}-ds"
            action => "create"
            ssl_certificate_verification => false
        }
}

Error:
[2024-06-07T12:35:42,574][INFO ][logstash.outputs.opensearch][output_oss][oss_output_cluster] Retrying failed action {:status=>403, :action=>[\"create\", {:_id=>nil, :_index=>\"commercialcorp-cclstaging-17507-ds\", :routing=>nil}...}}}], :error=>{\"type\"=>\"security_exception\", \"reason\"=>\"no permissions for [indices:admin/create] and User [name=logstash_user, backend_roles=[], requestedTenant=null]\"}}

My assumption is that this particular document cannot be sent to ESS, but that's not the case. Nothing can come through because of this error.

I finally fixed it by adding this index pattern into the role that Logstash is assuming, but my question is that why failure in ingestion for 1 index will affect the ingestion for the whole pipeline?

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

Hello and welcome,

Are you using multiple pipelines or one pipeline with multiple outputs? You didn't share neither your pipelines.yml nor your entire logstash pipeline, so it is not possible to know.

If you are using one pipeline with multiple outputs, then this is expected.

5

This is wrong, there is no dead_letter_queue output, this does not exist.

I would recommend that you validate those AI generated answers as they can be misleading and just invent some things that does not exist.

6

Its a RAG with your support documentation which is current. There is no making anything up

7

I do not work for Elastic, but this answer is wrong.

In the output block it is saying to use a plugin named dead_letter_queue, but this plugin does not exists for the output, just for the input.

You can validate that it does not exist just by looking on the output documentation page here.

So it is mixing up things and making up stuff that does not exist.

For example, you can use the dead_letter_queue output plugin which can store failed events for further inspection without stalling the whole pipeline.

This is entirely made up because there is no dead_letter_queue output plugin, only the input plugin for dead_letter_queue exists.

The DLQ is configured on logstash.yml or pipelines.yml as mentioned in the correct documentation.

8

The main issue with those AI generated answers as that they can just make up things, we have a couple of examples of this in the forum.

Depending on the case this can mislead the users and even cause problems for them.

I'm not sure what is the Elastic policy for AI generated answers on this forum, but for example for their contributor program, AI generated content is not allowed.

10

This doesn´t change the fact that at least this specific answer is wrong by telling the end user to use something that does not exist, that is what I'm saying.

There is no logstash-output-dead_letter_queue plugin, so you could never have a configuration like this:

output {
    dead_letter_queue { path => "/any/path"}
}

If you start logstash with a configuration like that, you will got an error like this:

[2024-06-19T12:23:23,928][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'dead_letter_queue'. Are you sure this is correct? Trying to load the dead_letter_queue output plugin resulted in this error: Unable to load the requested plugin named dead_letter_queue of type output. The plugin is not installed."

If you then try to install a plugin named logstash-output-dead_letter_queue you get this another error:

sudo ./logstash-plugin install logstash-output-dead_letter_queue
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-output-dead_letter_queue
Plugin logstash-output-dead_letter_queue does not exist
ERROR: Installation aborted, verification failed for logstash-output-dead_letter_queue

So, as mentioned the tool you used generated an answer with some step that does not exist and would confuse the users.

If this is wrong about it, what else it is wrong about? That's why there is a need to validate the answers being generated.

You may use this tool to generate answers, but you should also try to validate these answers as they can provide wrong information.

1 Like