ILM enabled Logstash Pipeline Issue

Hi Folks,

I am facing an issue with a ILM enabled logstash pipeline. Let me describe my objective. So I have created a logstash pipeline that has upsert feature to update/insert documents in the elasticsearch index. My another requirement is to enable the ILM policy for that index. As we know that for ILM policy we generally need to create a data stream but as data stream has only the "create" option that is why I used an alternative approach as below.

  1. Created ILM Policy
PUT _ilm/policy/ilm_policy_test_and_delete
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
  1. Created Index template
PUT _index_template/index_template_test_and_delete
{
  "index_patterns": ["test_and_delete-*"],                 
  "template": {
	"settings": {
	  "number_of_shards": 1,
	  "number_of_replicas": 1,
	  "index.lifecycle.name": "ilm_policy_test_and_delete",      
	  "index.lifecycle.rollover_alias": "test_and_delete"    
	}
  }
}
  1. Bootstraped index
PUT test_and_delete-2024-05-03-000001
{
  "aliases": {
	"test_and_delete": {
	  "is_write_index": true
	}
  }
}

4 And then started my logstash pipeline, having the below Output plugin configuration:

output {
   if [type] == "test_and_delete" {
     elasticsearch {
        hosts => ["${ES_HOST1}","${ES_HOST2}","${ES_HOST3}"]
        index => "test_and_delete"
        document_id => "%{[@metadata][_id]}"
        user => "${ES_USER}"
        password => "${ES_PASSWORD}"
		doc_as_upsert => true
	    action => "update"
        manage_template => true
		ilm_rollover_alias => "test_and_delete"
		ilm_pattern => "{now/d}-000001"
		ilm_policy => "ilm_policy_test_and_delete"
        }
    }
}

But when I started the pipeline it is throwing the below error continuously --

[2024-05-03T05:54:43,346][ERROR][logstash.outputs.elasticsearch][main][e60ee26ef629e7dbe4db94c41156641c2c467ca0257882fe1031cc13b8662410] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"Got response code '403' contacting Elasticsearch at URL '
http://xxx.xx.x.xxx:9200/test_and_delete'"
, :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError}

however, data is being added in the index "test_and_delete-2024-05-03-000001".

Hope I am able to explain the objective and the issue. If you need any further information, please do let me know.

Why I am getting the error? How to resolve it?

My ELK Version is: 7.11.1

Please help!

Kind regards,
Souvik

Can anyone from @elastic help me with this, please?

Hi Alex, @spinscale, apologies for pinging you like this. I have seen that you responded to similar issues. Could you please help me with this?

The work has been stuck because of this. I need to resolve the issue ASAP. Kindly help.

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

A 403 error indicates a forbidden access to an API I think. Check the roles of the ${ES_USER} user.

Also what are the Elasticsearch logs?

My ELK Version is: 7.11.1

I'd upgrade to 7.17 in case this is a bug that has been fixed later on.

Hi David @dadoonet, I apologies for my action. Normally, I wouldn't have pinged anyone like I did this time. However, I felt compelled to do so because it has been blocking a priority work for some time. I hope you understand. Noted on the "Be Patient" facts.

On the 403 error, The user ${ES_USER} seems not creating the issue as it is "elastic". Regarding the elasticsearch log, I have already shared a snippet of it, do you want the entire log file?
Lastly, on the ELK stack upgrade, yes, you're right. We are in the process of upgrading the whole stack to the latest version but it will take some time as it is a Production Deployment and multiple checks, and decisions need to be taken before start upgrading the stack and the work related to the issue needs to be done before it. Hope you understand.

If I'm not mistaken, it's a Logstash log, not Elasticsearch logs. Could you share the later? It might give a clue.

Hi David @dadoonet , Yes, you're right, the log I shared is the logstash log, my apologies for the overlook. I have checked the elasticsearch cluster log for the day but couldn't find anything related to the index "test_and_delete".

Is there any other way to troubleshoot it?

This is an 403 error, which means that the user are you using in the elasticsearch output is missing some credentials.

Does the user you are using have the permissions mentioned in the documenntation?

Also, any reason to manage the template through logstash and not outside it? Personally I would use logstash only to send data, everything else I prefer to manage outside of it,

You just set manage_template, but you didn't specify the path for the json with the template file, with this setting it will manage only the template for the indices starting with logstash-* as mentioned in the documentation.

Unless you have a requirement to edit and manage templates through logstash, I would remove all of these settings:

manage_template => true
ilm_rollover_alias => "test_and_delete"
ilm_pattern => "{now/d}-000001"
ilm_policy => "ilm_policy_test_and_delete"

You configured all these manually before.

Hi Leandro @leandrojmp ,

Thanks for your response. It helped. I removed the ILM-related parameters from the logstash config and rerun the pipeline and it worked.