Elasticsearch is unable to authenticate the credentials

I had enabled authentication on my ELK stack (single node) , every thing works as expected (role,users etc). however my elasticsearch logs have logged some errors such as

[2021-07-06T15:33:59,086][INFO ][o.e.x.s.a.AuthenticationService] [dev-eng05] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

I have confirmed the credentials via curl to be correct

root@dev-eng05:/etc/logstash# curl -u kibana_system 'http://172.26.205.249:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'kibana_system':
{
  "username" : "kibana_system",
  "roles" : [
    "kibana_system"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "_reserved" : true
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "lookup_realm" : {
    "name" : "reserved",
    "type" : "reserved"
  },
  "authentication_type" : "realm"
}
root@dev-eng05:/etc/logstash# 

*# If your Elasticsearch is protected with basic authentication, these settings provide*

*# the username and password that the Kibana server uses to perform maintenance on the Kibana*

*# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which*

*# is proxied through the Kibana server.*

***elasticsearch.username*** *:* *"kibana_system"*

***elasticsearch.password*** *:* *"Test123"*

When trying to run my logstash

[2021-07-06T15:42:19,047][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://172.26.205.249:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://172.26.205.249:9200/'"}