Metricbeats failed to connect Elasticsearch with X-Pack enabled

I setup an Elasticsearch 7.2.0 with the following elasticsearch.yml config file, , using docker, "docker-cluster"
discovery.type: single-node
xpack.license.self_generated.type: basic true
xpack.monitoring.collection.enabled: true

then I set password for built-in users using:

bin/elasticsearch-setup-passwords auto --batch

Also, in another (external) server, I setup Metricbeats 7.2.0 with the following config file again by docker,

   hosts: ["http://my_elasticsearch_server_ip:9200"]
   username: elastic

but Metricbeats fails to connect Elasticsearch server, its log says:

metricbeat | 2019-07-15T10:08:11.144Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://my_elasticsearch_server_ip:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

I should mention that, Metricbeats successfully connects to elastic when I set false.

Does user have privileges to publish events?
Are you able to log in using provided user:password combination?
Do you see anything useful in ES logs?

I assumed elastic user have that access. How can I check that?
I can log in using "elastic" user and that password in Kibana, I can also get the result of:
curl -X GET -u elastic:RANDOM_PASS_GENERATED_FOR_ELASTIC_USER "http://my_elasticsearch_server_ip:9200/_cluster/health"

log of ES:

elasticsearch_1 | {"type": "server", "timestamp": "2019-07-15T11:53:29,411+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "": "docker-cluster", "": "xxxxxx", "cluster.uuid": "xxxxxx", "": "-xxxxxx", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }

can you check security index is there. if not please run setup-passwords again and configure metric beat with new password.

Do you also installed x-pack on Kibana and provided same credentials there?
Have you restarted metric beat after changing credentials?

Sorry I am a bit newb, just started elastic, what do you mean by this?
Should I generate a specific password for Metricbeats or I can use elastic user and passwords in Metricbeats client?

I setup ES, Kibana and Logstash using docker-compose, and set the passwords generated by bin/elasticsearch-setup-passwords auto --batch for Kibana and Logstash in their config files like:

elasticsearch.username: kibana
elasticsearch.password: RANDOM_PASS_GENERATED_FOR_KIBANA

Yes, I have restarted Metricbeats several times so far.

For the security index you can go into Kibana -> Management (Gear Icon) -> Index Management and then click on the Include System Indices in the upper right hand corner. That should list some extra indices and one should be .security.

Thanks Michal and Ryan, my issue solved!
Actually, the main problem was in docker-compose that I used for Metricbeats, it binds a config file to Metricbeats container, which I was changing elastic password there, but it also get elastic password from env variable and set it using docker-compose command:

command: metricbeat -e -system.hostfs=/hostfs -E output.elasticsearch.username=elastic -E output.elasticsearch.password=${ES_PASSWORD} -strict.perms=false

this command overwrites the password that I set in config file, I just removed -E from the above command and the issue resolved!

You're welcome! I don't know everything about Elastic but try to help out in the forums when I can. Good luck on your future endeavors with Elastic. It can be tough for sure but its also a pretty powerful tool once you get used to it.

Thanks! That is very good.
Yes, Elastic is really great!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.