Metricbeats failed to connect Elasticsearch with X-Pack enabled

Elastic Stack Beats
1

I setup an Elasticsearch 7.2.0 with the following elasticsearch.yml config file, , using docker,

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.type: single-node
xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

then I set password for built-in users using:

bin/elasticsearch-setup-passwords auto --batch

Also, in another (external) server, I setup Metricbeats 7.2.0 with the following config file again by docker,

output.elasticsearch:
   hosts: ["http://my_elasticsearch_server_ip:9200"]
   username: elastic
   password: RANDOM_PASS_GENERATED_FOR_ELASTIC_USER

but Metricbeats fails to connect Elasticsearch server, its log says:

metricbeat | 2019-07-15T10:08:11.144Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://my_elasticsearch_server_ip:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

I should mention that, Metricbeats successfully connects to elastic when I set xpack.security.enabled: false.

2

Does user have privileges to publish events?
Are you able to log in using provided user:password combination?
Do you see anything useful in ES logs?

3

I assumed elastic user have that access. How can I check that?
I can log in using "elastic" user and that password in Kibana, I can also get the result of:
curl -X GET -u elastic:RANDOM_PASS_GENERATED_FOR_ELASTIC_USER "http://my_elasticsearch_server_ip:9200/_cluster/health"

log of ES:

elasticsearch_1 | {"type": "server", "timestamp": "2019-07-15T11:53:29,411+0000", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "docker-cluster", "node.name": "xxxxxx", "cluster.uuid": "xxxxxx", "node.id": "-xxxxxx", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]" }

4

can you check security index is there. if not please run setup-passwords again and configure metric beat with new password.

Do you also installed x-pack on Kibana and provided same credentials there?
Have you restarted metric beat after changing credentials?

5

Sorry I am a bit newb, just started elastic, what do you mean by this?
Should I generate a specific password for Metricbeats or I can use elastic user and passwords in Metricbeats client?

I setup ES, Kibana and Logstash using docker-compose, and set the passwords generated by bin/elasticsearch-setup-passwords auto --batch for Kibana and Logstash in their config files like:

elasticsearch.username: kibana
elasticsearch.password: RANDOM_PASS_GENERATED_FOR_KIBANA

Yes, I have restarted Metricbeats several times so far.

6

For the security index you can go into Kibana -> Management (Gear Icon) -> Index Management and then click on the Include System Indices in the upper right hand corner. That should list some extra indices and one should be .security.

7

Thanks Michal and Ryan, my issue solved!
Actually, the main problem was in docker-compose that I used for Metricbeats, it binds a config file to Metricbeats container, which I was changing elastic password there, but it also get elastic password from env variable and set it using docker-compose command:

command: metricbeat -e -system.hostfs=/hostfs -E output.elasticsearch.username=elastic -E output.elasticsearch.password=${ES_PASSWORD} -strict.perms=false

this command overwrites the password that I set in config file, I just removed -E from the above command and the issue resolved!

8

You're welcome! I don't know everything about Elastic but try to help out in the forums when I can. Good luck on your future endeavors with Elastic. It can be tough for sure but its also a pretty powerful tool once you get used to it.

9

Thanks! That is very good.
Yes, Elastic is really great!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.