Error.message HTTP error 403 in : 403 Forbidden event.dataset elasticsearch.enrich

Elastic Stack Beats
1

I have this problem with metricbeat:
error.message HTTP error 403 in : 403 Forbidden
event.dataset elasticsearch.enrich
This is my metricbeat.yml

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["my.ip:9200", "my.ip:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "remote_monitoring_user"
  password: "${remote.monitoring.user}"

setup.ilm.enabled: false

And elasticsearch-xpack.yml:

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://my.ip:9200"]
  username: "remote_monitoring_user"
  password: "${remote.monitoring.user}"
  xpack.enabled: true

Elasticsearch version is 7.5.1
Metricbeat version is 7.5.2

Log entry:

2020-01-24T14:36:49.234+0100    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.enrich: HTTP error 403 in : 403 Forbidden
2

I have 2 elasticsearch nodes and only 1 node has this entry. Configuration of both files is the same.

3

Hi @tennaen, sorry I'm trying to figure out the setup here. Do you have one Metricbeat running and trying to write data to two different Elasticsearch nodes? Have you tried to specify protocol: "https" under output.elasticsearch config?

4

Hi @Kaiyan_Sheng, my elasticsearch isn't secured with https, so i'm not using this config. Well yes, i have 2 nodes of elasticsearch and 2 metricbeats running on each Elastic node. I can try to change config and write output only to one of elastic nodes.

5

One more thing: i've updated all my ELK to 7.5.2.

6

Problem solved. Monitoring user was lacking enrich_user role. I have created custom monitoring user and now everything works fine.