Status=403): {"type":"security_exception","reason":"action [indices:data/write/bulk[s]

[mmk1995 (MatthewMak)]

Hello,

It took me the whole day and I still cant work it out.
Please see below what is wrong

My Kibana user setting:

metricbeat.yml:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.name: "metricbeat"
setup.template.pattern: "metricbeat-*"
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
  host: "xxxxxxx:5601"
setup.ilm.check_exists: false
output.elasticsearch:
  hosts: ["xxxxxxx:9200"]
  index: "metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  protocol: "https"
  username: "metricbeat_writer"
  password: "xxxxxxxxxxxxxx"

  ssl.certificate_authorities: ["/etc/logstash/elasticsearch-ca.pem"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
monitoring.enabled: true

Error:

Jul 8 16:26:45 ls02 metricbeat: 2020-07-08T16:26:45.021+0800#011WARN#011elasticsearch/client.go:266#011monitoring bulk item insert failed (i=0, status=403): {"type":"security_exception","reason":"action [indices:data/write/bulk[s]] is unauthorized for user [metricbeat_writer]"}

For what I have tested, setting monitoring.enabled: true to false would work, but what I need have to be true.

Please help! Thanks.

[mmk1995 (MatthewMak)]

Setting index privilege from "metricbeat-" to "" would also work, but this probably not a proper solution

[James_Cribbs]

I have the same problem when attempting to write to a filebeat index with logstash. We are running 7.8 across the board here. Our index privileges are view_index_metadata, create_index, create_doc, index. I tried adding the cluster privilages cluster:admin/ingest/pipeline/get as well but that doesn't change the outcome.

I have been looking at this article yesterday to troubleshoot this.

It does appear to be a permissions issue because when you give it basically all permissions to an index it doesn't error out. However that seems a bit risky.

[mmk1995 (MatthewMak)]

I tried to set the index permissions to "all" but the security_exception still exist.

[James_Cribbs]

@mmk1995,
I have a support case opened and we are working to diagnose the issue. As of right now when I disable the service and reboot the server and start it again its able to send logs for a while before getting bogged down with errors. Can you try this to see if you see the same behavior?

[mmk1995 (MatthewMak)]

@James_Cribbs
The only cases I am able to send log for me are as mentioned:

1. Setting index privilege from "metricbeat-" to "" would work
2. setting monitoring.enabled: true to false would work, but what I need have to be true.

For your suggestion, which server have to be rebooted? However, I doubt that reboot is not the solution since it is clearly the security issue, which I think it is the application level bug.

[mmk1995 (MatthewMak)]

Any help from Elastic Team pls?

[James_Cribbs]

@mmk1995 My problem was logstash was reading a variable to specify the index it wrote too, and when an event came with inconsistent data in that field it tried to create a new index. This was identified with assigning the account SuperUser privileges and identifying a suspiciously new index appear.

[James_Cribbs]

One other thing, when using logstash if you have multiple pipeline configs ensure that your input filters and outputs are using if statements control what events flow through them. When logstash starts up it complies a single configuration from each .conf file in the conf.d directory.