I have the same problem when attempting to write to a filebeat index with logstash. We are running 7.8 across the board here. Our index privileges are view_index_metadata, create_index, create_doc, index. I tried adding the cluster privilages cluster:admin/ingest/pipeline/get as well but that doesn't change the outcome.
I have been looking at this article yesterday to troubleshoot this.
It does appear to be a permissions issue because when you give it basically all permissions to an index it doesn't error out. However that seems a bit risky.
@mmk1995,
I have a support case opened and we are working to diagnose the issue. As of right now when I disable the service and reboot the server and start it again its able to send logs for a while before getting bogged down with errors. Can you try this to see if you see the same behavior?
@James_Cribbs
The only cases I am able to send log for me are as mentioned:
Setting index privilege from "metricbeat-" to "" would work
setting monitoring.enabled: true to false would work, but what I need have to be true.
For your suggestion, which server have to be rebooted? However, I doubt that reboot is not the solution since it is clearly the security issue, which I think it is the application level bug.
@mmk1995 My problem was logstash was reading a variable to specify the index it wrote too, and when an event came with inconsistent data in that field it tried to create a new index. This was identified with assigning the account SuperUser privileges and identifying a suspiciously new index appear.
One other thing, when using logstash if you have multiple pipeline configs ensure that your input filters and outputs are using if statements control what events flow through them. When logstash starts up it complies a single configuration from each .conf file in the conf.d directory.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.