Hi,
Here are the logs for auditbeat and filebeat, I would really appreciate if you can shed some light on this.
2020-12-09T14:50:09.542+1100 INFO instance/beat.go:299 Setup Beat: auditbeat; Version: 7.9.1
2020-12-09T14:50:09.542+1100 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'auditbeat-7.9.1' as ILM is enabled.
2020-12-09T14:50:09.544+1100 INFO eslegclient/connection.go:99 elasticsearch url: https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443
2020-12-09T14:50:09.546+1100 INFO [publisher] pipeline/module.go:113 Beat name: xxxxx-devr101
2020-12-09T14:50:09.546+1100 INFO eslegclient/connection.go:99 elasticsearch url: https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443
2020-12-09T14:50:09.556+1100 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"11111111111"},"availability_zone":"ap-southeast-2a","image":{"id":"ami-xxxxxx"},"instance":{"id":"i-xxxxxxx"},"machine":{"type":"t2.small"},"provider":"aws","region":"ap-southeast-2"}
2020-12-09T14:50:09.609+1100 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.9.1
2020-12-09T14:50:09.613+1100 ERROR instance/beat.go:951 Exiting: request checking for ILM availability failed: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}
I'm having the same issue with Filebeat too.
Filebeat logs:
2021-01-11T11:53:40.447+1100 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-01-11T11:53:40.447+1100 INFO [publisher] pipeline/retry.go:223 done
2021-01-11T11:53:40.450+1100 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.9.1
2021-01-11T11:54:02.033+1100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1430,"time":{"ms":3}},"total":{"ticks":3460,"time":{"ms":10},"value":3460},"user":{"ticks":2030,"time":{"ms":7}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"xxxxxxx-9604-4188-b443-eebaef6f3299","uptime":{"ms":8430040}},"memstats":{"gc_next":10944688,"memory_alloc":6283936,"memory_total":251621456},"runtime":{"goroutines":53}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":2}},"output":{"read":{"bytes":837},"write":{"bytes":470}},"pipeline":{"clients":4,"events":{"active":60,"retry":50}}},"registrar":{"states":{"current":14}},"system":{"load":{"1":0.18,"15":0.07,"5":0.11,"norm":{"1":0.18,"15":0.07,"5":0.11}}}}}}
2021-01-11T11:54:32.032+1100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1440,"time":{"ms":5}},"total":{"ticks":3470,"time":{"ms":9},"value":3470},"user":{"ticks":2030,"time":{"ms":4}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"xxxxxxx-9604-4188-b443-eebaef6f3299","uptime":{"ms":8460040}},"memstats":{"gc_next":10944688,"memory_alloc":7138768,"memory_total":252476288},"runtime":{"goroutines":53}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":2}},"pipeline":{"clients":4,"events":{"active":60}}},"registrar":{"states":{"current":14}},"system":{"load":{"1":0.11,"15":0.07,"5":0.1,"norm":{"1":0.11,"15":0.07,"5":0.1}}}}}}
2021-01-11T11:54:36.000+1100 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: request checking for ILM availability failed: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}
2021-01-11T11:54:36.000+1100 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443)) with 6 reconnect attempt(s)