Disable x-pack on Auditbeat and Metricbeat clients 7.9 (AWS Elasticsearch service) using OSS version

Hi all,

I'm using AWS Elasticsearch service (7.9) and I have installed metricbeat and auditbeat clients (oss) 7.9.1 on a linux node:

Beats client source:


However somehow x-pack check is enabled, so I'm getting the following error message:

Connection marked as failed because the onConnect callback failed: request checking for ILM availability failed: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}

I have installed the client using the RPM ( auditbeat-oss-7.9.1-x86_64.rpm)

Any help will be appreciated. What am I missing?

Could you please share some logs you're observing?

Hi,

Here are the logs for auditbeat and filebeat, I would really appreciate if you can shed some light on this. :slight_smile:

2020-12-09T14:50:09.542+1100	INFO	instance/beat.go:299	Setup Beat: auditbeat; Version: 7.9.1
2020-12-09T14:50:09.542+1100	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'auditbeat-7.9.1' as ILM is enabled.
2020-12-09T14:50:09.544+1100	INFO	eslegclient/connection.go:99	elasticsearch url: https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443
2020-12-09T14:50:09.546+1100	INFO	[publisher]	pipeline/module.go:113	Beat name: xxxxx-devr101
2020-12-09T14:50:09.546+1100	INFO	eslegclient/connection.go:99	elasticsearch url: https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443
2020-12-09T14:50:09.556+1100	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:93	add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"11111111111"},"availability_zone":"ap-southeast-2a","image":{"id":"ami-xxxxxx"},"instance":{"id":"i-xxxxxxx"},"machine":{"type":"t2.small"},"provider":"aws","region":"ap-southeast-2"}
2020-12-09T14:50:09.609+1100	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.9.1
2020-12-09T14:50:09.613+1100	ERROR	instance/beat.go:951	Exiting: request checking for ILM availability failed: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}

I'm having the same issue with Filebeat too.

Filebeat logs:

2021-01-11T11:53:40.447+1100	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-01-11T11:53:40.447+1100	INFO	[publisher]	pipeline/retry.go:223	  done
2021-01-11T11:53:40.450+1100	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.9.1
2021-01-11T11:54:02.033+1100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1430,"time":{"ms":3}},"total":{"ticks":3460,"time":{"ms":10},"value":3460},"user":{"ticks":2030,"time":{"ms":7}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"xxxxxxx-9604-4188-b443-eebaef6f3299","uptime":{"ms":8430040}},"memstats":{"gc_next":10944688,"memory_alloc":6283936,"memory_total":251621456},"runtime":{"goroutines":53}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":2}},"output":{"read":{"bytes":837},"write":{"bytes":470}},"pipeline":{"clients":4,"events":{"active":60,"retry":50}}},"registrar":{"states":{"current":14}},"system":{"load":{"1":0.18,"15":0.07,"5":0.11,"norm":{"1":0.18,"15":0.07,"5":0.11}}}}}}
2021-01-11T11:54:32.032+1100	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1440,"time":{"ms":5}},"total":{"ticks":3470,"time":{"ms":9},"value":3470},"user":{"ticks":2030,"time":{"ms":4}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"xxxxxxx-9604-4188-b443-eebaef6f3299","uptime":{"ms":8460040}},"memstats":{"gc_next":10944688,"memory_alloc":7138768,"memory_total":252476288},"runtime":{"goroutines":53}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":2}},"pipeline":{"clients":4,"events":{"active":60}}},"registrar":{"states":{"current":14}},"system":{"load":{"1":0.11,"15":0.07,"5":0.1,"norm":{"1":0.11,"15":0.07,"5":0.1}}}}}}
2021-01-11T11:54:36.000+1100	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: request checking for ILM availability failed: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}
2021-01-11T11:54:36.000+1100	INFO	[publisher_pipeline_output]	pipeline/output.go:145	Attempting to reconnect to backoff(elasticsearch(https://vpc-xxxxxxx.ap-southeast-2.es.amazonaws.com:443)) with 6 reconnect attempt(s)

The issue was sorted, no help required. Thanks anyway!

Hi @francisca.munhoz
Can you share the fix? In case someone else has the same issue.

1 Like

Added:
setup.ilm.enabled: false
Fixed the issue.

1 Like