Filebeat 6.7.0 not working with AWS Elasticsearch

Elastic Stack Beats
1

After upgrading filebeat to 6.7.0, I've noticed that it doesn't ship logs anymore. I'm using AWS managed Elasticsearch, which does not provide license information.

Related to: https://github.com/elastic/beats/pull/11296

filebeat.log:

2019-04-04T12:21:58.641Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://elk:80)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials

AWS Elasticsearch is configured to allow connections without authorisation from selected instances (access policy allows es:* calls).

In this setup, Elasticsearch does not allow to list xpack licenses:

$ curl -i elk/_xpack/license
HTTP/1.1 401 Unauthorized
Access-Control-Allow-Origin: *
Content-Type: application/json
x-amzn-RequestId: 760c652a-56d4-11e9-a7b5-8b0e14beb14d
Content-Length: 61
Connection: keep-alive

{"Message":"Your request: '/_xpack/license' is not allowed."}

Is there any way to run Filebeat (versions 6.7+) with AWS Elasticsearch service?

Best regards,
Marek Obuchowicz
KoreKontrol - managed cloud hosting for eCommerce

1 Like
2

You should be able to use the oss distribution.

3

This request will fail with AWS managed Elasticsearch as you need to run the Beats OSS versions, still the HTTP response returned by AWS is not correct if you say that authorization is no problem.

4

Thanks for the hint... Is it possible to get opensource packages also from any debian repository?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.