Unable to use of beats (here journalbeat) with AWS ElasticSearch 7.10

Reopening this as my previous entry was closed for what I consider as a wrong reason
not leading to any helpful answer/solution

As detailed in Unable to use of beats (here journalbeat) with AWS ElasticSearch 7.x · Issue #30192 · elastic/beats · GitHub

I used to have a perfectly working AWS ES 6.8 + journalbeat OSS 7.12.1 setup

I also know about the breaking change of beats 7.13.0 which prevented me from upgrading my version of journalbeat
and I am therefore in the process of upgrading my AWS ES to 7.x and therefore be able to upgrade (at least I thought I could)

I have read a lot about such issues in different posts and have tried to play with stuff like adding the following to my
working journalbeat.yml file

setup.ilm.enabled: false
setup.pack.security.enabled: false
setup.xpack.graph.enabled: false
setup.xpack.watcher.enabled: false
setup.xpack.monitoring.enabled: false
setup.xpack.reporting.enabled: false

and also tried each and every of the following:

Journalbeat OSS 7.15.2 November 10, 2021
Journalbeat OSS 7.15.1 October 14, 2021
Journalbeat OSS 7.15.0 September 22, 2021
Journalbeat OSS 7.14.2 September 21, 2021
Journalbeat OSS 7.14.1 September 01, 2021
Journalbeat OSS 7.14.0 August 03, 2021
Journalbeat OSS 7.13.4 July 20, 2021
Journalbeat OSS 7.13.3 July 07, 2021
Journalbeat OSS 7.13.2 June 14, 2021
Journalbeat OSS 7.13.1 June 02, 2021
Journalbeat OSS 7.13.0 May 25, 2021

cross the following AWS Elasticsearch versions from 6.8 (where this works) up to 7.10 and all of the 7.x versions have the issue:

2022-02-03T17:25:01.873Z	INFO	[esclientleg]	eslegclient/connection.go:273	Attempting to connect to Elasticsearch version 7.1.1
2022-02-03T17:25:02.929Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(https://vpc-logs-u4unartdme74hqwzn7bmrgzfku.eu-west-1.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: unauthorized access, could not connect to the xpack endpoint, verify your credentials
2022-02-03T17:25:02.929Z	INFO	[publisher_pipeline_output]	pipeline/output.go:145	Attempting to reconnect to backoff(elasticsearch(https://vpc-logs-u4unartdme74hqwzn7bmrgzfku.eu-west-1.es.amazonaws.com:443)) with 1 reconnect attempt(s)
2022-02-03T17:25:02.930Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2022-02-03T17:25:02.930Z	INFO	[publisher]	pipeline/retry.go:223	  done

One thing that puzzles me also is the fact that if on this failing setup I download and use journalbeat 7.12.1, use it temporarily, kill it and relaunch the failing journalbeat version > 7.12.1 then the failing version is not failing any more (everything else the same)

Could someone please shed some light on this obscure behaviour and how to fix it properly and definitively ?

Many thanks in advance.

Elasticsearch version 7.1 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

I AM NOT USING ES 7.1 but 7.10 please fix your bot

Any insights on this issue ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.