Reopening this as my previous entry was closed for what I consider as a wrong reason
not leading to any helpful answer/solution
As detailed in Unable to use of beats (here journalbeat) with AWS ElasticSearch 7.x · Issue #30192 · elastic/beats · GitHub
I used to have a perfectly working AWS ES 6.8 + journalbeat OSS 7.12.1 setup
I also know about the breaking change of beats 7.13.0 which prevented me from upgrading my version of journalbeat
and I am therefore in the process of upgrading my AWS ES to 7.x and therefore be able to upgrade (at least I thought I could)
I have read a lot about such issues in different posts and have tried to play with stuff like adding the following to my
working journalbeat.yml file
setup.ilm.enabled: false
setup.pack.security.enabled: false
setup.xpack.graph.enabled: false
setup.xpack.watcher.enabled: false
setup.xpack.monitoring.enabled: false
setup.xpack.reporting.enabled: false
and also tried each and every of the following:
Journalbeat OSS 7.15.2 November 10, 2021
Journalbeat OSS 7.15.1 October 14, 2021
Journalbeat OSS 7.15.0 September 22, 2021
Journalbeat OSS 7.14.2 September 21, 2021
Journalbeat OSS 7.14.1 September 01, 2021
Journalbeat OSS 7.14.0 August 03, 2021
Journalbeat OSS 7.13.4 July 20, 2021
Journalbeat OSS 7.13.3 July 07, 2021
Journalbeat OSS 7.13.2 June 14, 2021
Journalbeat OSS 7.13.1 June 02, 2021
Journalbeat OSS 7.13.0 May 25, 2021
cross the following AWS Elasticsearch versions from 6.8 (where this works) up to 7.10 and all of the 7.x versions have the issue:
2022-02-03T17:25:01.873Z INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.1.1
2022-02-03T17:25:02.929Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://vpc-logs-u4unartdme74hqwzn7bmrgzfku.eu-west-1.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: unauthorized access, could not connect to the xpack endpoint, verify your credentials
2022-02-03T17:25:02.929Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://vpc-logs-u4unartdme74hqwzn7bmrgzfku.eu-west-1.es.amazonaws.com:443)) with 1 reconnect attempt(s)
2022-02-03T17:25:02.930Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-02-03T17:25:02.930Z INFO [publisher] pipeline/retry.go:223 done
One thing that puzzles me also is the fact that if on this failing setup I download and use journalbeat 7.12.1, use it temporarily, kill it and relaunch the failing journalbeat version > 7.12.1 then the failing version is not failing any more (everything else the same)
Could someone please shed some light on this obscure behaviour and how to fix it properly and definitively ?
Many thanks in advance.
)