Do filebeat connected to logstash if it got the error (how it can be fixed?):
2021-03-14T15:06:08.306Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://vpc-automation-testing.eu-west-1.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license from the /_license endpoint, Filebeat requires the default distribution of Elasticsearch. Please make the endpoint accessible to Filebeat so it can verify the license.: unauthorized access, could not connect to the xpack endpoint, verify your credentials
full log:
2021-03-14T15:06:08.306Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
2021-03-14T15:06:08.306Z INFO [publisher] pipeline/retry.go:217 done
2021-03-14T15:06:08.306Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://vpc-qa-automation-testing-z754oqnaxm7zm75dtrcnlm2ygm.eu-west-1.es.amazonaws.com:443)) with 626 reconnect attempt(s)
2021-03-14T15:06:08.309Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.9.1
2021-03-14T15:06:17.286Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:17.289Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5/*.log]
2021-03-14T15:06:17.290Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:17.290Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5/*.log]
2021-03-14T15:06:17.290Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:17.290Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5/*.log]
2021-03-14T15:06:17.290Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:17.290Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5/*.log]
2021-03-14T15:06:17.290Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5-json.log
2021-03-14T15:06:17.290Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:17.290Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5/*.log]
2021-03-14T15:06:17.290Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:17.291Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/eefbe03007f9142de47f2528e8b8b14db6fde96af030e36ddad882c40d8491e5/*.log]
2021-03-14T15:06:18.048Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:18.049Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/735f372760adec78e5486f2f87f15e8e0cadf4ec171c76b49ea144a65021782e/*.log]
2021-03-14T15:06:18.049Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:18.049Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/735f372760adec78e5486f2f87f15e8e0cadf4ec171c76b49ea144a65021782e/*.log]
2021-03-14T15:06:18.050Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/735f372760adec78e5486f2f87f15e8e0cadf4ec171c76b49ea144a65021782e/735f372760adec78e5486f2f87f15e8e0cadf4ec171c76b49ea144a65021782e-json.log
2021-03-14T15:06:18.666Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:18.667Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/20f6b7c4035839475f89c4efa3c2477c1990316755c3fd286d894d68c11112f4/*.log]
2021-03-14T15:06:18.667Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:18.667Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/20f6b7c4035839475f89c4efa3c2477c1990316755c3fd286d894d68c11112f4/*.log]
2021-03-14T15:06:18.667Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/20f6b7c4035839475f89c4efa3c2477c1990316755c3fd286d894d68c11112f4/20f6b7c4035839475f89c4efa3c2477c1990316755c3fd286d894d68c11112f4-json.log
2021-03-14T15:06:19.930Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:19.930Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/a86c8945025191273d72e7895875c1c06ddf11c56a82a67939c564bb00b42112/*.log]
2021-03-14T15:06:19.930Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2021-03-14T15:06:19.931Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/a86c8945025191273d72e7895875c1c06ddf11c56a82a67939c564bb00b42112/*.log]
2021-03-14T15:06:19.931Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/a86c8945025191273d72e7895875c1c06ddf11c56a82a67939c564bb00b42112/a86c8945025191273d72e7895875c1c06ddf11c56a82a67939c564bb00b42112-json.log
2021-03-14T15:06:28.206Z INFO input/input.go:136 input ticker stopped
2021-03-14T15:06:29.498Z INFO input/input.go:136 input ticker stopped
2021-03-14T15:06:34.948Z INFO input/input.go:136 input ticker stopped
2021-03-14T15:06:35.477Z INFO input/input.go:136 input ticker stopped
Filebeat.yml:
filebeat.autodiscover:
providers:
- type: docker
templates:
- condition:
not:
contains:
docker.container.image: filebeat
config:
- type: docker
containers.ids:
- "${data.docker.container.id}"
logging.metrics.enabled: false
output.elasticsearch:
ssl.verification_mode: none
hosts: ["vpc-automation-testing.eu-west-1.es.amazonaws.com:443"]
protocol: "https"