ERR Failed to connect: Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template: couldn't load template: couldn't load json. Error: 400 Bad Request


(Shubham Sureka) #1

I get this error when I deploy filebeat. it was tested before with 6.6.1 version and was working fine but now the same yaml is giving this error.


(Jaime Soriano) #2

Hi @ishu52 and welcome :slight_smile:

What version of filebeat are you using? Filebeat < 6.7.0 versions are incompatible with Elasticsearch 7.0.

You can check the supported versions matrix in https://www.elastic.co/es/support/matrix#matrix_compatibility


(Shubham Sureka) #3

the version was 6.0.1. Now I pointed to 7.0.0 and getting below error. Pod is going to 'CrashLoopBackOff' state


(Jaime Soriano) #4

As the error message says, filebeat.config.prospectors has been removed. Prospectors were removed to inputs and both options were accepted during a time. Now you have to use filebeat.inputs instead.

Take a look to the release notes, this and other breaking changes are listed there.


(Shubham Sureka) #5

Thank you @jsoriano.

It is working. but compared to previous version 6.0.1, with 7.0.0 i cannot see any data on fields like namespace, container name, requestid etc.

It is just showing index type, index id, message


(Shubham Sureka) #6

Hi,

I have removed filebeat.config.prospectors and added filebeat.input.

Now i want kubernetes log to be forwarded for which in prospectors config 'add_kubernetes_metadata' was added. with latest 7.0.0 ,I am adding preprocessor in filebeat.yml

Output: Not able to read logs from kubernetes pods

Expectation: read logs from kubernetes pods


(Jaime Soriano) #7

Hi @ishu52,

Are your pod logs in /var/data/kubeletlogs? Are these files accessible from the filebeat pod?

You can also use autodiscover to collect logs from all your pods, using autodiscover also adds the metadata to the messages of each pod automatically. But this may require further configuration if you don't have your logs in /var/lib/docker/containers.


(Shubham Sureka) #8

Yes logs are available at var/data/kubeletlogs and are getting harvested as well.

Since the cluster is containerd so logs are not coming to var/lib/docker/containers