All details and code snippets are included into the SO post: elasticsearch - Elastic Filebeat does not index into custom indices with mappings - Stack Overflow
Below is the summary with the latest updates:
I am trying to configure Filebeats to index events into a custom-named index with a custom mapping for some of the fields. The problem is that Filebeat does not send events to my index but tries to send them to the default ' filebeats-xxx ' index instead, and is failing with parsing/mapping exception since the events do not conform to the default filebeat event format.
See details below.
UPDATE 09/10/2021: after finding a similar issue reported/discussed here: https://github.com/elastic/beats/issues/11866 I tried to follow the workaround and added the following parameter to the filebeats.yml to disable ILM in ES:
This did not help much - filebeat logs were saying that ILM is already set. So I've added one more parameter:
and after this - the errors stopped, but when I try to push events through - filebeat seems to be hung after the following logs:
2021-09-09T21:42:58.057-0400 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://testbeats1.es.us-central1.gcp.cloud.es.io:9243)) 2021-09-09T21:42:58.391-0400 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1 2021-09-09T21:42:58.495-0400 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://testbeats1.es.us-central1.gcp.cloud.es.io:9243)) established
I don't think it is a connectivity issue as the log says that a connection is established .... Any idea how to debug it further?