Filebeats does not create custom-named indices and hangs if ILM is disabled

[ppine7 (Marina Popova)]

All details and code snippets are included into the SO post: elasticsearch - Elastic Filebeat does not index into custom indices with mappings - Stack Overflow

Below is the summary with the latest updates:

I am trying to configure Filebeats to index events into a custom-named index with a custom mapping for some of the fields. The problem is that Filebeat does not send events to my index but tries to send them to the default ' filebeats-xxx ' index instead, and is failing with parsing/mapping exception since the events do not conform to the default filebeat event format.

See details below.

UPDATE 09/10/2021: after finding a similar issue reported/discussed here: https://github.com/elastic/beats/issues/11866 I tried to follow the workaround and added the following parameter to the filebeats.yml to disable ILM in ES:

setup.ilm.enabled: false

This did not help much - filebeat logs were saying that ILM is already set. So I've added one more parameter:

setup.ilm.overwrite: true

and after this - the errors stopped, but when I try to push events through - filebeat seems to be hung after the following logs:

    2021-09-09T21:42:58.057-0400    INFO    [publisher_pipeline_output] pipeline/output.go:143  Connecting to backoff(elasticsearch(https://testbeats1.es.us-central1.gcp.cloud.es.io:9243))
    2021-09-09T21:42:58.391-0400    INFO    [esclientleg]   eslegclient/connection.go:273   Attempting to connect to Elasticsearch version 7.14.1
    2021-09-09T21:42:58.495-0400    INFO    [publisher_pipeline_output] pipeline/output.go:151  Connection to backoff(elasticsearch(https://testbeats1.es.us-central1.gcp.cloud.es.io:9243)) established

I don't think it is a connectivity issue as the log says that a connection is established .... Any idea how to debug it further?

[rugenl (Len Rugen)]

It will work as you plan, there has to be a config error somewhere. Since you are getting an index created as "filebeats-xxx", if it has data from THIS HOST, it's not a connection error. (That name is odd, it's usually filebeat-xxx, no "s").

Verify that the template exists in elastic. I don't see what version you are using, there are new "index" and "component" templates to add confusion to what are now "legacy" templates. Make sure that a index exists that has the write-alias and that it has is_write_index": true.

As always, check the filebeat log and the elasticsearch log for related events.

[ppine7 (Marina Popova)]

sorry, the default indices for filebeat are "filebeat-xxx" (without 's' ) - that was my typo.
If I disable custom index name and templates settings - and send an event to Filebeat that is NOT JSON (and, thus, qualifies as a string and not an object) - the data does get into the filebeat-xxx index.

It is only when I try to use custom index + mappings and send events with JSON payload - I run into this issue.

There is nothing in Filebeat logs - other that what I showed. Filebeat just "hangs" - no more logs after this point, even when I send more messages to the input

My custom mapping does exist in ES - because I've created/PUT it there manually, and verified I can add an event into an index with this mapping - also manually, by POSTing the event into an index via Developer Tools console.

thank you for the quick response!

[ppine7 (Marina Popova)]

Hi, an update on the issue:
After I disabled the ILM in filebeat.yml
setup.ilm.enabled: false

and wiped out all templates.indices form ES - I was able to index events into my custom index name, specified as:

output.elasticsearch:
  enabled: true
  index: "marina-test2"

However, I also need to be able to rotate the indices daily, so I'd like to use the day pattern:

output.elasticsearch:
  enabled: true
  index: "marina-%{+yyyy.MM.dd}"

When I make this change and index events - they correctly go into a corresponding index, like:
marina-2021.09.21

but there are no aliases created and I believe no rotation will happen - since the ILM is disabled now....
Question: how do I have both custom index name and ILM enabled and setup for this index pattern?