Filebeats does not create custom-named indices and hangs if ILM is disabled

All details and code snippets are included into the SO post: elasticsearch - Elastic Filebeat does not index into custom indices with mappings - Stack Overflow

Below is the summary with the latest updates:

I am trying to configure Filebeats to index events into a custom-named index with a custom mapping for some of the fields. The problem is that Filebeat does not send events to my index but tries to send them to the default ' filebeats-xxx ' index instead, and is failing with parsing/mapping exception since the events do not conform to the default filebeat event format.

See details below.

UPDATE 09/10/2021: after finding a similar issue reported/discussed here: I tried to follow the workaround and added the following parameter to the filebeats.yml to disable ILM in ES:

setup.ilm.enabled: false

This did not help much - filebeat logs were saying that ILM is already set. So I've added one more parameter:

setup.ilm.overwrite: true

and after this - the errors stopped, but when I try to push events through - filebeat seems to be hung after the following logs:

    2021-09-09T21:42:58.057-0400    INFO    [publisher_pipeline_output] pipeline/output.go:143  Connecting to backoff(elasticsearch(
    2021-09-09T21:42:58.391-0400    INFO    [esclientleg]   eslegclient/connection.go:273   Attempting to connect to Elasticsearch version 7.14.1
    2021-09-09T21:42:58.495-0400    INFO    [publisher_pipeline_output] pipeline/output.go:151  Connection to backoff(elasticsearch( established

I don't think it is a connectivity issue as the log says that a connection is established .... Any idea how to debug it further?

It will work as you plan, there has to be a config error somewhere. Since you are getting an index created as "filebeats-xxx", if it has data from THIS HOST, it's not a connection error. (That name is odd, it's usually filebeat-xxx, no "s").

Verify that the template exists in elastic. I don't see what version you are using, there are new "index" and "component" templates to add confusion to what are now "legacy" templates. Make sure that a index exists that has the write-alias and that it has is_write_index": true.

As always, check the filebeat log and the elasticsearch log for related events.

sorry, the default indices for filebeat are "filebeat-xxx" (without 's' ) - that was my typo.
If I disable custom index name and templates settings - and send an event to Filebeat that is NOT JSON (and, thus, qualifies as a string and not an object) - the data does get into the filebeat-xxx index.

It is only when I try to use custom index + mappings and send events with JSON payload - I run into this issue.

There is nothing in Filebeat logs - other that what I showed. Filebeat just "hangs" - no more logs after this point, even when I send more messages to the input

My custom mapping does exist in ES - because I've created/PUT it there manually, and verified I can add an event into an index with this mapping - also manually, by POSTing the event into an index via Developer Tools console.

thank you for the quick response!

Hi, an update on the issue:
After I disabled the ILM in filebeat.yml
setup.ilm.enabled: false

and wiped out all templates.indices form ES - I was able to index events into my custom index name, specified as:

  enabled: true
  index: "marina-test2"

However, I also need to be able to rotate the indices daily, so I'd like to use the day pattern:

  enabled: true
  index: "marina-%{+yyyy.MM.dd}"

When I make this change and index events - they correctly go into a corresponding index, like:

but there are no aliases created and I believe no rotation will happen - since the ILM is disabled now....
Question: how do I have both custom index name and ILM enabled and setup for this index pattern?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.