Different index for new filebeat module "bind9"

Elastic Stack Beats
1

Dear All,
My environment: ELK stack 7.9.2 on Debian buster
I wrote a new module called "bind9" which picks up the bind query logs from a DNS server. This is writing its data to the filebeat index. I would like to have this in a different dedicated index. There is a hint modifying /etc/filebeat/filebeat.yml like this in section:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  indices:
    - index: "bind9-1.0-000001"
      when.equals:
        event.module: "bind9"
    - default: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

But unfortunately it does not work. The documents are in filebeat-index and the event.module is "bind9" in the document. Also index "bind9-1.0-000001" exists.

Is there maybe a way to define output anywhere in a manifest file for this module ?

Kind regards
Hans

--

2

Hi!

According to https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#indices-option-es what you are trying to do is doable. Are you sure the condition you are using is correct? Maybe to try with a different one, so as to better debug it?

C.

3

Hi Chris,
many thank for your swift reply.
Unfortunately filebeat stoped working after midnight. Actually it was restarted by systemctl, but stoped again, and so on. I found information that ILM is enabled. I disabled it. When I GET /_ilm/status I receive

{
"operation_mode": "STOPPED"
}

But I still see the following line in daemon-log:

Oct 26 18:39:44 localhost filebeat[3256]: 2020-10-26T18:39:44.331+0100#011INFO#011[index-management]#011idxmgmt/std.go:184#011Set output.elasticsearch.index to 'filebeat-7.9.2' as ILM is enabled.

I changed the config to this:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  indices:
    - index: "bind9-1.0-*"
      when.equals:
        event.module: "bind9"
    - default: "filebeat-7.9.2"

But this does also not work. The filebeat process is terminating shortly after it starts. The only way to bring it up and running is to remove the indices if-definition completly.

Currently I am confronted with two issues:

  • ILM claims to be active, but it isn't
  • the index condition in filebeat.yml doesn't work at all

Kind regards
Hans

4

Hi!

Is there anything in your config that enables ILM? (https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html)
If so, please try to remove it and retry with a clean setup.

C.

5

Hi Chris,

I assume I have to look in /etc/filebeat/filebeat.reference.yml
There was a comment line

#setup.ilm.enabled: auto

I un-commented and changed it to

setup.ilm.enabled: false

and restarted the filebeat service.

But no change. filebeat is not running and in daemon log I can see:

Oct 28 21:22:01 localhost filebeat[6358]: 2020-10-28T21:22:01.834+0100#011DEBUG#011[beat]#011instance/beat.go:325#011Initializing output plugins
Oct 28 21:22:01 localhost filebeat[6358]: 2020-10-28T21:22:01.834+0100#011INFO#011[index-management]#011idxmgmt/std.go:184#011Set output.elasticsearch.index to 'filebeat-7.9.2' as ILM is enabled.
Oct 28 21:22:01 localhost filebeat[6358]: 2020-10-28T21:22:01.834+0100#011DEBUG#011[conditions]#011conditions/conditions.go:98#011New condition equals: map[event.module:{0 bind9 false}]

When I check with 'curl' I see:

nuc:root# curl -X GET http://localhost:9200/_ilm/status | jq 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    28  100    28    0     0  28000      0 --:--:-- --:--:-- --:--:-- 28000
{
  "operation_mode": "STOPPED"
}

Kind regards
Hans

6

Hi!

I see in a previous post that you modify /etc/filebeat/filebeat.yml. So this is the configuration file you are providing to Filebeat?

Filebeat can take configuration file as parameter on startup like: ./filebeat -c /etc/myconf_filebeat.yml.

C.

7

Hi Chris,

thanks for coming back.
As I made a copy of the original filebeat.yml before changing anything I could try with the original config. This works but of course not moving some documents into a different index. Then I changed just this part

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  indices:
    - index: "bind9-1.0"
      when.equals:
        event.module: "bind9"
    - default: "filebeat-7.9.2"

But filebeat process terminates. These are the last 3 lines:

2020-10-30T20:30:57.059+0100 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.9.2' as ILM is enabled.
2020-10-30T20:30:57.059+0100 INFO instance/beat.go:419 filebeat stopped.
2020-10-30T20:30:57.059+0100 ERROR instance/beat.go:951 Exiting: error initializing publisher: missing output.elasticsearch.indices.1.index
Exiting: error initializing publisher: missing output.elasticsearch.indices.1.index

So what is output.elasticsearch.indices.1.index ?
How could I create or define it ?

I tried als to set the default to filebeat-7.9.2-2020.10.12 or filebeat-7.9.2-2020.10.12-000001
But there is no change.

Kind regards
Hans

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.