Set indices based on (sub) module name

I'm trying to send my log files into multiple indices based on a sub (module) value. With the code below it works for the main module.

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  indices:
    - index: "ModuleA-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "Module"

However, in the structure I use, there are (sub) modules? I am not sure how to call them or how to use them to sent them to an index.:

C:\ELK\filebeat-7.11.1-windows-x86_64\module\Module
C:\ELK\filebeat-7.11.1-windows-x86_64\module\Module\A
C:\ELK\filebeat-7.11.1-windows-x86_64\module\Module\B
C:\ELK\filebeat-7.11.1-windows-x86_64\module\Module\C
C:\ELK\filebeat-7.11.1-windows-x86_64\module\Module\D

I would like a separate index for A, B, C and D.

How can I do this? I been trying for a long time but looks like I am stuck.

Where are the module and sub module names come from? If you have a field for the submodule name you could include it in the when condition.

I created the modules and names myself. I am trying to parse custom logging, but would like to get them into different indexes.

So in my module.d I have the following:

- module: Example
  
  ExampleA:
    enabled: true
    var.paths: ["D:/example/*ExampleA*.csv"]

  ExampleB:
    enabled: true
    var.paths: ["D:/Example/*Exampleb*.csv"]
    
  ExampleC:
    enabled: true
    var.paths: ["D:/Example/ExampleC*.csv"]

  ExampleD:
    enabled: true
    var.paths: ["D:/Example/ExampleD*.csv"]

This is monitoring a folder, where the logging will be dumped into and based on the filename the right module will trigger. This all works, but now all logging will show up in the index filebeat.

I would like to have the output of the submodules in a different index,.

I tried things the things below, but only the main module "example" seems to work .

when.equals:
        event.module: "example.exampleA"

or

 when.equals:
        event.module: "example_exampleA"

or 

when.equals:
        event.module: "exampleA"

You're talking about the datasets/filesets. Use event.dataset which is in the format of <module>.<filesets> like apache.access.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.