Unable to push data into multiple indices

Elastic Stack Beats
1

I am trying to push different streams of windows events into different elastic indices based on the event.provider

But unfortunately, all my data is pushed into the default index.

I am not sure if the filtering is not working or the indices block is not set up properly

Here is my config

setup.template.name: "sysmonbeat-%{[agent.version]}"

setup.template.pattern: "sysmonbeat-%{[agent.version]}-*"

setup.ilm.rollover_alias: "sysmonbeat-%{[agent.version]}"

output.elasticsearch:

    hosts: ["URL"]

    username: "USERNAME"

    password: "PASSWORD"

    max_retries: 3

    indices:

        - index: 'sysmonbeat-%{[agent.version]}-%{+yyyy.MM.dd}'       

          when.equals:

            event.provider: 'Microsoft-Windows-Sysmon'

        - index: 'winlogbeat-%{[agent.version]}-%{+yyyy.MM.dd}'       # default winlogbeat index

processors:

    - add_tags:

        tags:

            - "TAG"

    - add_labels:

        labels:

            tag: "TAG"

            datacenter: "NONPROD"

    - drop_fields:

        fields:

            - event.kind

            - process     # drop the whole process subobject

            - agent     # drop the whole process subobject

            - winlog.api

            - winlog.channel

            - winlog.opcode

            - winlog.provider_guid

            - winlog.record_id

            - winlog.task

            - winlog.computer_name

            - winlog_event_data.Binary

            - winlog.event_data.Image

            - winlog.event_data.ProcessGuid

            - winlog.event_data.ProcessId

            - winlog.event_data.RuleName

            - winlog.event_data.UtcTime

            - winlog.event_id

            - winlog.process.pid

            - winlog.process.thread.id

            - winlog.record_id

            - winlog.user.domain

            - winlog.user.identifier

            - winlog.user.name

            - winlog.user.type

            - winlog.version

            - winlog.user.type

            - winlog.user.identifier

            - winlog.user.name

            - winlog.user.domain

winlogbeat.event_logs:

    -   name: Application

        ignore_older: 1s  # do not push previously existing events

    -   name: System

        ignore_older: 1s  # do not push previously existing events

    -   name: Microsoft-Windows-Sysmon/Operational

        event_id: 3

        ignore_older: 1s  # do not push previously existing events

        provider:

            - Microsoft-Windows-Sysmon

        processors:

            - drop_fields:

                fields:

                    - event.kind

                    - log.level

                    - message

                    - winlog.event_data.User

    -   name: Microsoft-Windows-Sysmon/Operational

        event_id: 22

        ignore_older: 1s  # do not push previously existing events

        provider:

            - Microsoft-Windows-Sysmon

        processors:

            - drop_fields:

                fields:

                    - event.kind

                    - log.level

                    - message

                    - winlog.event_data.User
2

Winlogbeat doesn't use a field named event.provider. There is a winlog.provider_name that should work.

3

Nope, that doesn't work for me

Updated indices block

     indices:
        - index: 'sysmonbeat-%{[agent.version]}-%{+yyyy.MM.dd}'       # More like an if else ladder
          when.equals:
            winlog.provider_name: 'Microsoft-Windows-Sysmon'
        - index: 'winlogbeat-%{[agent.version]}-%{+yyyy.MM.dd}'       # default winlogbeat index
4

I think you need to turn off ILM to be able to customize the target index like this. Try it with

setup.ilm.enabled: false

6

All data is now going to the default index rather the custom index

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.