[@metadata][field] not behaving as expected

Elastic Stack Logstash
#1

I am using [@metadata] fields for filtering and outputs. I have an issue with how the field is behaving when there are two @metadata fields with different names. Below is a stripped down config I am using for testing:

input {
  tcp {
    codec => json_lines { charset => "CP1252" }
    port => 80
    type => "eventlog"
    add_field => { "[@metadata][source]" => "sysmon_raw" }
    ssl_enable => false
  }
}
filter{
  if [@metadata][source] == "sysmon_raw" {
    if [Channel] != "Microsoft-Windows-Sysmon/Operational" {
      mutate {
        add_field => { "[@metadata][eventlog]" => "eventlog_raw" }
      }
    }
  }
}
output {
  if [@metadata][eventlog] == "eventlog_raw" {
    stdout { codec => rubydebug { metadata => true } }
  }
}

This works as expected and data with [@metadata][eventlog] gets printed to screen. But when I use an output to account for both @metadata fields, the data goes into both outputs.

Example output:

output {
  if [@metadata][source] == "sysmon_raw" {
    kafka {
      bootstrap_servers => "host1.com:9092"
      topic_id => "sysmon_raw"
      compression_type => "lz4"
      codec => "json"
    }
  }
  if [@metadata][eventlog] == "eventlog_raw" {
    kafka {
      bootstrap_servers => "host2.com:9092"
      topic_id => "win_eventlog_raw"
      compression_type => "lz4"
      codec => "json"
    }
  }
}

The data from the [@metadata][eventlog] field foes into the [@metadata][source] output, but not the opposite way around. How can I use the @metadata field to ensure only the data from each field goes into the correct output. I have tried removing the [@metadata][source] field but no luck removing it.

Example of the output of the data:

{
                  "Task" => 0,
              "Keywords" => -9223372036854775808,
               "Message" => "%SYSTEM32%\\LOCATIONNOTIFICATIONWINDOWS.EXE was allowed to run.",
             "EventType" => "INFO",
              "UserData" => "",
                "Opcode" => "Info",
                  "type" => "eventlog",
         "SeverityValue" => 2,
               "Version" => 0,
                "UserID" => "S-1-5-21-3710767279-2211127119-3867209129-1003",
              "@version" => "1",
                  "host" => "10.223.57.0",
     "ExecutionThreadID" => 25072,
           "OpcodeValue" => 0,
      "SourceModuleType" => "im_msvistalog",
             "@metadata" => {
        "eventlog" => "eventlog_raw",
          "source" => "sysmon_raw"
    },
               "Channel" => "Microsoft-Windows-AppLocker/EXE and DLL",
              "Hostname" => "hostname.com",
            "SourceName" => "Microsoft-Windows-AppLocker",
    "ExecutionProcessID" => 996,
              "Severity" => "INFO",
           "AccountType" => "User"
     "EventReceivedTime" => "2017-12-12 14:38:32",
      "SourceModuleName" => "eventlogs",
          "ProviderGuid" => "{CBDA4DBF-8D5D-4F69-9578-BE14AA540D22}",
            "@timestamp" => 2017-12-12T19:40:58.149Z,
                  "port" => 65524,
             "EventTime" => "2017-12-12 14:38:30",
               "EventID" => 8002,
                "Domain" => "domain.name",
          "RecordNumber" => 334,
           "AccountName" => "user"
#2

Perhaps

if [@metadata][eventlog] == "eventlog_raw" {
  ...
} else if [@metadata][source] == "sysmon_raw" {
  ...
}

is what you're looking for, i.e. send eventlog_raw events to one output, otherwise send it to another output if it's a sysmon_raw event, but never sent events to both outputs.

#3

Yes I am looking to do that exact thing, send each to their own output but never to both. What that logic above souldn't it still out eventlog_raw data in sysmon_raw since all [@metadata][source] data contains sysmon_raw and eventlog_raw fields.

if [@metadata][source] == "sysmon_raw" { 
  if [Channel] != "Microsoft-Windows-Sysmon/Operational" { 
    mutate { add_field => { "[@metadata][eventlog]" => "eventlog_raw" } } 
  } 
}

So when it outputs any event with sysmon_raw it will contain all eventlog_raw data. How can I drop the [@metadata][source] for any data that meets the criteria above to receive the [@metadata][eventlog] field so that the value sysmon_raw doesn't even exist in that data.

#4

How can I drop the [@metadata][source] for any data that meets the criteria above to receive the [@metadata][eventlog] field so that the value sysmon_raw doesn't even exist in that data.

That's of course easy to do, but what's the point? If [@metadata][eventlog] is "eventlog_raw" then the event won't reach the sysmon_raw topic. Removing [@metadata][source] won't make a difference.

#5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.