Get the first line of [message] field coming from winlogbeat

yquirion · November 9, 2021, 2:34pm

Hello,

I'm trying to get the first line of the message field because this is an important information to keep.

The [message] line, when I set the output to file using rudydebug option looks like this:

 "message" => "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID: 62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -"

I would like to have this:
"An HTTP request was received. See audit 510 with the same Instance ID for headers."

Which is the first line before a \n.

In logstash I tried several solution I've found on this forum, but there is no one working:

  mutate {
    copy => { "message" => "message_head" }
  }

  mutate {
    gsub => [ "message_head", "^([^\n]*)$", "\1" ]
  }

I also tried this:

  mutate {
    copy => { "message" => "message_head" }
  }

  mutate {
    split => ["message_head", " 
"]}

  mutate {
    replace => { "message_head" => "%{message_head[0]}" }
  }

And:

mutate { split => [ "message_head" => "\n" ] }

The result is almost all the same; the 'split' or 'gsub' are just not being applied:

"message_head" => "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID: 62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -",
"message" => "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID:62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -",

In some solution I've found, people said it was working, such as this thread: https://discuss.elastic.co/t/add-winlogbeat-option-to-truncate-security-message-field-to-just-first-line/49409, but it is not the case on my side.

To see when I see into Kibana, please look at the attached screenshot.

You will notice the current filter, which is:

  mutate {
    copy => { "message" => "message_head" }
  }

  mutate { 
    gsub => [ "message_head", "^([^\n]*)$", "\1" ]
  }

... did nothing.

If I test my regular expression into a "regular expression tester", I can see that it is working:

So I'm out of solutions here...

Is there anybody who can help me out with this issue?

Thank you and best regards,
Yanick

aaron-nimocks · November 9, 2021, 2:59pm

Might be more efficient to do in Ruby. But here's how you can do it without.

Test Conf

input { generator { codec => json count => 1 lines => [ '{ "message" : "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID: 62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -" }' ] } }
filter {
    mutate {
       gsub => ["message", " \n\n", "::"]
    }
    mutate {
       split => {"message" => "::"}
       add_field => ["result", "%{[message][0]}"]
    }
}      
output {
  stdout { codec => json_lines }
}

Output

"result": "An HTTP request was received. See audit 510 with the same Instance ID for headers."

Badger · November 9, 2021, 6:13pm

That is a no-op. It replaces the first line with the first line. Try

mutate { gsub => [ "message_head", "(?m)^([^\n]*)$.*", "\1" ] }

yquirion · November 9, 2021, 6:28pm

Hi aaron.

I would like to know the ruby version as well. I tried both, but my knowledge are limited with this.

I will try you solution shortly.

Regards,
Yanick

yquirion · November 9, 2021, 6:39pm

Hi Badger,

Much thanks, your solution works. However I'm not sure when means the:

(?m)

After asking my friend Google, it told me that means "multi-line". Is it correct?

Thank you again for your help,
Yanick

aaron-nimocks · November 9, 2021, 6:43pm

I would go with Badgers suggestion vs Ruby. I was thinking it wasn't possible to target \n with gsub and might require Ruby to do it. I was wrong.

Badger · November 9, 2021, 7:24pm

That is correct. By default a regexp will not match multiple lines, so you have to use (?m) to make it do so.

system · December 7, 2021, 7:25pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.