Get the first line of [message] field coming from winlogbeat

Hello,

I'm trying to get the first line of the message field because this is an important information to keep.

The [message] line, when I set the output to file using rudydebug option looks like this:

 "message" => "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID: 62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -"

I would like to have this:
"An HTTP request was received. See audit 510 with the same Instance ID for headers."

Which is the first line before a \n.

In logstash I tried several solution I've found on this forum, but there is no one working:

  mutate {
    copy => { "message" => "message_head" }
  }

  mutate {
    gsub => [ "message_head", "^([^\n]*)$", "\1" ]
  }

I also tried this:

  mutate {
    copy => { "message" => "message_head" }
  }

  mutate {
    split => ["message_head", " 
"]}

  mutate {
    replace => { "message_head" => "%{message_head[0]}" }
  }

And:

mutate { split => [ "message_head" => "\n" ] }

The result is almost all the same; the 'split' or 'gsub' are just not being applied:

"message_head" => "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID: 62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -",
"message" => "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID:62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -",

In some solution I've found, people said it was working, such as this thread: https://discuss.elastic.co/t/add-winlogbeat-option-to-truncate-security-message-field-to-just-first-line/49409, but it is not the case on my side.

To see when I see into Kibana, please look at the attached screenshot.

You will notice the current filter, which is:

  mutate {
    copy => { "message" => "message_head" }
  }

  mutate { 
    gsub => [ "message_head", "^([^\n]*)$", "\1" ]
  }

... did nothing.

If I test my regular expression into a "regular expression tester", I can see that it is working:

So I'm out of solutions here...

Is there anybody who can help me out with this issue?

Thank you and best regards,
Yanick

Might be more efficient to do in Ruby. But here's how you can do it without.

Test Conf

input { generator { codec => json count => 1 lines => [ '{ "message" : "An HTTP request was received. See audit 510 with the same Instance ID for headers. \n\nInstance ID: 8ac2ca87-8958-4e0e-a79a-8be672341b5b \n\nActivity ID: 62f3ac59-1f9b-9254-97c8-b00a8bd00357 \n\nRequest Details: \n    Date And Time: 2021-11-09 14:15:16 \n    Client IP: 111.111.1.111 \n    HTTP Method: POST \n    Url Absolute Path: /adfs/oauth2/token/ \n    Query string: - \n    Local Port: 443 \n    Local IP: 100.132.4.89 \n    User Agent: Windows-AzureAD-Authentication-Provider/1.0 \n    Content Length: 4223 \n    Caller Identity: - \n    Certificate Identity (if any): - \n    Targeted relying party: - \n    Through proxy: False \n    Proxy DNS name: -" }' ] } }
filter {
    mutate {
       gsub => ["message", " \n\n", "::"]
    }
    mutate {
       split => {"message" => "::"}
       add_field => ["result", "%{[message][0]}"]
    }
}      
output {
  stdout { codec => json_lines }
}

Output

"result": "An HTTP request was received. See audit 510 with the same Instance ID for headers."
1 Like

That is a no-op. It replaces the first line with the first line. Try

mutate { gsub => [ "message_head", "(?m)^([^\n]*)$.*", "\1" ] }
2 Likes

Hi aaron.

I would like to know the ruby version as well. I tried both, but my knowledge are limited with this.

I will try you solution shortly.

Regards,
Yanick

Hi Badger,

Much thanks, your solution works. However I'm not sure when means the:

(?m)

After asking my friend Google, it told me that means "multi-line". Is it correct?

Thank you again for your help,
Yanick

I would go with Badgers suggestion vs Ruby. I was thinking it wasn't possible to target \n with gsub and might require Ruby to do it. I was wrong.

1 Like

That is correct. By default a regexp will not match multiple lines, so you have to use (?m) to make it do so.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.