Logstash unable to parse specific format of log

Elastic Stack Logstash
1

Hello

I am looking for some help since getting some headaches when trying to parse some logs

Raw logs

cs1Label=username cs1=/test@test.com cn1Label=actionSuccess cn1=1 deviceCustomDate1Label=userActionTime deviceCustomDate1=May 31 2023, 06:42:18 UTC cs2Label=malopID cs2= cs3Label=linkToMalop cs3= cs4Label=actionType cs4=Add cn2Label=affectedIOCCount cn2=1\n

What I would like at the end is having this format:

{
        "username" => "test@test.com",
        "actionSuccess" => "1",
        "userActionTime" => "May 31 2023, 06:42:18 UTC",
        "malopID" => "",
        "linkToMalop" => "",
        "actionType" => "Add",
        "affectedIOCCount" => "1"
}

Since the field name is the n-1 value and the field data is the n value, I have to use gsub with Kv filter but I don't find a way to achieve it.

Some help to put me on the right track would be very appreciated.

Best regards

2

What is your input? Please share the entire configuration pipeline.

This is the entire log or you have anything else? This is a CEF message, the CEF codec would be able to parse this easily.

3

Hello @leandrojmp , thank you for the reply.

Yes it is indeed CEF
Here is the entire raw logs:

<150>May 31 06:42:18 cyber-420 auditSyslogLogger CEF:0|Cyber|Cyber||UserAction|Malop Investigation/ManualCustomReputations|0|cs1Label=username cs1=/test@test.com cn1Label=actionSuccess cn1=1 deviceCustomDate1Label=userActionTime deviceCustomDate1=May 31 2023, 06:42:18 UTC cs2Label=malopID cs2= cs3Label=linkToMalop cs3= cs4Label=actionType cs4=Add cn2Label=affectedIOCCount cn2=1\n
4

And what is your configuration? Are you using the CEF codec?

5

I currently don't have anything in the filter since I don't really know how to start to parse these label and data fields.

Here is my pipeline, nothing fancy atm, and I didn't use the cef codec

input {
  syslog {
    port => 5020
    type => "edr"
  }
}

filter {
  if [type] == "edr" {
  }
}

output {
  if [type] == "edr" {
    file {
      path => "C:/edr.txt"
    }
  }

}
6

Add a CEF codec in your input and see if this help you with the parse.

input {
  syslog {
    port => 5020
    type => "edr"
    codec => "cef"
  }
}
7

@leandrojmp I edited the input with the cef codec plugin which managed to format the logs this way for the CEF part:

    "cef": {
        "device_custom_string_6": {
            "value": "test@test.com",
            "label": "username"
        },
        "device_custom_string_1": {
            "value": "",
            "label": "malopID"
        },
        "device_custom_string_2": {
            "value": "",
            "label": "linkToMalop"
        },
        "device_custom_string_4": {
            "value": "Add",
            "label": "actionType"
        }
    }
....

Is there any way to have something like this instead ?

"username": "test@test.com",
"malopID": "",
"linkToMalop": "",
"actionType": "Add"

Thank you in advance

8

@leandrojmp maybe writing a ruby script could help ?

9

If you have just a couple of fields you could solve this using a mutate filter.

Something like this

    mutate {
        rename => {
            "[cef][device_custom_string_1][value]" => "malopID"
            "[cef][device_custom_string_2][value]" => "linkToMalop"
            "[cef][device_custom_string_4][value]" => "actionType"
            "[cef][device_custom_string_6][value]" => "username"
        }
    }

But if you have a lot of those fields or they may change, a ruby script would work.

Not sure how the ruby code would look like, but if you search in the forum you will find some examples on how to do this.

10

Thank you for the answer @leandrojmp
I wish I could only have these field, the problem is I have irregular fields name but at the end of the day, they are all in the format "device_custom_XXXXXX"

11

You could try

    ruby {
        code => '
            cef = event.get("cef")
            if cef.is_a? Hash
                cef.each { |k, v|
                    if k =~ /^device_custom_(number|string)_\d+/ and v.is_a? Hash
                        event.set(v["label"], v["value"])
                    end
                }
                event.remove("cef")
            end
        '
    }
1 Like
12

Exactly what I was looking for, thank you very much @Badger !!!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.