Netscout arbor legacy syslog parsing - Audit log issue

Elastic Stack Logstash
1

Hi Folks ,
I need your help to resolve the issue below . I'm trying since many days to parse some audit logs .You will find some sample below . Idea is to simply extract these field .

Field Need to be extracted


           "received_from" => "1.1.1.1",
    "syslog_facility_code" => 1,
         "hostname" => "FW-DOS01",
          "syslog_message" => "user: kyloren, timestamp: 18:21 04/12/22, device: FW-DOS01, name: PG-FW-01-DD, action: Update, type: Protection Group, message: , descr_short: Updated Protection Group PG-FW-01-DD, descr_long: Updated Protection Group PG-FW-01-DD, url: https://FW-DOS01/audittrail?audit_component=25&audit_obj_id=161",
                    "type" => "syslog",
                    "host" => "1.1.1.1",
    "syslog_severity_code" => 6,
         "syslog_severity" => "informational",
              "@timestamp" => 2022-04-12T16:21:37.000Z,
             "received_at" => "2022-04-12T18:21:37.379Z",
              "syslog_pri" => "14",
        "syslog_timestamp" => "Apr 12 18:21:37",
          "syslog_program" => "NSI",
                 "message" => "<14>Apr 12 18:21:37 FW-DOS01 NSI: user: kyloren, timestamp: 18:21 04/12/22, device: FW-DOS01, name: PG-FW-01-DD, action: Update, type: Protection Group, message: test, descr_short: Updated Protection Group PG-FW-01-DD, descr_long: Updated Protection Group PG-FW-01-DD, url: https://FW-DOS01/audittrail?audit_component=25&audit_obj_id=161",
                 "user:" => "kyloren"
                "action:" => "Update"
                "message:" =>"test"
                "name:" => "PG-FW-01-DD"
                "type:" => "Protection Group"
                "descr_short:" => "Updated Protection Group PG-FW-01-DD"
                "descr_long:" => "Updated Protection Group PG-FW-01-DD"
                "url:" => "https://FW-DOS01/audittrail?audit_component=25&audit_obj_id=161"
                    "tags" => [
        [0] "FW-DD01"

Audit Log Sample

<14>Apr 13 11:06:22 FW-DOS01 NSI: user: kyloren, timestamp: 11:06 04/13/22, device: FW-DOS01, name: PG-FW-01-DD, action: Update, type: Protection Group, message: test , descr_short: Updated Protection Group PG-FW-01-DD, descr_long: Updated Protection Group PG-FW-01-DD, url: https://FW-DOS01/audittrail?audit_component=25&audit_obj_id=161

<14>Apr 13 10:33:01 FW-DOS01 NSI: user: kyloren, timestamp: 10:32 04/13/22, device: FW-DOS01, name: Updated server type ST-TEST-ED, action: Update, type: system, message: test , descr_short: Updated server type ST-FW-FFS  url: https://FW-DOS01/audittrail?audit_component=1&audit_obj_id=113

<14>Apr 13 11:09:22 FW-DOS01 NSI: user: kyloren, timestamp: 11:09 04/13/22, device: FW-DOS01, name: Backup, action: Edit, type: Backup, message: , descr_short: Created configuration backup, url: https://FW-DOS01/audittrail?audit_component=17&audit_obj_id=0

<14>Apr 13 11:08:22 FW-DOS01 NSI: user: kyloren, timestamp: 11:08 04/13/22, device: FW-DOS01, name: Disk, action: Delete, type: Filesystem, message: , descr_short: Deleted file from 1.1.1.1, descr_long: File: ext0ext1capture-lelynx.pcap, url: https://FW-DOS01/audittrail?audit_component=19&audit_obj_id=2

Logstash pipeline used

input {
  udp {
    port => 514
    type => syslog
    tags => ["FW-DD01"]

  }
}
filter {
  if "FW-DD01" in [tags] {
    grok {
      match => { "message" => "(<%{POSINT:priority}>)?%{SYSLOGTIMESTAMP:syslog_timestamp} ?(%{SYSLOGHOST:hostname})? %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri {
      syslog_pri_field_name => "syslog_pri"

    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
    elasticsearch {
              index => "auditAem-00001"
              hosts => ["https://es-node-01:9200"]
              ssl => true
              ssl_certificate_verification => true
              cacert => "/etc/logstash/elasticsearch-ca.pem"
              manage_template => true
              user => "elastic"
              password => ''
              codec => "plain"
                    }
}

I've tried to use KV but it's not working . can you please help me to find achieve my goal .

kv {
         source => "syslog_message"
         field_split => ","
         trim_key => "<>\[\], "
         trim_value => "<>\[\],"
2

Hi,

In the kv filter you need to set the value_split option because default is '='.
See here

Cad.

3

In addition to setting value_split => ":" you need to consider whether it makes sense for you to set `allow_empty_values => true'. In most of your events the original [message] field will be unchanged, but sometimes it will get overwritten with

         "message" => "test ",

You may want it to get overwritten with

         "message" => "",

when the original message has an empty message. (And you may want to change trim_value to include a space.)

4

Hi Badger ,
Thanks for your help , Below the kv filter that I'm using but I have some issue

 kv {
        source => "syslog_message"
        field_split => " "
        value_split => ":"
        trim_key => "\s"
}

The KV filter doesnt parse correctly these field , I think that it's stop parsing when meeting the first space . 


          "descr_long" => "Updated",
         "descr_short" => "Updated",

How the parsing supposed to be 

                "descr_short:" => "Updated Protection Group PG-FW-01-DD"
                "descr_long:" => "Updated Protection Group PG-FW-01-DD"

Is there any way to ignore space with KV ?
5

Your field split is not a space, it is a comma.

Try this kv

filter {
    kv {
        field_split => ","
        value_split => ":"
        trim_key => "\s"
    }
}

Running it for the message

user: kyloren, timestamp: 18:21 04/12/22, device: FW-DOS01, name: PG-FW-01-DD, action: Update, type: Protection Group, message: , descr_short: Updated Protection Group PG-FW-01-DD, descr_long: Updated Protection Group PG-FW-01-DD, url: https://FW-DOS01/audittrail?audit_component=25&audit_obj_id=161

It gives the following result:

{
      "timestamp" => "18:21 04/12/22",
           "name" => "PG-FW-01-DD",
           "user" => "kyloren",
    "descr_short" => "Updated Protection Group PG-FW-01-DD",
       "@version" => "1",
            "url" => "https://FW-DOS01/audittrail?audit_component=25&audit_obj_id=161",
     "@timestamp" => 2022-04-14T12:37:04.320Z,
         "action" => "Update",
        "message" => "user: kyloren, timestamp: 18:21 04/12/22, device: FW-DOS01, name: PG-FW-01-DD, action: Update, type: Protection Group, message: , descr_short: Updated Protection Group PG-FW-01-DD, descr_long: Updated Protection Group PG-FW-01-DD, url: https://FW-DOS01/audittrail?audit_component=25&audit_obj_id=161",
     "descr_long" => "Updated Protection Group PG-FW-01-DD",
         "device" => "FW-DOS01",
           "type" => "Protection Group"
}
6

Thanks it's working as expected .

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.