How can ser grok if log have different content from the same equipment?

Elastic Stack Logstash
1 
Jun  2 16:45:49 tp-id01.test.corp zorp/scb_ssh[1379]: core.alerting(3): (svc/wwVRLvYoUMToHT2Xz9vEbj/ssh_alert:11/ssh): Audit event was recognized; rule='PatternMatcherRule', type='adp.event.command', pattern='user', actions_taken='LOG, NOTIFY, METADB', event_data='ls -l'
Jun  8 17:00:11 sc-id01.test.corp zorp/scb_ssh[1416]: ssh.auth(3): (svc/778EuyyZrBFgb3eLsyVsh3/ssh_orig_target:78/ssh): User authentication failure on server; username='Sc', gateway_user=''
Jun  9 11:18:43 tp-id01.test.corp restserver[871]: User authentication failed; realm='Balabit', username='admin'

The log has different contents when users use different act.

In fact ,the log types have more variable contents.

How can set grok if log have different content from the same equipment?

Do I must use multiple pattern to one on one parse these logs?

2

I would start with something like

    grok { match => { "message" => "^(?<[@metadata][timestamp]>.{15}) %{HOSTNAME:hostname} %{NOTSPACE:program}\[%{POSINT:pid}\]: (?<msg>[^;]+); %{GREEDYDATA:[@metadata][restOfLine]}" } }
    date { match => [ "[@metadata][timestamp]", "MMM  d HH:mm:ss", "MMM d HH:mm:ss" ] }
    kv { source => "[@metadata][restOfLine]" field_split_pattern => ", " }

which will get you

       "pid" => "1416",
  "hostname" => "sc-id01.test.corp",
       "msg" => "ssh.auth(3): (svc/778EuyyZrBFgb3eLsyVsh3/ssh_orig_target:78/ssh): User authentication failure on server",
  "username" => "'Sc',",
   "program" => "zorp/scb_ssh",
"@timestamp" => 2021-06-08T21:00:11.000Z

Note that the empty gateway_user is not added as a field. Or

       "pid" => "871",
     "realm" => "'Balabit',",
  "hostname" => "tp-id01.test.corp",
       "msg" => "User authentication failed",
  "username" => "admin",
   "program" => "restserver",

You may then want another grok to parse things out of the msg field.

3

Hi, Thank you for your responses.

I want to test @metadata but some grokdebug webs don't test the complete conf.d.

Have any webs can test ?

4

Then use logstash to test the config, not a site maintained by a third party that does not work like grok.

See here and here.

5

I use your pattern but have one problem.

event_data should contain ls -l , but the output is 'ls

{
             "rule" => "PatternMatcherRule",
             "host" => "idc-securitu-elk01",
          "message" => "<284>Jun  2 16:45:49 tp-id01.test.corp zorp/scb_ssh[1379]: core.alerting(3): (svc/wwVRLvYoUMToHT2Xz9vEbj/ssh_alert:11/ssh): Audit event was recognized; rule='PatternMatcherRule', type='adp.event.command', pattern='user', actions_taken='LOG, NOTIFY, METADB', event_data='ls -l'",
               "id" => "284",
             "time" => "Jun  2 16:45:49",
       "event_data" => "'ls",
       "@timestamp" => 2021-06-13T08:44:14.381Z,
         "hostname" => "tp-id01.test.corp",
    "actions_taken" => "LOG, NOTIFY, METADB",
             "type" => "adp.event.command",
              "msg" => "core.alerting(3): (svc/wwVRLvYoUMToHT2Xz9vEbj/ssh_alert:11/ssh): Audit event was recognized",
         "@version" => "1",
          "pattern" => "user",
          "program" => "zorp/scb_ssh",
              "pid" => "1379"
}

I have been use mutate and gsub like this:

filter{
  grok {
     match => { "message" => "<(?<id>\d+)>(?<time>.{15}) %{HOSTNAME:hostname} %{NOTSPACE:program}\[%{POSINT:pid}\][\:\;] (?<msg>[^;]+)[\;] %{GREEDYDATA:[@metadata][restOfLine]} " }
   }
  mutate{
    gsub => ["[@metadata][restOfLine]", "(\S+=)", ", \1"]
   }
   kv { source => "[@metadata][restOfLine]"
        field_split_pattern => ", "
        field_split => ","
   }
}

But the output still wrong.
How can I modify the filter ?
Thanks

6

If your fields can contain spaces then do not use a space in field_split_pattern. You can use field_split => "," instead, then use the trim_key option to remove the space.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.