Help with Grok (syntax issue as well as question regarding double quotes)

Elastic Stack Logstash
1

This is a sample log that I want to parse:
type=EXECVE msg=audit(1684525987.999:148345): argc=2 a0="vim" a1="logstash-syslog.conf"

This is the grok filter I am trying:
type=%{WORD:type} msg=audit\(%{NUMBER:audit}\): argc=%{NUMBER:argc} a0="%{WORD:a0}" a1="%{DATA:a1}"

It doesn't seem to be working in online debuggers. Also assuming you can get it to work, can you also show the correct syntax in applying it in the Logstash config?

2

Hi @roman-tasi,

Your grok pattern doesn´t match.
I am not sure if you wanted the number part after the colon in the audit field.
This pattern excludes that number:

type\=%{WORD:type} msg\=audit\(%{NUMBER:audit}\:%{NUMBER}\)\: argc\=%{NUMBER:argc} a0\=\"%{WORD:a0}\" a1\=\"%{DATA:a1}\"

This results in:

[
  {
    "type": "EXECVE",
    "audit": 1684525987.999,
    "argc": 2,
    "a0": "vim",
    "a1": "logstash-syslog.conf"
  }
]

And this one includes it but it is no longer a number.:

type\=%{WORD:type} msg\=audit\((?<audit>[\d\.\:]*)\)\: argc\=%{NUMBER:argc} a0\=\"%{WORD:a0}\" a1\=\"%{DATA:a1}\"

It wil results in:

[
  {
    "type": "EXECVE",
    "audit": "1684525987.999:148345",
    "argc": 2,
    "a0": "vim",
    "a1": "logstash-syslog.conf"
  }
]
3

Also, see this thread for another approach to such messages.

2 Likes
4

Hi, I am trying this in my Logstash config:

if [log][file][path]=="/var/log/audit/audit.log" {
            kv {
                field_split => " "
                value_split => "="
                source => "message"
                trim_key => "\""
                trim_value => "\""
                remove_field => ["message"]
            }
    }

However it causes all logs from /var/log/audit/audit.log to not come into Kibana. Once I comment that code out, the logs start coming in again. What do you think its causing it to act like this? And is there a fix?

5

Also @Badger ->this<- seems to be the exact issue I am having, which you also commented on but was not resolved. I'm just curious on the solution!

6

Remove the current output and replace it with

output { stdout { codec => rubydebug } }

and see what the messages look like.

7

@Badger I think I already have been implementing that. Here's my current output in my Logstash config:

output {
        if [type]=="syslog" {
                if [m] in [ "1080", "745", "263", "1079" ] {
                elasticsearch {
                hosts => ["localhost:9200"]
                index => "account-access-%{+yyyy.MM}"
                }
                } else {
                elasticsearch {
                hosts => ["localhost:9200"]
                index => "syslog-%{+yyyy.MM.dd}"
                }
                }
        stdout { codec => rubydebug }
        }

        if [type]=="beats" {
                if [host][os][type]=="linux" {
                    if [host][name]=="mail.uhtasi.org" {
                        if [cmd]=="Auth"{
                            elasticsearch {
                            hosts => ["localhost:9200"]
                            index => "account-access-%{+yyyy.MM}"
                            }
                            } else {
                            elasticsearch {
                            hosts => ["localhost:9200"]
                            index => "zimbra-%{+yyyy.MM.dd}"
                            }
                        }
                    } else {
                        elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "linux-%{+yyyy.MM.dd}"
                        }
                    }
                }
                if [host][os][type]=="windows" or [agent][name]=="DCON2" or [agent][name]=="Dcon3" {
                    if [event][code] in [ "307", "4624", "4625", "4634", "4723", "4740", "4767", "11707" ] {
                        elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "account-access-%{+yyyy.MM}"
                        }
                    } else {
                        elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "winlogbeat-%{+yyyy.MM.dd}"
                        }
                    }
                }
        stdout { codec => rubydebug }
        }
}
8

Also here is some lines from the /var/log/messages file related to /var/log/audit/audit.log :

Jun  1 20:40:03 ELK-Stack filebeat: "path": "/var/log/audit/audit.log"
Jun  1 20:40:03 ELK-Stack filebeat: "path": "/var/log/audit/audit.log"
Jun  1 20:40:07 ELK-Stack filebeat: 2023-06-01T20:40:07.799-1000#011DEBUG#011[input.filestream]#011filestream/prospector.go:188#011File /var/log/audit/audit.log has been updated#011{"id": "my-filestream-id", "prospector": "file_prospector", "operation": "write", "source_name": "native::201877637-64768", "os_id": "201877637-64768", "new_path": "/var/log/audit/audit.log", "old_path": "/var/log/audit/audit.log"}
Jun  1 20:40:09 ELK-Stack filebeat: "message": "Jun  1 20:40:03 ELK-Stack filebeat: \"path\": \"/var/log/audit/audit.log\"",
Jun  1 20:41:07 ELK-Stack filebeat: 2023-06-01T20:41:07.092-1000#011DEBUG#011[input.filestream]#011filestream/filestream.go:131#011End of file reached: /var/log/audit/audit.log; Backoff now.#011{"id": "my-filestream-id", "source": "filestream::my-filestream-id::native::201877637-64768", "path": "/var/log/audit/audit.log", "state-id": "native::201877637-64768"}
Jun  1 20:41:37 ELK-Stack filebeat: 2023-06-01T20:41:37.103-1000#011DEBUG#011[input.filestream]#011filestream/filestream.go:131#011End of file reached: /var/log/audit/audit.log; Backoff now.#011{"id": "my-filestream-id", "source": "filestream::my-filestream-id::native::201877637-64768", "path": "/var/log/audit/audit.log", "state-id": "native::201877637-64768"}

Also notice its related to filebeat not logstash

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.