Grok filter in logstash for more than one different lines

[root_linux]

I have logs in which there are different types of lines are present. And I want to write grok filter for each line. How I can I achieve it? My logs are as given below:

Oct  5 14:05:53 centos-8gb-nbg1-1 openvpn: Tue Oct  5 14:05:53 2021 TCP connection established with [AF_INET6]::ffff:192.168.10.1:55630
Oct  5 14:05:54 centos-8gb-nbg1-1 openvpn: Tue Oct  5 14:05:54 2021 192.168.10.1:55630 peer info: IV_GUI_VER=windows.NT.x86_64
Oct  5 14:05:54 centos-8gb-nbg1-1 openvpn: Tue Oct  5 14:05:54 2021 192.168.10.1:55630 peer info: IV_PLAT_VER=30_11_arm64-v8a_samsung_exynos9610_SM-A505F
Oct  5 14:31:44 centos-8gb-nbg1-1 openvpn: Tue Oct  5 14:01:44 2021 google/192.168.10.1:55418 Connection reset, restarting [0]

Please guide me how can I write grok filter for these lines.

[AquaX (Andreas Helmer)]

Use this website to help you write your patterns.
It looks like it's already semi-well structured so you may be able to do something like this:

%{MONTH} %{MONTHDAY} %{TIME} %{HOSTNAME:hostname} %{NOTSPACE:service} %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{GREEDYDATA:log_message}

{
"MONTH": [
[
"Oct",
"Oct"
]
],
"MONTHDAY": [
[
"5",
"5"
]
],
"TIME": [
[
"14:05:53",
"14:05:53"
]
],
"HOUR": [
[
"14",
"14"
]
],
"MINUTE": [
[
"05",
"05"
]
],
"SECOND": [
[
"53",
"53"
]
],
"hostname": [
[
"centos-8gb-nbg1-1"
]
],
"service": [
[
"openvpn:"
]
],
"DAY": [
[
"Tue"
]
],
"log_message": [
[
"2021 TCP connection established with [AF_INET6]::ffff:192.168.10.1:55630"
]
]
}

[root_linux]

Hi @AquaX , Thanks for your response.
I am beginner in ELK. So could you guide me how can write different grok patterns in logstash.conf for each line?

%{SYSLOGTIMESTAMP:time} %{HOSTNAME:hostname} %{SYSLOGHOST:service}: %{DAY:day} %{SYSLOGTIMESTAMP:connection_est_time} %{YEAR:year} %{IP:server_ip}:%{NUMBER:port} peer info: IV_GUI_VER=%{GREEDYDATA:client_version}
%{SYSLOGTIMESTAMP:time} %{HOSTNAME:hostname} %{SYSLOGHOST:service}: %{DAY:day} %{SYSLOGTIMESTAMP:conn_est_time} %{YEAR:year} %{IP:server_ip}:%{NUMBER:port} peer info: IV_PLAT_VER=%{GREEDYDATA:device_details}

I have written these grok pattern for 2nd and 3rd lines of above logs. How can I use it in logstash configuration file?

[Rios]

Yes, you can have multiple patterns per line.

Logstash.conf

filter{
    grok {
      patterns_dir => "/pathtopaterns/filepatterns.txt"
      match => { 
        "message" => [ "%{CUSTOM_1}", "%{CUSTOM_2}" ]
      }
    }
}

filepatterns.txt content:

# Pattern 1
CUSTOM_1 
%{SYSLOGTIMESTAMP:time} %{HOSTNAME:hostname} %{SYSLOGHOST:service}: %{DAY:day} %{SYSLOGTIMESTAMP:connection_est_time} %{YEAR:year} %{IP:server_ip}:%{NUMBER:port} peer info: IV_GUI_VER=%{GREEDYDATA:client_version}

# Pattern 2
CUSTOM_2 %{SYSLOGTIMESTAMP:time} %{HOSTNAME:hostname} %{SYSLOGHOST:service}: %{DAY:day} %{SYSLOGTIMESTAMP:conn_est_time} %{YEAR:year} %{IP:server_ip}:%{NUMBER:port} peer info: IV_PLAT_VER=%{GREEDYDATA:device_details}

Also you can you use "if" instead of multiple match:

if [field1] == "something" {
mutate {rename, update, replace ...}
}

[Badger]

You can match against an array of patterns.

grok {
    match => {
        "message" => [
            "%{SYSLOGTIMESTAMP:time} %{HOSTNAME:hostname} %{SYSLOGHOST:service}: %{DAY:day} %{SYSLOGTIMESTAMP:connection_est_time} %{YEAR:year} %{IP:server_ip}:%{NUMBER:port} peer info: IV_GUI_VER=%{GREEDYDATA:client_version}",
            "%{SYSLOGTIMESTAMP:time} %{HOSTNAME:hostname} %{SYSLOGHOST:service}: %{DAY:day} %{SYSLOGTIMESTAMP:conn_est_time} %{YEAR:year} %{IP:server_ip}:%{NUMBER:port} peer info: IV_PLAT_VER=%{GREEDYDATA:device_details}"
        ]
    }
}

If the lines start with the SYSLOGTIMESTAMP then I strongly recommend that you anchor your patterns to the start of line: "^%{SYSLOGTIMESTAMP:time} ...". This makes it much cheaper to evaluate them when they do not match, and if you need an array of patterns there will always be some that do not match.

[root_linux]

@Rios and @Badger, thanks for solution