How do I know which filters apply to many types of logs?

Elastic Stack Logstash
1

I am currently receiving firewalls and IPSec VPN, WAF, IPS logs.
There are four types of logs: Firewall, IPSec VPN, WAF, and IPS.
I create an index pattern for each device's log and set it on the grok filter.
So how do I know which index pattern is the index pattern of which instrument log?
Or what configuration should I do?

2

Please show your current Logstash configuration.

By "index pattern", are you talking about the index patterns in Kibana?

3

Yes, I'm talking about the index pattern seen in Kivana.
The current logstash.conf settings are as follows, and I first set only the IPSec VPN pattern.

input {
udp {
port => "514"
type => "syslog"
}
}

filter {
grok {
% {DATA: user} ]% {DATA: machine_num}% {DATA} % {DATA: dst_zone} src =% {IP: src} dst =% {{DATA}} {% DATA}% {GREEDYDATA: msg} src zone =% {DATA: src_zone} % {DATA} port =% {DATA} session_id =% {DATA} dst-xlated =% {DATA} =% {DATA: session_id} reason =% {GREEDYDATA: reason} "}
}
}

output {
elasticsearch {
hosts => "localhost: 9200"
index => "logstash -% {+ YYYY.MM.dd}"
}
}

I created a pattern for the logs of two devices as below, and I want to set it in logstash.conf.
The first is a firewall, the second is a pattern of IPSec VPN.

  1. Firewall logs and patterns
  • Log :
    Time = 20 policy_id = 80 service = https proto = start_time = "2018-08-13 11:16:00" VM_F / W: NetScreen device_id = VM_F / W [Root] 6 src zone = Trust dst zone = Untrust action = Permit sent = 140 rcvd = 0 src = 172.30.11.61 dst = 180.87.4.153 src_port = 53714 dst_port = 443 src-xlated ip = 172.30.11.61 port = 53714 dst-xlated ip = 180.87.4.153 port = 443 session_id = 46123 reason = Close - AGE OUT
  • pattern :
    \ {% DATA} }% {DATA} \ {% DATA} \ {% DATA} % {DATA} ip =% {GREEDYDATA}% {DATA}% {GREEDYDATA: msg1} src zone =% {DATA: src_zone} dst zone =% {DATA: dst_zone}% {GREEDYDATA: msg2} port =% {DATA} session_id =% {DATA: session_id} reason =% {GREEDYDATA: reason}
  1. IPSec VPN logs and patterns
  • Log :
    The IPsec_VPN: NetScreen device_id = IPSec_VPN [Root] system-notification-00257 (traffic): start_time = "2018-08-13 11:16:18" duration = 0 policy_id = 331 service = dns proto = 17 src zone = VPN (KG) dst zone = Trust action = Permit sent = 0 rcvd = 0 src = 192.168.76.20 dst = 110.45.222.21 src_port = 55916 dst_port = 53 src-xlated ip = 192.168.76.20 port = 55916 dst-xlated ip = 110.45 .222.21 port = 53 session_id = 44128 reason = Creation
  • pattern :
    % {DATA} %% {DATA: device}}:% {DATA: machine_id}% {DATA} \ [% {DATA} {DATA} port =% {DATA}% {DATA}% {GREEDYDATA: msg} src zone =% {DATA: src_zone} dst zone =% {DATA: dst_zone}% {GREEDYDATA: msg2} % {DATA} session_id =% {DATA: session_id} reason =% {GREEDYDATA: reason}

If you show me a short description and an example, it will help me understand.
Thank you.

4

Sorry, I don't understand what you're asking.

5

I'll tell you again.
I received firewalls and IPS logs, and set the pattern in logstash.conf.
And I run the logstash.conf file, but I get an error message.
What's wrong?
Below is the logstash.conf file.

input {
udp {
port => "514"
type => "syslog"
}
}

filter {
grok {
match => { "message" => "<%{DATA}>%{DATA:device}: %{DATA:machine_id} %{DATA} [%{DATA:user}]%{DATA:machine_num}(%{WORD})%{DATA} %{GREEDYDATA:msg} src zone=%{DATA:src_zone} dst zone=%{DATA:dst_zone} src=%{IP:srcip} dst=%{IP:dstip} src_port=%{NUMBER:srcport} dst_port=%{NUMBER:dstport} src-xlated ip=%{DATA} port=%{DATA} dst-xlated ip=%{DATA} port=%{DATA} session_id=%{DATA:session_id} reason=%{GREEDYDATA:reason}" }
match => { "message" => "<%{DATA:field00}>[%{DATA:Machine_Name}] [%{DATA:Attack_Name}], [%{DATA:Time}], [%{DATA:Hacker}], [%{DATA:Victim}], [%{DATA:Protocol}], [%{DATA:Risk}], [%{DATA:Handling}], [%{DATA:Information}], [%{DATA:SrcPort}], [%{DATA:Hacktype}]" }
}
}

output {
elasticsearch {
hosts => "localhost:9200"
index => "logstash-%{+YYYY.MM.dd}"
}
}

(The first match is the firewall pattern and the second match is the IPS pattern.)
Thank you.

6

And I run the logstash.conf file, but I get an error message.

What error message?

7

This question was solved with the help of another internal employee.
(It was a process issue on the server)
This part is likely to end.
Thank you.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.