Kibana dashboard creation issue

vipeen · December 29, 2021, 2:43pm

Hi, I am trying to move logs from a Gateway server to Syslog and then to Elasticsearch and finally build dashboards on Kibana. I am newbie and try to do my best to get this completed. I actually got this task from an ex-colleague who left the company all of a sudden. We have logs on Elasticsearch, but we dont get all the fields in the event while creating dashboard panel. As you can see from the image, the "Message" field container other fields inside it. I need to split them individually to create dashboard panels. Can anyone guide me how to do that? Thank you.

Iker · December 29, 2021, 4:09pm

You have to parse the fields, it could be done with a pipeline or if you are using logstash with the filters; this is a good place to start

vipeen · December 29, 2021, 6:30pm

Hi lker, I tried using Grok pattern inside the logstash.conf file. But, still the same. I think the problem might be the grok pattern, but not sure what exactly is wrong.

Iker · December 29, 2021, 6:51pm

Please post multiple sample of the messages to determine exactly what pattern could be a good fit for your use case; it seems to be from a Firewall, but I don't know which one specifically.

vipeen · December 30, 2021, 12:17pm

Sorry, I dint understand. Are you asking the messages from event list? Just like the earlier picture attached? I am still attaching an event from event list and the Grok pattern that is used inside the /opt/bitnami/logstash/pipeline/logstash.conf.

Iker · December 30, 2021, 2:34pm

Yes, but... as text, so people could test the grok patterns without having to type by hand the messages.

vipeen · December 30, 2021, 4:30pm

Understood, sorry my bad.

filter {

grok {

match => { "message" => <%{INT:syslog_pri}>\s*%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:logging_host}\s+%{PROG:logging_process}(?:[%{INT:logging_pid}])?:\s+[%{DATA:version}]\s+class="%{DATA:class}"\s+category="%{DATA:class}"\s+ctx="%{DATA:ctx}"\s+src="%{IP:src}"\s+srcPort="%{POSINT:srcPort}"\s+dest="%{IP:dest}"\s+destPort="%{POSINT:destPort}"\s+identity="%{DATA:identity}"\s+gwAction="%{DATA:gwAction}"\s+gwMode="%{DATA:gwMode}"}

}

stephenb · December 30, 2021, 4:36pm

We need txt samples of the log lines as well that is what @Iker is looking for.

vipeen · December 30, 2021, 4:43pm

@timestamp:

Nov 30, 2021 @ 22:33:37.383

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:37 BAGW-R kernel: [BlackRidge|Gateway|5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

wJnNcX0BxLs4OOovRAUN

_index:

logstash-2021.11.30

_score:

vipeen · December 30, 2021, 4:48pm

Is this the correct format? I copied them directly to editors and here, but I get this format.

Iker · December 30, 2021, 5:19pm

Yeah, it works; I requested you several samples, because in my experience there are a multiple logs (auth, vpn connection, filter, allow, deny, and so on; are very different) generated by a Firewall. The following grok expression works for the log sample that you provided, it's based in the one that you had, only required a couple of tweaks here and there; however it's very possible that other messages requires some other expression:

<%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:logging_host}\s+%{PROG:logging_process}(?:[%{INT:logging_pid}])?:\s+\[%{DATA:version}]\s+class="%{DATA:class}"\s+category="%{DATA:category}"\s+ctx="%{DATA:ctx}"\s+filterNumber="%{POSINT:filterNumber}"\s+src="%{IP:src}"\s+srcPort="%{POSINT:srcPort}"\s+dest="%{IP:dest}"\s+destPort="%{POSINT:destPort}"\s+gwAction="%{DATA:gwAction}"\s+gwMode="%{DATA:gwMode}"

This Grok debugger could be of help:

https://grokdebug.herokuapp.com

vipeen · December 30, 2021, 5:28pm

Hi, sorry, am pasting 10 events now.

Nov 30, 2021 @ 22:33:37.383 @timestamp:

Nov 30, 2021 @ 22:33:37.383

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:37 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

wZnNcX0BxLs4OOovRAUN

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:33:37.383 @timestamp:

Nov 30, 2021 @ 22:33:37.383

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:37 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

wJnNcX0BxLs4OOovRAUN

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:33:26.271 @timestamp:

Nov 30, 2021 @ 22:33:26.271

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:26 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

v5nNcX0BxLs4OOovGAWm

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:33:26.271 @timestamp:

Nov 30, 2021 @ 22:33:26.271

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:26 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

vpnNcX0BxLs4OOovGAWm

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:33:15.159 @timestamp:

Nov 30, 2021 @ 22:33:15.159

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:15 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

vZnMcX0BxLs4OOov7QU9

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:33:15.159 @timestamp:

Nov 30, 2021 @ 22:33:15.159

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:15 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

vJnMcX0BxLs4OOov7QU9

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:33:04.051 @timestamp:

Nov 30, 2021 @ 22:33:04.051

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:04 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

u5nMcX0BxLs4OOovwQXZ

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:33:04.051 @timestamp:

Nov 30, 2021 @ 22:33:04.051

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:04 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

upnMcX0BxLs4OOovwQXZ

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:32:52.955 @timestamp:

Nov 30, 2021 @ 22:32:52.955

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:02:52 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

uZnMcX0BxLs4OOovlgWC

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:32:52.955 @timestamp:

Nov 30, 2021 @ 22:32:52.955

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

uJnMcX0BxLs4OOovlgWC

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:32:52.939 @timestamp:

Nov 30, 2021 @ 22:32:52.939

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:02:52 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="56277" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

t5nMcX0BxLs4OOovlgWC

_index:

logstash-2021.11.30

_score:

_type:

_doc
Nov 30, 2021 @ 22:32:52.939 @timestamp:

Nov 30, 2021 @ 22:32:52.939

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="56277" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

tpnMcX0BxLs4OOovlgWC

_index:

logstash-2021.11.30

_score:

_type:

_doc

stephenb · December 30, 2021, 5:43pm

@vipeen We meant what you raw log lines.

To format your code / text etc please select it and press the format button.

Example

<4>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="56277" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

vipeen · December 30, 2021, 5:47pm

Nov 30, 2021 @ 22:33:37.383 @timestamp:

Nov 30, 2021 @ 22:33:37.383

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:37 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

wJnNcX0BxLs4OOovRAUN

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:33:37.383 @timestamp:

Nov 30, 2021 @ 22:33:37.383

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:37 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

wZnNcX0BxLs4OOovRAUN

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:33:26.271 @timestamp:

Nov 30, 2021 @ 22:33:26.271

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:26 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

vpnNcX0BxLs4OOovGAWm

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:33:26.271 @timestamp:

Nov 30, 2021 @ 22:33:26.271

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:26 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

v5nNcX0BxLs4OOovGAWm

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:33:15.159 @timestamp:

Nov 30, 2021 @ 22:33:15.159

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:15 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

vJnMcX0BxLs4OOov7QU9

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:33:15.159 @timestamp:

Nov 30, 2021 @ 22:33:15.159

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:15 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

vZnMcX0BxLs4OOov7QU9

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:33:04.051 @timestamp:

Nov 30, 2021 @ 22:33:04.051

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:03:04 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

upnMcX0BxLs4OOovwQXZ

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:33:04.051 @timestamp:

Nov 30, 2021 @ 22:33:04.051

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:03:04 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

u5nMcX0BxLs4OOovwQXZ

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:32:52.955 @timestamp:

Nov 30, 2021 @ 22:32:52.955

@version:

1

host:

192.168.120.135

host.keyword:

192.168.120.135

message:

<4>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

uJnMcX0BxLs4OOovlgWC

_index:

logstash-2021.11.30

_score:

-

_type:

_doc
Nov 30, 2021 @ 22:32:52.955 @timestamp:

Nov 30, 2021 @ 22:32:52.955

@version:

1

host:

192.168.120.145

host.keyword:

192.168.120.145

message:

<4>Nov 30 17:02:52 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"

_id:

uZnMcX0BxLs4OOovlgWC

_index:

logstash-

vipeen · December 30, 2021, 5:49pm

Hoping that the above is the format you expected. Apologies, am new to this and trying do better.

vipeen · December 30, 2021, 5:53pm

Hi lker, thank you. I tried using the pattern you shared. Now I can restart logstash (until that logstash status was "logstash not running"). Also, there are a few things I noticed and would like to inform to you.

Currently, the logs in Elasticsearch are only from 10th November till 30th November. There are no live logs updated. So, does using the grok pattern now will change the already existing logs in Elasticsearch or only the logs which are generated from now?
Am not sure why the logs are not getting into Elasticsearch after 30th November.

My current logstash.conf file is,

input
{
    beats
    {
        ssl => false
        host => "0.0.0.0"
        port => 5044
    }
    gelf
    {
        host => "0.0.0.0"
        port => 12201
    }
    http
    {
        ssl => false
        host => "0.0.0.0"
        port => 8080
    }
    tcp
    {
        mode => "server"
        host => "0.0.0.0"
        port => 5010
    }
    udp
    {
        host => "0.0.0.0"
        port => 5000
    }

}

filter {

grok {

match => { "message" => <%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:logging_host}\s+%{PROG:logging_process}(?:[%{INT:logging_pid}])?:\s+\[%{DATA:version}]\s+class="%{DATA:class}"\s+category="%{DATA:category}"\s+ctx="%{DATA:ctx}"\s+filterNumber="%{POSINT:filterNumber}"\s+src="%{IP:src}"\s+srcPort="%{POSINT:srcPort}"\s+dest="%{IP:dest}"\s+destPort="%{POSINT:destPort}"\s+gwAction="%{DATA:gwAction}"\s+gwMode="%{DATA:gwMode}"}

}

}


output
{
    elasticsearch
    {
        hosts => ["127.0.0.1:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
    }
}

Iker · December 30, 2021, 6:32pm

Well, that's a lot, lets started point by point:

It's difficult to diagnose remotely why your logs stop flowing; my recommendation will be, start from scratch, create a new Logstash file, with just one syslog input, no filters and a stdout output, the input should be in a specific port, check the network connectivity and firewall filters; once you are completely sure that the source could reach Logstash and the logs are getting printed in the output move to the next step.
In Logstash, it's recommended to use just one input type per pipeline, in this case the syslog input (Syslog input plugin | Logstash Reference [7.16] | Elastic); this allows you to use the Grok pattern at the input and not in the filter; however there are other ways to parse the information, maybe a kv filter could work as well.
The grok expression need to be enclosed with quotes to work; an example is present in the documentation (Grok filter plugin | Logstash Reference [7.16] | Elastic)

My general advice to you is to read the entire Logstash guide, there are several items that you are going to need in the future, and it's better to get a good foundation about the architecture, configuration, file sections and capabilities of the tool. However, I left you with a couple examples of pipelines that I have deployed in production for parsing syslog messages:

Example 1:

input {
    syslog {
        port => 12345
		grok_pattern => "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"
    }
}

filter {
	kv {
		value_split => "=" 
		source => "message"
	}

	mutate {
		rename => {
			"eventtime" => "[event.created]"
			"srcip" => "[source.ip]"
			"dstip" => "[destination.ip]"
			"action" => "[event.action]"
			"proto" => "[network.transport]"
			"service" => "[network.protocol]"
			"policyid" => "[rule.id]"
			"attack" => "[threat.name]"
			"srcport" => "[source.port]"
			"dstport" => "[destination.port]"
			"direction" => "[network.direction]"
			"attackid" => "[threat.id]"
			"profile" => "[rule.name]"
			"ref" => "[threat.reference]"
			"crlevel" => "[threat.severity]"
		}

		lowercase => [ "network.protocol" ]
		copy => { "[source.ip]" => "[source.address]" }
	}

	date{
		match => ["timestamp", "UNIX"]
		timezone => "tz"
		target => "@timestamp"
	}

	date{
		match => ["event.created", "UNIX"]
		timezone => "tz"
		target => "event.created"
	}
}

output {
	elasticsearch {
		hosts => ["https://elastic01:9200", "https://elastic02:9200"]
		index => "siem_perimetral_ips"
	}
}

Example 2:

input {
    syslog {
        port => 12345
		grok_pattern => "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"
    }
}

filter {	
	grok {
		match => { "message" => "<%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:logging_host}\s+%{PROG:logging_process}(?:[%{INT:logging_pid}])?:\s+\[%{DATA:version}]\s+class=\"%{DATA:class}\"\s+category=\"%{DATA:category}\"\s+ctx=\"%{DATA:ctx}\"\s+filterNumber=\"%{POSINT:filterNumber}\"\s+src=\"%{IP:src}\"\s+srcPort=\"%{POSINT:srcPort}\"\s+dest=\"%{IP:dest}\"\s+destPort=\"%{POSINT:destPort}\"\s+gwAction=\"%{DATA:gwAction}\"\s+gwMode=\"%{DATA:gwMode}\"" }
	}
}

output {
	stdout { codec => rubydebug }
}

vipeen · January 1, 2022, 11:32am

Hi lker, thank you for the suggestions. I am going through them now and will try implementing the same. Just give me a day or two, so that I can come back with my results. Wish you a happy new year! Thank you.

vipeen · January 3, 2022, 2:18pm

Hi lker,

I tried creating a new logstash.conf file and a new ELK VM. I removed the filter stage as you said. But, I am still unable to see the logs. I tried to check if logstash is litening on specific ports, but none of the ports returned a value. I am now attaching the logstash.conf file and the port check commands. Will you be able to assist me further? Sorry, I am new to this and still trying to learn. I was able to learn Regular Expressions and Grok pattern upto a level. Thank you.
default logstash.conf

Iker · January 9, 2022, 12:38am

Sorry I was out for a while, happy new year to you too. Could you please try with this simple Logstash file:

input {
    syslog {
        port => 12345
		codec => cef
    }
}

filter {
}

output {
	stdout { codec => rubydebug }
}

Topic		Replies	Views
Splitting message log Logstash	4	1197	March 25, 2022
How to strip events from message field? Logstash syslog Logstash	8	698	August 18, 2021
Kibana Dashboard is empty with a lot of errors Kibana docker	4	267	April 5, 2023
Filter Logs from Firewall Logstash	4	1847	April 22, 2019
Logstash Grok Pattern Watchguard Firewall Elasticsearch	7	778	October 13, 2023

Kibana dashboard creation issue

Related Topics