# Kibana dashboard creation issue **URL:** \ **Category:** Kibana\ **Created:** [December 29, 2021, 2:43pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132 "2021-12-29T14:43:34Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 29, 2021, 2:43pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/1 "2021-12-29T14:43:34Z")
Hi, I am trying to move logs from a Gateway server to Syslog and then to Elasticsearch and finally build dashboards on Kibana. I am newbie and try to do my best to get this completed. I actually got this task from an ex-colleague who left the company all of a sudden. We have logs on Elasticsearch, but we dont get all the fields in the event while creating dashboard panel. As you can see from the image, the "Message" field container other fields inside it. I need to split them individually to create dashboard panels. Can anyone guide me how to do that? Thank you. ![Event](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9fcf92f193f8f997491162537c369ae3f50a89ea.png) ---
**Author:** ![Iker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iker/32/91708_2.png) [@Iker](https://discuss.elastic.co/u/Iker)\ **Post date:** [December 29, 2021, 4:09pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/2 "2021-12-29T16:09:55Z")
You have to parse the fields, it could be done with a pipeline or if you are using logstash with the filters; this is a good place to start > **[Logstash configuration examples | Logstash Reference \[7.16\] | Elastic](https://www.elastic.co/guide/en/logstash/current/config-examples.html)** ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 29, 2021, 6:30pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/3 "2021-12-29T18:30:06Z")
Hi lker, I tried using Grok pattern inside the logstash.conf file. But, still the same. I think the problem might be the grok pattern, but not sure what exactly is wrong. ---
**Author:** ![Iker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iker/32/91708_2.png) [@Iker](https://discuss.elastic.co/u/Iker)\ **Post date:** [December 29, 2021, 6:51pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/4 "2021-12-29T18:51:19Z")
Please post multiple sample of the messages to determine exactly what pattern could be a good fit for your use case; it seems to be from a Firewall, but I don't know which one specifically. ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 12:17pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/5 "2021-12-30T12:17:23Z")
Sorry, I dint understand. Are you asking the messages from event list? Just like the earlier picture attached? I am still attaching an event from event list and the Grok pattern that is used inside the /opt/bitnami/logstash/pipeline/logstash.conf. ![Event](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9fcf92f193f8f997491162537c369ae3f50a89ea.png) ![Grok](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e061734c06a581df0f4eb45870e53fb1cb91cb5.png) ---
**Author:** ![Iker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iker/32/91708_2.png) [@Iker](https://discuss.elastic.co/u/Iker)\ **Post date:** [December 30, 2021, 2:34pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/6 "2021-12-30T14:34:28Z")
Yes, but... as text, so people could test the grok patterns without having to type by hand the messages. ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 4:30pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/7 "2021-12-30T16:30:12Z")
Understood, sorry my bad. filter { grok { match =\> { "message" =\> \<%{INT:syslog\_pri}\>\s\*%{SYSLOGTIMESTAMP:syslog\_timestamp}\s+%{HOSTNAME:logging\_host}\s+%{PROG:logging\_process}(?:[%{INT:logging\_pid}])?:\s+[%{DATA:version}]\s+class="%{DATA:class}"\s+category="%{DATA:class}"\s+ctx="%{DATA:ctx}"\s+src="%{IP:src}"\s+srcPort="%{POSINT:srcPort}"\s+dest="%{IP:dest}"\s+destPort="%{POSINT:destPort}"\s+identity="%{DATA:identity}"\s+gwAction="%{DATA:gwAction}"\s+gwMode="%{DATA:gwMode}"} } } ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 30, 2021, 4:36pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/8 "2021-12-30T16:36:08Z")
We need txt samples of the log lines as well that is what @Iker is looking for. ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 4:43pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/9 "2021-12-30T16:43:28Z")
@timestamp: Nov 30, 2021 @ 22:33:37.383 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: \<4\>Nov 30 17:03:37 BAGW-R kernel: [BlackRidge|Gateway|5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: wJnNcX0BxLs4OOovRAUN \_index: logstash-2021.11.30 \_score: - ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 4:48pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/10 "2021-12-30T16:48:53Z")
Is this the correct format? I copied them directly to editors and here, but I get this format. ---
**Author:** ![Iker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iker/32/91708_2.png) [@Iker](https://discuss.elastic.co/u/Iker)\ **Post date:** [December 30, 2021, 5:19pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/11 "2021-12-30T17:19:43Z")
> [@vipeen](#): > > src="%{IP:src}"\s+srcPort="%{POSINT:srcPort}"\s+dest="%{IP:dest}"\s+destPort="%{POSINT:destPort}"\s+identity="%{DATA:identity}"\s+gwAction="%{DATA:gwAction}"\s+gwMode="%{DATA:gwMode} Yeah, it works; I requested you several samples, because in my experience there are a multiple logs (auth, vpn connection, filter, allow, deny, and so on; are very different) generated by a Firewall. The following grok expression works for the log sample that you provided, it's based in the one that you had, only required a couple of tweaks here and there; however it's very possible that other messages requires some other expression: ```auto <%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:logging_host}\s+%{PROG:logging_process}(?:[%{INT:logging_pid}])?:\s+\[%{DATA:version}]\s+class="%{DATA:class}"\s+category="%{DATA:category}"\s+ctx="%{DATA:ctx}"\s+filterNumber="%{POSINT:filterNumber}"\s+src="%{IP:src}"\s+srcPort="%{POSINT:srcPort}"\s+dest="%{IP:dest}"\s+destPort="%{POSINT:destPort}"\s+gwAction="%{DATA:gwAction}"\s+gwMode="%{DATA:gwMode}" ``` This Grok debugger could be of help: [https://grokdebug.herokuapp.com](https://grokdebug.herokuapp.com) ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 5:28pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/12 "2021-12-30T17:28:33Z")
Hi, sorry, am pasting 10 events now. Nov 30, 2021 @ 22:33:37.383 @timestamp: Nov 30, 2021 @ 22:33:37.383 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: \<4\>Nov 30 17:03:37 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: wZnNcX0BxLs4OOovRAUN \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:33:37.383 @timestamp: Nov 30, 2021 @ 22:33:37.383 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: \<4\>Nov 30 17:03:37 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: wJnNcX0BxLs4OOovRAUN \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:33:26.271 @timestamp: Nov 30, 2021 @ 22:33:26.271 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: \<4\>Nov 30 17:03:26 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: v5nNcX0BxLs4OOovGAWm \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:33:26.271 @timestamp: Nov 30, 2021 @ 22:33:26.271 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: \<4\>Nov 30 17:03:26 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: vpnNcX0BxLs4OOovGAWm \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:33:15.159 @timestamp: Nov 30, 2021 @ 22:33:15.159 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: \<4\>Nov 30 17:03:15 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: vZnMcX0BxLs4OOov7QU9 \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:33:15.159 @timestamp: Nov 30, 2021 @ 22:33:15.159 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: \<4\>Nov 30 17:03:15 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: vJnMcX0BxLs4OOov7QU9 \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:33:04.051 @timestamp: Nov 30, 2021 @ 22:33:04.051 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: \<4\>Nov 30 17:03:04 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: u5nMcX0BxLs4OOovwQXZ \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:33:04.051 @timestamp: Nov 30, 2021 @ 22:33:04.051 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: \<4\>Nov 30 17:03:04 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: upnMcX0BxLs4OOovwQXZ \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:32:52.955 @timestamp: Nov 30, 2021 @ 22:32:52.955 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: \<4\>Nov 30 17:02:52 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: uZnMcX0BxLs4OOovlgWC \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:32:52.955 @timestamp: Nov 30, 2021 @ 22:32:52.955 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: \<4\>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: uJnMcX0BxLs4OOovlgWC \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:32:52.939 @timestamp: Nov 30, 2021 @ 22:32:52.939 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: \<4\>Nov 30 17:02:52 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="56277" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: t5nMcX0BxLs4OOovlgWC \_index: logstash-2021.11.30 \_score: - \_type: \_doc Nov 30, 2021 @ 22:32:52.939 @timestamp: Nov 30, 2021 @ 22:32:52.939 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: \<4\>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To\_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="56277" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" \_id: tpnMcX0BxLs4OOovlgWC \_index: logstash-2021.11.30 \_score: - \_type: \_doc ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 30, 2021, 5:43pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/13 "2021-12-30T17:43:39Z")
@vipeen We meant what you raw log lines. To format your code / text etc please select it and press the format button. ![Screen Shot 2021-12-30 at 9.42.53 AM](https://us1.discourse-cdn.com/elastic/original/3X/9/2/92029c79e689954bd8f2ddd0dd0e7813bfdf83b1.png) Example `<4>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="56277" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce"` ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 5:47pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/14 "2021-12-30T17:47:39Z")
```auto Nov 30, 2021 @ 22:33:37.383 @timestamp: Nov 30, 2021 @ 22:33:37.383 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: <4>Nov 30 17:03:37 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: wJnNcX0BxLs4OOovRAUN _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:33:37.383 @timestamp: Nov 30, 2021 @ 22:33:37.383 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: <4>Nov 30 17:03:37 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="46165" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: wZnNcX0BxLs4OOovRAUN _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:33:26.271 @timestamp: Nov 30, 2021 @ 22:33:26.271 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: <4>Nov 30 17:03:26 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: vpnNcX0BxLs4OOovGAWm _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:33:26.271 @timestamp: Nov 30, 2021 @ 22:33:26.271 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: <4>Nov 30 17:03:26 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="49483" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: v5nNcX0BxLs4OOovGAWm _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:33:15.159 @timestamp: Nov 30, 2021 @ 22:33:15.159 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: <4>Nov 30 17:03:15 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: vJnMcX0BxLs4OOov7QU9 _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:33:15.159 @timestamp: Nov 30, 2021 @ 22:33:15.159 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: <4>Nov 30 17:03:15 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="38700" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: vZnMcX0BxLs4OOov7QU9 _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:33:04.051 @timestamp: Nov 30, 2021 @ 22:33:04.051 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: <4>Nov 30 17:03:04 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: upnMcX0BxLs4OOovwQXZ _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:33:04.051 @timestamp: Nov 30, 2021 @ 22:33:04.051 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: <4>Nov 30 17:03:04 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="60392" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: u5nMcX0BxLs4OOovwQXZ _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:32:52.955 @timestamp: Nov 30, 2021 @ 22:32:52.955 @version: 1 host: 192.168.120.135 host.keyword: 192.168.120.135 message: <4>Nov 30 17:02:52 BAGW-R kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: uJnMcX0BxLs4OOovlgWC _index: logstash-2021.11.30 _score: - _type: _doc Nov 30, 2021 @ 22:32:52.955 @timestamp: Nov 30, 2021 @ 22:32:52.955 @version: 1 host: 192.168.120.145 host.keyword: 192.168.120.145 message: <4>Nov 30 17:02:52 BAGW-I kernel: [BlackRidge Gateway 5.0.0.8928M] class="Attribution" category="Filter Rule: To_Trusted Drop" ctx="bump0" filterNumber="1022" src="192.168.120.173" srcPort="33705" dest="192.168.120.100" destPort="9080" gwAction="DISCARD" gwMode="Enforce" _id: uZnMcX0BxLs4OOovlgWC _index: logstash- ``` ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 5:49pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/15 "2021-12-30T17:49:51Z")
Hoping that the above is the format you expected. Apologies, am new to this and trying do better. ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [December 30, 2021, 5:53pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/16 "2021-12-30T17:53:57Z")
Hi lker, thank you. I tried using the pattern you shared. Now I can restart logstash (until that logstash status was "logstash not running"). Also, there are a few things I noticed and would like to inform to you. - Currently, the logs in Elasticsearch are only from 10th November till 30th November. There are no live logs updated. So, does using the grok pattern now will change the already existing logs in Elasticsearch or only the logs which are generated from now? - Am not sure why the logs are not getting into Elasticsearch after 30th November. My current logstash.conf file is, ```auto input { beats { ssl => false host => "0.0.0.0" port => 5044 } gelf { host => "0.0.0.0" port => 12201 } http { ssl => false host => "0.0.0.0" port => 8080 } tcp { mode => "server" host => "0.0.0.0" port => 5010 } udp { host => "0.0.0.0" port => 5000 } } filter { grok { match => { "message" => <%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:logging_host}\s+%{PROG:logging_process}(?:[%{INT:logging_pid}])?:\s+\[%{DATA:version}]\s+class="%{DATA:class}"\s+category="%{DATA:category}"\s+ctx="%{DATA:ctx}"\s+filterNumber="%{POSINT:filterNumber}"\s+src="%{IP:src}"\s+srcPort="%{POSINT:srcPort}"\s+dest="%{IP:dest}"\s+destPort="%{POSINT:destPort}"\s+gwAction="%{DATA:gwAction}"\s+gwMode="%{DATA:gwMode}"} } } output { elasticsearch { hosts => ["127.0.0.1:9200"] index => "logstash-%{+YYYY.MM.dd}" } } ``` ---
**Author:** ![Iker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iker/32/91708_2.png) [@Iker](https://discuss.elastic.co/u/Iker)\ **Post date:** [December 30, 2021, 6:32pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/17 "2021-12-30T18:32:20Z")
Well, that's a lot, lets started point by point: - It's difficult to diagnose remotely why your logs stop flowing; my recommendation will be, start from scratch, create a new Logstash file, with just one syslog input, no filters and a stdout output, the input should be in a specific port, check the network connectivity and firewall filters; once you are completely sure that the source could reach Logstash and the logs are getting printed in the output move to the next step. - In Logstash, it's recommended to use just one input type per pipeline, in this case the syslog input ([Syslog input plugin | Logstash Reference [7.16] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html)); this allows you to use the Grok pattern at the input and not in the filter; however there are other ways to parse the information, maybe a kv filter could work as well. - The grok expression need to be enclosed with quotes to work; an example is present in the documentation ([Grok filter plugin | Logstash Reference [7.16] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)) My general advice to you is to read the entire Logstash guide, there are several items that you are going to need in the future, and it's better to get a good foundation about the architecture, configuration, file sections and capabilities of the tool. However, I left you with a couple examples of pipelines that I have deployed in production for parsing syslog messages: **Example 1:** ```auto input { syslog { port => 12345 grok_pattern => "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}" } } filter { kv { value_split => "=" source => "message" } mutate { rename => { "eventtime" => "[event.created]" "srcip" => "[source.ip]" "dstip" => "[destination.ip]" "action" => "[event.action]" "proto" => "[network.transport]" "service" => "[network.protocol]" "policyid" => "[rule.id]" "attack" => "[threat.name]" "srcport" => "[source.port]" "dstport" => "[destination.port]" "direction" => "[network.direction]" "attackid" => "[threat.id]" "profile" => "[rule.name]" "ref" => "[threat.reference]" "crlevel" => "[threat.severity]" } lowercase => ["network.protocol"] copy => { "[source.ip]" => "[source.address]" } } date{ match => ["timestamp", "UNIX"] timezone => "tz" target => "@timestamp" } date{ match => ["event.created", "UNIX"] timezone => "tz" target => "event.created" } } output { elasticsearch { hosts => ["https://elastic01:9200", "https://elastic02:9200"] index => "siem_perimetral_ips" } } ``` **Example 2:** ```auto input { syslog { port => 12345 grok_pattern => "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}" } } filter { grok { match => { "message" => "<%{INT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}\s+%{HOSTNAME:logging_host}\s+%{PROG:logging_process}(?:[%{INT:logging_pid}])?:\s+\[%{DATA:version}]\s+class=\"%{DATA:class}\"\s+category=\"%{DATA:category}\"\s+ctx=\"%{DATA:ctx}\"\s+filterNumber=\"%{POSINT:filterNumber}\"\s+src=\"%{IP:src}\"\s+srcPort=\"%{POSINT:srcPort}\"\s+dest=\"%{IP:dest}\"\s+destPort=\"%{POSINT:destPort}\"\s+gwAction=\"%{DATA:gwAction}\"\s+gwMode=\"%{DATA:gwMode}\"" } } } output { stdout { codec => rubydebug } } ``` ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [January 1, 2022, 11:32am UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/18 "2022-01-01T11:32:20Z")
Hi lker, thank you for the suggestions. I am going through them now and will try implementing the same. Just give me a day or two, so that I can come back with my results. Wish you a happy new year! Thank you. ---
**Author:** ![vipeen](https://avatars.discourse-cdn.com/v4/letter/v/53a042/32.png) [@vipeen](https://discuss.elastic.co/u/vipeen)\ **Post date:** [January 3, 2022, 2:18pm UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/19 "2022-01-03T14:18:10Z")
Hi lker, I tried creating a new logstash.conf file and a new ELK VM. I removed the filter stage as you said. But, I am still unable to see the logs. I tried to check if logstash is litening on specific ports, but none of the ports returned a value. I am now attaching the logstash.conf file and the port check commands. Will you be able to assist me further? Sorry, I am new to this and still trying to learn. I was able to learn Regular Expressions and Grok pattern upto a level. Thank you. ![default logstash.conf](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9e774819ab57277029e5e5cd4dc1170971675dc0.png) ![Port check](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3df6e65cea30d92d0e73693a85a59843ed1bf019.png) ---
**Author:** ![Iker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iker/32/91708_2.png) [@Iker](https://discuss.elastic.co/u/Iker)\ **Post date:** [January 9, 2022, 12:38am UTC](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132/20 "2022-01-09T00:38:50Z")
Sorry I was out for a while, happy new year to you too. Could you please try with this simple Logstash file: ```auto input { syslog { port => 12345 codec => cef } } filter { } output { stdout { codec => rubydebug } } ``` [Next page](https://discuss.elastic.co/t/kibana-dashboard-creation-issue/293132.md?page=2)