# How to Parse syslogs before sending to Kibana using logstash **URL:** \ **Category:** Logstash\ **Created:** [April 15, 2018, 10:54pm UTC](https://discuss.elastic.co/t/how-to-parse-syslogs-before-sending-to-kibana-using-logstash/128088 "2018-04-15T22:54:21Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sahotay](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sahotay](https://discuss.elastic.co/u/sahotay)\ **Post date:** [April 15, 2018, 10:54pm UTC](https://discuss.elastic.co/t/how-to-parse-syslogs-before-sending-to-kibana-using-logstash/128088/1 "2018-04-15T22:54:21Z")
Greeting: I'm trying to send syslogs to my elasticsearch by using logstash, My syslogs containers entries like below ```auto 1. Apr 15 22:10:23 myubuntuhost Console[16]: time="2018-04-15T22:10:23.462460809Z" type="scan_summary" log_type="image" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest" vulnerabilties="612" compliance="2" 2. Apr 15 22:10:23 myubuntuhost Console[16]: time="2018-04-15T22:10:23.462460809Z" type="scan_summary" log_type="image" image_id="mynewregistry.org/image3:latest" image_name="mynewregistry.org/image3:latest" vulnerabilties="138" compliance="2" ``` I'm using below config file to filter ``` input { file { path => ["/var/log/console.log"] type => "syslog" } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"] } syslog_pri { } date { match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } } } output { elasticsearch { hosts => ["https://localhost:443"] index => "regscan-%{+YYYY.MM.dd}" document_type => "system_logs" } stdout { codec => rubydebug } } ``` while sending logs, i'm seeing results are going in this way ```auto { "syslog_severity" => "notice", "type" => "syslog", "syslog_pid" => "16", "syslog_facility_code" => 1, "message" => "Apr 15 22:11:00 myubuntuhost Console[16]: time=\"2018-04-15T22:11:00.521019771Z\" type=\"scan_summary\" log_type=\"image\" image_id=\"mynewregistry.org/image:latest\" image_name=\"mynewregistry.org/image:latest\" vulnerabilties=\"138\" compliance=\"2\"", "syslog_hostname" => "myubuntuhost", "syslog_severity_code" => 5, "@timestamp" => 2018-04-15T22:11:00.000Z, "syslog_program" => "Console", "syslog_message" => "time=\"2018-04-15T22:11:00.521019771Z\" type=\"scan_summary\" log_type=\"image\" image_id=\"mynewregistry.org/image:latest\" image_name=\"mynewregistry.org/image:latest\" vulnerabilties=\"138\" compliance=\"2\"", "syslog_facility" => "user-level", "@version" => "1", "host" => "myubuntuhost", "received_at" => "2018-04-15T22:11:00.753Z", "received_from" => "myubuntuhost", "path" => "/var/log/console.log", "syslog_timestamp" => "Apr 15 22:11:00" } ``` I would like to in such a way that i can filter by each column such as image name, vulnerability and compliance ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 16, 2018, 4:20am UTC](https://discuss.elastic.co/t/how-to-parse-syslogs-before-sending-to-kibana-using-logstash/128088/2 "2018-04-16T04:20:59Z")
Use a kv filter to parse the `syslog_message` field. ---
**Author:** ![sahotay](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sahotay](https://discuss.elastic.co/u/sahotay)\ **Post date:** [April 18, 2018, 1:45pm UTC](https://discuss.elastic.co/t/how-to-parse-syslogs-before-sending-to-kibana-using-logstash/128088/3 "2018-04-18T13:45:47Z")
Thanks a lot @magnusbaeck I'm actually new to ELK stack, can you please help to verify if below configuration is correct ```auto input { file { path => ["/var/log/Console.log"] type => "syslog" } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"] } kv { } syslog_pri { } date { match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } } } output { elasticsearch { hosts => ["https://localhost:443"] index => "regscan-%{+YYYY.MM.dd}" document_type => "system_logs" } stdout { codec => rubydebug } } ``` because I'm seeing each filed parsed but entire message is also getting loaded to Kibana (like below) ```auto |@timestamp|April 18th 2018, 08:30:50.000| |---|---| |t @version|1| |t _id|sXfz2GIBIo-yhkFYDmD| |t _index|regscan-2018.04.18| |# _score|1| |t _type|system_logs| |t cve|TEMP-0000000| |t description|Image contains vulnerablity| |t host|myubuntuhost.org.com| |t image_id|mynewregistry.org/image:latest| |t image_name|mynewregistry.org/image:latest| |t log_type|vulnerability| |t message|Apr 18 13:30:50 myubuntuhost Console[16]: time="2018-04-18T13:30:50.678061141Z" type="scan" log_type="vulnerability" vulnerability_id="0" description="Image contains vulnerablity" cve="TEMP-0000000" severity="unimportant" package="gnutls28" rule="Default - alert all components" host="myubuntuhost.org.com" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest"| |t package|gnutls28| |t path|/var/log/Console.log| | received_at|April 18th 2018, 08:30:50.876| |t received_from|myubuntuhost| |t rule|Default - alert all components| |t severity|unimportant| |t syslog_facility|user-level| |# syslog_facility_code|1| |t syslog_hostname|myubuntuhost| |t syslog_message|time="2018-04-18T13:30:50.678061141Z" type="scan" log_type="vulnerability" vulnerability_id="0" description="Image contains vulnerablity" cve="TEMP-0000" severity="unimportant" package="gnutls28" rule="Default - alert all components" host="myubuntuhost.org.com" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest"| |t syslog_pid|16| |t syslog_program|Console| |t syslog_severity|notice| |# syslog_severity_code|5| |t syslog_timestamp|Apr 18 13:30:50| | time|April 18th 2018, 08:30:50.678| |t type|registry_scan| |t vulnerability_id|2| ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 19, 2018, 6:08am UTC](https://discuss.elastic.co/t/how-to-parse-syslogs-before-sending-to-kibana-using-logstash/128088/4 "2018-04-19T06:08:10Z")
> because I'm seeing each filed parsed but entire message is also getting loaded to Kibana (like below) You mean the `message` field? If you don't want that field then remove it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 17, 2018, 6:20am UTC](https://discuss.elastic.co/t/how-to-parse-syslogs-before-sending-to-kibana-using-logstash/128088/5 "2018-05-17T06:20:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.