How to Parse syslogs before sending to Kibana using logstash

Elastic Stack Logstash
3

Thanks a lot @magnusbaeck

I'm actually new to ELK stack, can you please help to verify if below configuration is correct

input {
  file {
    path => ["/var/log/Console.log"]
    type => "syslog"
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    kv { }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["https://localhost:443"]
    index => "regscan-%{+YYYY.MM.dd}"
    document_type => "system_logs"
  }
  stdout { codec => rubydebug }
}

because I'm seeing each filed parsed but entire message is also getting loaded to Kibana (like below)

|@timestamp|April 18th 2018, 08:30:50.000|
|---|---|
|t  @version|1|
|t  _id|sXfz2GIBIo-yhkFYDmD|
|t  _index|regscan-2018.04.18|
|#  _score|1|
|t  _type|system_logs|
|t  cve|TEMP-0000000|
|t  description|Image contains vulnerablity|
|t  host|myubuntuhost.org.com|
|t  image_id|mynewregistry.org/image:latest|
|t  image_name|mynewregistry.org/image:latest|
|t  log_type|vulnerability|
|t  message|Apr 18 13:30:50 myubuntuhost Console[16]: time="2018-04-18T13:30:50.678061141Z" type="scan" log_type="vulnerability" vulnerability_id="0" description="Image contains vulnerablity" cve="TEMP-0000000" severity="unimportant" package="gnutls28" rule="Default - alert all components" host="myubuntuhost.org.com" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest"|
|t  package|gnutls28|
|t  path|/var/log/Console.log|
| received_at|April 18th 2018, 08:30:50.876|
|t  received_from|myubuntuhost|
|t  rule|Default - alert all components|
|t  severity|unimportant|
|t  syslog_facility|user-level|
|#  syslog_facility_code|1|
|t  syslog_hostname|myubuntuhost|
|t  syslog_message|time="2018-04-18T13:30:50.678061141Z" type="scan" log_type="vulnerability" vulnerability_id="0" description="Image contains vulnerablity" cve="TEMP-0000" severity="unimportant" package="gnutls28" rule="Default - alert all components" host="myubuntuhost.org.com" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest"|
|t  syslog_pid|16|
|t  syslog_program|Console|
|t  syslog_severity|notice|
|#  syslog_severity_code|5|
|t  syslog_timestamp|Apr 18 13:30:50|
| time|April 18th 2018, 08:30:50.678|
|t  type|registry_scan|
|t  vulnerability_id|2|