How to Parse syslogs before sending to Kibana using logstash

Elastic Stack Logstash
1

Greeting:

I'm trying to send syslogs to my elasticsearch by using logstash, My syslogs containers entries like below

1. Apr 15 22:10:23 myubuntuhost Console[16]: time="2018-04-15T22:10:23.462460809Z" type="scan_summary" log_type="image" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest" vulnerabilties="612" compliance="2"
2. Apr 15 22:10:23 myubuntuhost Console[16]: time="2018-04-15T22:10:23.462460809Z"  type="scan_summary" log_type="image" image_id="mynewregistry.org/image3:latest" image_name="mynewregistry.org/image3:latest" vulnerabilties="138" compliance="2"

I'm using below config file to filter

input {
  file {
    path => ["/var/log/console.log"]
    type => "syslog"
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["https://localhost:443"]
    index => "regscan-%{+YYYY.MM.dd}"
    document_type => "system_logs"
  }
  stdout { codec => rubydebug }
}

while sending logs, i'm seeing results are going in this way

    {
             "syslog_severity" => "notice",
                        "type" => "syslog",
                  "syslog_pid" => "16",
        "syslog_facility_code" => 1,
                     "message" => "Apr 15 22:11:00 myubuntuhost Console[16]: time=\"2018-04-15T22:11:00.521019771Z\" type=\"scan_summary\" log_type=\"image\" image_id=\"mynewregistry.org/image:latest\" image_name=\"mynewregistry.org/image:latest\" vulnerabilties=\"138\" compliance=\"2\"",
             "syslog_hostname" => "myubuntuhost",
        "syslog_severity_code" => 5,
                  "@timestamp" => 2018-04-15T22:11:00.000Z,
              "syslog_program" => "Console",
              "syslog_message" => "time=\"2018-04-15T22:11:00.521019771Z\" type=\"scan_summary\" log_type=\"image\" image_id=\"mynewregistry.org/image:latest\" image_name=\"mynewregistry.org/image:latest\" vulnerabilties=\"138\" compliance=\"2\"",
             "syslog_facility" => "user-level",
                    "@version" => "1",
                        "host" => "myubuntuhost",
                 "received_at" => "2018-04-15T22:11:00.753Z",
               "received_from" => "myubuntuhost",
                        "path" => "/var/log/console.log",
            "syslog_timestamp" => "Apr 15 22:11:00"
    }

I would like to in such a way that i can filter by each column such as image name, vulnerability and compliance

2

Use a kv filter to parse the syslog_message field.

3

Thanks a lot @magnusbaeck

I'm actually new to ELK stack, can you please help to verify if below configuration is correct

input {
  file {
    path => ["/var/log/Console.log"]
    type => "syslog"
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    kv { }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["https://localhost:443"]
    index => "regscan-%{+YYYY.MM.dd}"
    document_type => "system_logs"
  }
  stdout { codec => rubydebug }
}

because I'm seeing each filed parsed but entire message is also getting loaded to Kibana (like below)

|@timestamp|April 18th 2018, 08:30:50.000|
|---|---|
|t  @version|1|
|t  _id|sXfz2GIBIo-yhkFYDmD|
|t  _index|regscan-2018.04.18|
|#  _score|1|
|t  _type|system_logs|
|t  cve|TEMP-0000000|
|t  description|Image contains vulnerablity|
|t  host|myubuntuhost.org.com|
|t  image_id|mynewregistry.org/image:latest|
|t  image_name|mynewregistry.org/image:latest|
|t  log_type|vulnerability|
|t  message|Apr 18 13:30:50 myubuntuhost Console[16]: time="2018-04-18T13:30:50.678061141Z" type="scan" log_type="vulnerability" vulnerability_id="0" description="Image contains vulnerablity" cve="TEMP-0000000" severity="unimportant" package="gnutls28" rule="Default - alert all components" host="myubuntuhost.org.com" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest"|
|t  package|gnutls28|
|t  path|/var/log/Console.log|
| received_at|April 18th 2018, 08:30:50.876|
|t  received_from|myubuntuhost|
|t  rule|Default - alert all components|
|t  severity|unimportant|
|t  syslog_facility|user-level|
|#  syslog_facility_code|1|
|t  syslog_hostname|myubuntuhost|
|t  syslog_message|time="2018-04-18T13:30:50.678061141Z" type="scan" log_type="vulnerability" vulnerability_id="0" description="Image contains vulnerablity" cve="TEMP-0000" severity="unimportant" package="gnutls28" rule="Default - alert all components" host="myubuntuhost.org.com" image_id="mynewregistry.org/image:latest" image_name="mynewregistry.org/image:latest"|
|t  syslog_pid|16|
|t  syslog_program|Console|
|t  syslog_severity|notice|
|#  syslog_severity_code|5|
|t  syslog_timestamp|Apr 18 13:30:50|
| time|April 18th 2018, 08:30:50.678|
|t  type|registry_scan|
|t  vulnerability_id|2|
4

because I'm seeing each filed parsed but entire message is also getting loaded to Kibana (like below)

You mean the message field? If you don't want that field then remove it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.