How to debug Logstash filter with dissect?

Elastic Stack Logstash
1

There's a grok debugger, but how best to debug my dissect filter when none of my log data appear to "get caught" in it?

My filter:

filter
{
  if( [ type ] == "doc" )
  {
    dissect
    {
      mapping =>
      {
        "message" => "%{month} %{+day} %{+timestamp} %{hostname} CEF:%{v}|%{vendor}|%{product}|%{version}|%{id}|%{name}|%{severity}|%{extensions}"
      }
    }
  }
}

Log data sample:

Sep 19 08:26:10 nargothrond CEF:0|security|threatmanager|1.0|100|Detected a threat. No action needed.|10|src=10.0.0.1 msg=Detected a threat.

2

Do your documents have a field named type with the value doc so you even reach the dissect filter? Do you get any error tag in the output?

3

Yes, the type is doc. Here's one I've scraped from Kibana (using index pattern, filebeat-2018.11.20--indentation of values has disappeared as I paste it in here). Oops, it appears that it's _type: doc.

input.type:
log
tags:
beats_input_codec_plain_applied
host.name:
85cace60e9e8
source:
/opt/acme/logs/audit.log
offset:
5,504
message:
2018-11-04 15:11:09,639 CEF:0|AcmeExplosives|SOA|6.6.1|16|UpdateSettings|0|src=127.0.0.1 tenant=Default Current_#AuthorizationAdapters=1 New_#AuthorizationAdapters=0 shost=quicktest.int
prospector.type:
log
beat.name:
85cace60e9e8
beat.hostname:
85cace60e9e8
beat.version:
6.4.2
@timestamp:
November 20th 2018, 16:02:20.550
@version:
1
_id:
_OddM2cBOXqG0esZMziJ
_type:
doc
_index:
filebeat-2018.11.20
_score:
-

4

_type is a special field and not part of the document.

5

Very well, I've taken out that condition ( if[ type/_type == "doc" ). I still get no refinement of message into the mappings I was hoping to dissect. The output, as viewed in Kibana, is still as I just gave the earlier example.

Just to make sure, note that I have this dissect code (above) in /etc/logstash/conf.d/08-cef.conf, along with 02-beats-input.conf, 10-syslog.conf, 11-nginx.conf and 30-output.conf.

10-syslog.conf contains:

filter
{
  if[ type ] == "syslog"
  {
    grok
    {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date
    {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

I can't tell if 08-cef.conf is being used, or if 10-syslog.conf is interfering somehow, part of this question (how to debug). The other part, of course, will be why doesn't dissect work.

(I greatly appreciate your responding to this.)

6

All the files in the directory will be concatenated by Logstash, so could very well affect each other. The reason the sample event you showed does not work is probably that it has a different timestamp format compared to what you showed initially, 2018-11-04 15:11:09,639 vs Sep 19 08:26:10.

7

Thanks! (Alas, too overwhelmed by everything to see the obvious--you are very helpful.) It now works to a much greater degree:

New code:

filter
{
  dissect
  {
    mapping =>
    {
      "message" => "%{date_stamp} %{+time_stamp} CEF:%{v}|%{vendor}|%{product}|%{version}|%{id}|%{name}|%{severity}|%{extensions}"
    }
  }
}

and the output contains very much more what I'm looking for:

id:
    10
product:
    UpdateSettings
input.type:
    log
name:
    received update settings request
extensions:
    src=127.0.0.1 request=https://127.0.0.1/params/management/updateSettings service=param server tenant=Default shost=quicktest.int
offset:
    6,360
@timestamp:
    November 21st 2018, 08:24:39.923
host.name:
    7fcacd4855dc
tags:
    beats_input_codec_plain_applied
vendor:
    AcmeExplosives
date_stamp:
    November 7th 2018, 17:00:00.000
@version:
    1
v:
    0
version:
    6.6.1
severity:
    5
beat.version:
    6.4.2
beat.name:
    7fcacd4855dc
beat.hostname:
    7fcacd4855dc
time_stamp:
    22:21:17,801
prospector.type:
    log
message:
    2018-11-08 22:21:17
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.