# How to debug Logstash filter with dissect?

**URL:** <https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616>\
**Category:** Logstash\
**Created:** [November 21, 2018, 12:07am UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616 "2018-11-21T00:07:29Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 21, 2018, 12:07am UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/1 "2018-11-21T00:07:29Z")

</div>

There's a grok debugger, but how best to debug my dissect filter when none of my log data appear to "get caught" in it?

My filter:

> ```
> filter
> {
> if( [type] == "doc" )
> {
> dissect
> {
> mapping =>
> {
> "message" => "%{month} %{+day} %{+timestamp} %{hostname} CEF:%{v}|%{vendor}|%{product}|%{version}|%{id}|%{name}|%{severity}|%{extensions}"
> }
> }
> }
> }
> 
> ```

Log data sample:

`Sep 19 08:26:10 nargothrond CEF:0|security|threatmanager|1.0|100|Detected a threat. No action needed.|10|src=10.0.0.1 msg=Detected a threat.`

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 21, 2018, 6:47am UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/2 "2018-11-21T06:47:19Z")

</div>

Do your documents have a field named `type` with the value `doc` so you even reach the dissect filter? Do you get any error tag in the output?

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 21, 2018, 2:23pm UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/3 "2018-11-21T14:23:51Z")

</div>

Yes, the `type` is `doc`. Here's one I've scraped from Kibana (using index pattern, `filebeat-2018.11.20`--indentation of values has disappeared as I paste it in here). Oops, it appears that it's `_type: doc`.

> input.type:  
> log  
> tags:  
> beats\_input\_codec\_plain\_applied  
> host.name:  
> 85cace60e9e8  
> source:  
> /opt/acme/logs/audit.log  
> offset:  
> 5,504  
> message:  
> 2018-11-04 15:11:09,639 CEF:0|AcmeExplosives|SOA|6.6.1|16|UpdateSettings|0|src=127.0.0.1 tenant=Default Current\_#AuthorizationAdapters=1 New\_#AuthorizationAdapters=0 shost=quicktest.int  
> prospector.type:  
> log  
> beat.name:  
> 85cace60e9e8  
> beat.hostname:  
> 85cace60e9e8  
> beat.version:  
> 6.4.2  
> @timestamp:  
> November 20th 2018, 16:02:20.550  
> @version:  
> 1  
> \_id:  
> \_OddM2cBOXqG0esZMziJ  
> \_type:  
> doc  
> \_index:  
> filebeat-2018.11.20  
> \_score:  
> -

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 21, 2018, 2:27pm UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/4 "2018-11-21T14:27:44Z")

</div>

`_type` is a special field and not part of the document.

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 21, 2018, 3:15pm UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/5 "2018-11-21T15:15:26Z")

</div>

Very well, I've taken out that condition ( `if[ type/_type == "doc"` ). I still get no refinement of `message` into the mappings I was hoping to `dissect`. The output, as viewed in Kibana, is still as I just gave the earlier example.

Just to make sure, note that I have this `dissect` code (above) in _/etc/logstash/conf.d/08-cef.conf_, along with _02-beats-input.conf_, _10-syslog.conf_, _11-nginx.conf_ and _30-output.conf_.

_10-syslog.conf_ contains:

```
filter
{
  if[type] == "syslog"
  {
    grok
    {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    syslog_pri { }
    date
    {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
}

```

I can't tell if _08-cef.conf_ is being used, or if _10-syslog.conf_ is interfering somehow, part of this question (how to debug). The other part, of course, will be why doesn't `dissect` work.

(I greatly appreciate your responding to this.)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 21, 2018, 3:20pm UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/6 "2018-11-21T15:20:01Z")

</div>

All the files in the directory will be concatenated by Logstash, so could very well affect each other. The reason the sample event you showed does not work is probably that it has a different timestamp format compared to what you showed initially, `2018-11-04 15:11:09,639` vs `Sep 19 08:26:10`.

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 21, 2018, 3:43pm UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/7 "2018-11-21T15:43:20Z")

</div>

Thanks! (Alas, too overwhelmed by everything to see the obvious--you are very helpful.) It now works to a much greater degree:

New code:

```
filter
{
  dissect
  {
    mapping =>
    {
      "message" => "%{date_stamp} %{+time_stamp} CEF:%{v}|%{vendor}|%{product}|%{version}|%{id}|%{name}|%{severity}|%{extensions}"
    }
  }
}

```

and the output contains very much more what I'm looking for:

```
id:
    10
product:
    UpdateSettings
input.type:
    log
name:
    received update settings request
extensions:
    src=127.0.0.1 request=https://127.0.0.1/params/management/updateSettings service=param server tenant=Default shost=quicktest.int
offset:
    6,360
@timestamp:
    November 21st 2018, 08:24:39.923
host.name:
    7fcacd4855dc
tags:
    beats_input_codec_plain_applied
vendor:
    AcmeExplosives
date_stamp:
    November 7th 2018, 17:00:00.000
@version:
    1
v:
    0
version:
    6.6.1
severity:
    5
beat.version:
    6.4.2
beat.name:
    7fcacd4855dc
beat.hostname:
    7fcacd4855dc
time_stamp:
    22:21:17,801
prospector.type:
    log
message:
    2018-11-08 22:21:17
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2018, 3:43pm UTC](https://discuss.elastic.co/t/how-to-debug-logstash-filter-with-dissect/157616/8 "2018-12-19T15:43:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.