Error with dissect filter [SOLVED]

Elastic Stack Logstash
1

I am trying to parse this message:

Summary for local -------------- Succeeded: 478 (changed=180) Failed: 0 -------------- Total states run: 478 Total run time: 68.410 s [Tue, 05 Jun 2018 14:33:17 +0200] codedeploy-agent started Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service. The system is finally up, after 144.81 seconds

As the grok filter for some reason is unable to parse the message, I'm trying the dissect filter:

filter {
        dissect {
                mapping => { "message" => "Summary for local -------------- Succeeded: %{states_succeeded} (changed=%{states_changed}) Failed: %{states_failed} -------------- \
Total states run: %{states_run} Total run time: %{run_time} s [%{agent_timestamp}] codedeploy-agent started Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service \
to /usr/lib/systemd/system/codedeploy-agent.service. The system is finally up, after %{cloudinit_time} seconds" }
        }
}

This filter results in the assignment of the following indexes:

  • states_succeeded: 478
  • states_failed: (empty string)
  • states_changed: (empty string)
  • states_run: (empty string)
  • run_time: (empty string)
  • cloudinit_time: 180) Failed: 0 -------------- Total states run: 478 Total run time: 68.410 s [Tue, 05 Jun 2018 14:33:17 +0200] codedeploy-agent started Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service.
    The system is finally up, after 144.81 seconds

The only correctly parsed index is "states_succeeded". Why is it so? It looks like the parentheses are confusing the dissect parser, but escaping them produces other errors.

2

Do not use backslash-newline inside the mapping. Just use a single very long line. Once you do that everything parses OK except

"cloudinit_time" => "144.81 seconds"

and you can fix that with a trailing %{}.

1 Like
3

I have removed all backslash-newlines and I get exactly the same result. Does it work on your machine?

4

Yes, with 6.2.4 on Linux it works.

1 Like
5

I have exactly the same version and OS (v6.2.4 on Linux CentOS 7.5), and it is causing me endless headaches. Is there something else I should check? Here's my multiline config for Filebeat /etc/filebeat/filebeat.yml:

filebeat.prospectors:
- type: log
  paths:
    - /opt/elktest/cloud-init-output.log
  multiline:
    pattern: '^Summary for local$'
    negate: 'true'
    match: 'after'

output.logstash:
  hosts: ["localhost:5044"]

Here's my Logstash config /etc/logstash/logstash.yml:

path.data: /var/lib/logstash
path.logs: /var/log/logstash

And here's my Elasticsearch config /etc/elasticsearch/elasticsearch.yml:

cluster.name: mycluster
node.name: es01
node.data: true
node.master: true
http.cors.enabled: true
http.cors.allow-origin: "*"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

There's apparently something very wrong with my configuration, since all filter plugins (grok, dissect, ...) are showing unexpected behavior.

6

You have a multiline in filebeat? So your input will have embedded newlines. Your filter patterns need to match that. With no filters, what do you get from output { stdout { codec => rubydebug } } ?

1 Like
7

Here's the relevant block. How do I match the newlines in the filter?

2018-06-11T16:18:10.733+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-11T14:18:05.732Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/opt/elktest/cloud-init-output.log",
  "offset": 396,
  "message": "Summary for local\n--------------\nSucceeded: 478 (changed=180)\nFailed:      0\n--------------\nTotal states run:     478\nTotal run time:    68.410 s\n[Tue, 05 Jun 2018 14:33:17 +0200] codedeploy-agent started\nCreated symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service.\nThe system is finally up, after 144.81 seconds",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "version": "6.2.4",
    "name": "centos7",
    "hostname": "centos7"
  }
}
8

Use a multiline mapping

    dissect {
        mapping => { 
            "message" => "Summary for local
--------------
Succeeded: %{states_succeeded} (changed=%{states_changed})
Failed:      %{states_failed}
--------------
Total states run:     %{states_run}
Total run time:    %{run_time} s
[%{agent_timestamp}] codedeploy-agent started
Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service.
The system is finally up, after %{cloudinit_time} seconds%{}"
        }
    }
1 Like
9

Thanks for your answer. I solved it differently -- I've added a mutate filter to replace newlines with spaces, and it works at last. Thank you very much.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.