Error with dissect filter [SOLVED]


#1

I am trying to parse this message:

Summary for local -------------- Succeeded: 478 (changed=180) Failed: 0 -------------- Total states run: 478 Total run time: 68.410 s [Tue, 05 Jun 2018 14:33:17 +0200] codedeploy-agent started Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service. The system is finally up, after 144.81 seconds 

As the grok filter for some reason is unable to parse the message, I'm trying the dissect filter:

filter {
        dissect {
                mapping => { "message" => "Summary for local -------------- Succeeded: %{states_succeeded} (changed=%{states_changed}) Failed: %{states_failed} -------------- \
Total states run: %{states_run} Total run time: %{run_time} s [%{agent_timestamp}] codedeploy-agent started Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service \
to /usr/lib/systemd/system/codedeploy-agent.service. The system is finally up, after %{cloudinit_time} seconds" }
        }
}

This filter results in the assignment of the following indexes:

  • states_succeeded: 478
  • states_failed: (empty string)
  • states_changed: (empty string)
  • states_run: (empty string)
  • run_time: (empty string)
  • cloudinit_time: 180) Failed: 0 -------------- Total states run: 478 Total run time: 68.410 s [Tue, 05 Jun 2018 14:33:17 +0200] codedeploy-agent started Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service.
    The system is finally up, after 144.81 seconds

The only correctly parsed index is "states_succeeded". Why is it so? It looks like the parentheses are confusing the dissect parser, but escaping them produces other errors.


#2

Do not use backslash-newline inside the mapping. Just use a single very long line. Once you do that everything parses OK except

"cloudinit_time" => "144.81 seconds"

and you can fix that with a trailing %{}.


#3

I have removed all backslash-newlines and I get exactly the same result. Does it work on your machine?


#4

Yes, with 6.2.4 on Linux it works.


#5

I have exactly the same version and OS (v6.2.4 on Linux CentOS 7.5), and it is causing me endless headaches. Is there something else I should check? Here's my multiline config for Filebeat /etc/filebeat/filebeat.yml:

filebeat.prospectors:
- type: log
  paths:
    - /opt/elktest/cloud-init-output.log
  multiline:
    pattern: '^Summary for local$'
    negate: 'true'
    match: 'after'

output.logstash:
  hosts: ["localhost:5044"]

Here's my Logstash config /etc/logstash/logstash.yml:

path.data: /var/lib/logstash
path.logs: /var/log/logstash

And here's my Elasticsearch config /etc/elasticsearch/elasticsearch.yml:

cluster.name: mycluster
node.name: es01
node.data: true
node.master: true
http.cors.enabled: true
http.cors.allow-origin: "*"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

There's apparently something very wrong with my configuration, since all filter plugins (grok, dissect, ...) are showing unexpected behavior.


#6

You have a multiline in filebeat? So your input will have embedded newlines. Your filter patterns need to match that. With no filters, what do you get from output { stdout { codec => rubydebug } } ?


Grok pattern fails although it is valid
#7

Here's the relevant block. How do I match the newlines in the filter?

2018-06-11T16:18:10.733+0200	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-11T14:18:05.732Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "source": "/opt/elktest/cloud-init-output.log",
  "offset": 396,
  "message": "Summary for local\n--------------\nSucceeded: 478 (changed=180)\nFailed:      0\n--------------\nTotal states run:     478\nTotal run time:    68.410 s\n[Tue, 05 Jun 2018 14:33:17 +0200] codedeploy-agent started\nCreated symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service.\nThe system is finally up, after 144.81 seconds",
  "prospector": {
    "type": "log"
  },
  "beat": {
    "version": "6.2.4",
    "name": "centos7",
    "hostname": "centos7"
  }
}

#8

Use a multiline mapping

    dissect {
        mapping => { 
            "message" => "Summary for local
--------------
Succeeded: %{states_succeeded} (changed=%{states_changed})
Failed:      %{states_failed}
--------------
Total states run:     %{states_run}
Total run time:    %{run_time} s
[%{agent_timestamp}] codedeploy-agent started
Created symlink from /etc/systemd/system/multi-user.target.wants/codedeploy-agent.service to /usr/lib/systemd/system/codedeploy-agent.service.
The system is finally up, after %{cloudinit_time} seconds%{}"
        }
    }

#9

Thanks for your answer. I solved it differently -- I've added a mutate filter to replace newlines with spaces, and it works at last. Thank you very much.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.