Duplicated lines when parsing rabbitmq logs§

[Hamish1 (Hamish)]

Good day everyone!

We're using RabbitMQ version 3.6.10 and it has a weird log format:

closing AMQP connection <0.28817.3524> (HIDEN:50790 -> HIDEN:5672, vhost: '/', user: 'HIDEN')

=INFO REPORT==== 9-Oct-2020::22:41:24 ===
accepting AMQP connection <0.26955.3524> (HIDEN:43198 -> HIDEN:5672)

My configuration is:

filebeat:

- type: log
  enabled: true
  paths:
    - /var/log/rabbitmq/*prod.log
  exclude_files: ['\.gz$']
  multiline.pattern: '^='
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000
  multiline.timeout: 3s
  fields_under_root: true
  encoding: utf-8
  tags: ['rabbitmq']
  fields:
    rabbitmq: true
    document_type: rabbitmq

logstash:

if [document_type] == "rabbitmq" {

  mutate {
    gsub => ["message", "\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]", ""]
  }

  grok {
    pattern_definitions => { "RABBITMQDATE" => "%{MONTHDAY}-%{MONTH}-%{YEAR}::%{HOUR}:%{MINUTE}:%{SECOND}" }
    match => { "message" => "=%{DATA:report_type}==== %{RABBITMQDATE:timestamp} ===\n%{GREEDYDATA:message}" }
  }

  date {
    match => [ "timestamp", "dd-MMM-yyyy::HH:mm:ss" ]
    target => "@timestamp"
    timezone => "Europe/Berlin"
#    remove_field => ["timestamp"]
  }
}

In Kibana I see this:

So it's just duplicating a message string one more time after a comma (but neither in my pattern nor in log no comma at all)... Could you please advise where is the issue...?

Thank you in advance!

[Badger]

That is showing that the [message] field is an array. You parse [message] using grok and call one of the extracted fields [message]. If you do not use the overwrite option for grok then it will change the field type to an array of strings.

[Hamish1 (Hamish)]

Thank you Badger!

I've changed it to this
match => { "message" => "=%{GREEDYDATA:report_type}==== %{RABBITMQDATE:timestamp} ===\n%{GREEDYDATA:r_message}" }

and it got fixed!