# Duplicated lines when parsing rabbitmq logs§

**URL:** <https://discuss.elastic.co/t/duplicated-lines-when-parsing-rabbitmq-logs/251613>\
**Category:** Logstash\
**Created:** [October 9, 2020, 8:47pm UTC](https://discuss.elastic.co/t/duplicated-lines-when-parsing-rabbitmq-logs/251613 "2020-10-09T20:47:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hamish1](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@Hamish1](https://discuss.elastic.co/u/Hamish1)\
**Post date:** [October 9, 2020, 8:47pm UTC](https://discuss.elastic.co/t/duplicated-lines-when-parsing-rabbitmq-logs/251613/1 "2020-10-09T20:47:47Z")

</div>

Good day everyone!

We're using RabbitMQ version 3.6.10 and it has a weird log format:

```auto
closing AMQP connection <0.28817.3524> (HIDEN:50790 -> HIDEN:5672, vhost: '/', user: 'HIDEN')

=INFO REPORT==== 9-Oct-2020::22:41:24 ===
accepting AMQP connection <0.26955.3524> (HIDEN:43198 -> HIDEN:5672)

```

My configuration is:

filebeat:

```auto
- type: log
  enabled: true
  paths:
    - /var/log/rabbitmq/*prod.log
  exclude_files: ['\.gz$']
  multiline.pattern: '^='
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000
  multiline.timeout: 3s
  fields_under_root: true
  encoding: utf-8
  tags: ['rabbitmq']
  fields:
    rabbitmq: true
    document_type: rabbitmq

```

logstash:

```auto
if [document_type] == "rabbitmq" {

  mutate {
    gsub => ["message", "\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]", ""]
  }

  grok {
    pattern_definitions => { "RABBITMQDATE" => "%{MONTHDAY}-%{MONTH}-%{YEAR}::%{HOUR}:%{MINUTE}:%{SECOND}" }
    match => { "message" => "=%{DATA:report_type}==== %{RABBITMQDATE:timestamp} ===\n%{GREEDYDATA:message}" }
  }

  date {
    match => ["timestamp", "dd-MMM-yyyy::HH:mm:ss"]
    target => "@timestamp"
    timezone => "Europe/Berlin"
# remove_field => ["timestamp"]
  }
}

```

In Kibana I see this:

 ![1](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5fc69d11974476833248132463fc9f2ae1ee1e00.jpeg)

So it's just duplicating a message string one more time after a comma (but neither in my pattern nor in log no comma at all)... Could you please advise where is the issue...?

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 9, 2020, 9:32pm UTC](https://discuss.elastic.co/t/duplicated-lines-when-parsing-rabbitmq-logs/251613/2 "2020-10-09T21:32:26Z")

</div>

That is showing that the [message] field is an array. You parse [message] using grok and call one of the extracted fields [message]. If you do not use the overwrite option for grok then it will change the field type to an array of strings.

---

<div class="post-metadata">

**Author:** ![Hamish1](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@Hamish1](https://discuss.elastic.co/u/Hamish1)\
**Post date:** [October 9, 2020, 9:37pm UTC](https://discuss.elastic.co/t/duplicated-lines-when-parsing-rabbitmq-logs/251613/3 "2020-10-09T21:37:52Z")

</div>

Thank you Badger!

I've changed it to this  
` match => { "message" => "=%{GREEDYDATA:report_type}==== %{RABBITMQDATE:timestamp} ===\n%{GREEDYDATA:r_message}" }`

and it got fixed!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 6, 2020, 9:38pm UTC](https://discuss.elastic.co/t/duplicated-lines-when-parsing-rabbitmq-logs/251613/4 "2020-11-06T21:38:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.