# AMQP Metadata from RabbitMQ input plugin

**URL:** <https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711>\
**Category:** Logstash\
**Created:** [February 7, 2024, 8:25am UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711 "2024-02-07T08:25:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Big-Edd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/big-edd/32/131503_2.png) [@Big-Edd](https://discuss.elastic.co/u/Big-Edd)\
**Post date:** [February 7, 2024, 8:25am UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711/1 "2024-02-07T08:25:17Z")

</div>

Hello Everyone,

We are inputting AMQP messages from RabbitMQ. These come to RabbitMQ via SMTP utilizing the rabbitmq-email plugin, so the AMQP headers contain some SMTP information we need.

By default it seems that only the body of the email is inputted from RabbitMQ.

We require the email subject and destination address from the AMQP metadata. We know they are there, as we are able to see these when consuming messages with a python script.

Attempting to get this working has lead me to using the following logstash.conf file.  
Note: routing-key from AMPQ metadata is obtained from the SMTP email destination ("To") address.

```auto
input {
  rabbitmq {
    codec => "plain"
    tags => ["test","RabbitMQ","smtp"]
    host => "rabbitmq"
    port => 5672
    queue => "queue_t"
    durable => true
    passive => true
    user => "consumer_t"
    password => "BOGUS_PASSWORD"
    metadata_enabled => "extended"
    add_field => { "topic" => "%{[@metadata][rabbitmq_properties][routing-key]}"}
    add_field => { "subject" => "%{[@metadata][rabbitmq_headers][Subject]}"}
  }
}

filter {
  json {
      add_field => { "to_address" => "%{[@metadata][rabbitmq_properties][routing-key]}"}
      add_field => { "subject" => "%{[@metadata][rabbitmq_headers][Subject]}"}
  }
}

output {
  opensearch {
    hosts => ["opensearch:9200"]
    ssl => true
    ssl_certificate_verification => false
    user => "osearch-user"
    password => "BOGUS_PASSWORD"
    index => "test-rabbitmq-logstash-pipeline"
  }
  stdout { codec => rubydebug }
}

```

But it seems that I may have missed something, or made a mistake, as I cannot get the email info required.

Anyone able to see what might be wrong?

Cheers,  
Eddie.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2024, 8:25am UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711/2 "2024-02-07T08:25:17Z")

</div>

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 7, 2024, 5:59pm UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711/3 "2024-02-07T17:59:31Z")

</div>

> [@Big-Edd](#):
>
> `stdout { codec => rubydebug }`

I suggest you add `{ metadata => true }` as the codec options to verify that logstash is tagging the event with the metadata.

---

<div class="post-metadata">

**Author:** ![Big-Edd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/big-edd/32/131503_2.png) [@Big-Edd](https://discuss.elastic.co/u/Big-Edd)\
**Post date:** [February 12, 2024, 2:43am UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711/4 "2024-02-12T02:43:12Z")

</div>

Yes it does show "routing-key" and "Subject" in the RabbitMQ metadata.

```auto
{
       "@timestamp" => 2024-02-09T08:58:06.541860511Z,
        "@metadata" => {
        "rabbitmq_properties" => {
             "consumer-tag" => "amq.ctag-_36BYl7dfFbFo5s8kAOZEQ",
             "content-type" => "text/plain",
            "delivery-mode" => 2,
               "message-id" => "9054ce4cf059eadb9e4c809f3926d798",
              "routing-key" => "smtp_test@example.com",
                "timestamp" => 1707469086,
                 "exchange" => "smtp_exchange"
        },
           "rabbitmq_payload" => "Hello. Timestamp is 2024-02-09T16:58:10.721259 for email send.",
           "rabbitmq_headers" => {
            "Subject" => "SMTP MQ Test",
               "From" => "random_sender@invalid_domain"
        }
    },
       "email_dest" => "smtp_test@example.com",
         "@version" => "1",
          "message" => "Hello. Timestamp is 2024-02-09T16:58:10.721259 for email send.",
            "event" => {
        "original" => "Hello. Timestamp is 2024-02-09T16:58:10.721259 for email send."
    },
    "email_subject" => "SMTP MQ Test",
             "tags" => [
        [0] "test",
        [1] "RabbitMQ",
        [2] "smtp"
    ]
}

```

And it also shows email\_dest and email\_subject fields added, as required, after correcting the logstash.conf file.

```auto
input {
  rabbitmq {
    codec => "plain"
    tags => ["test","RabbitMQ","smtp"]
    host => "rabbitmq"
    port => 5672
    queue => "queue_t"
    durable => true
    passive => true
    user => "consumer_t"
    password => "BOGUS_PASSWORD"
    metadata_enabled => "extended"
  }
}
filter {
  mutate {
    add_field => { "email_dest" => "%{[@metadata][rabbitmq_properties][routing-key]}" }
    add_field => { "email_subject" => "%{[@metadata][rabbitmq_headers][Subject]}" }
  }
}
output {
  opensearch {
    hosts => ["opensearch:9200"]
    ssl => true
    ssl_certificate_verification => false
    user => "osearch-user"
    password => "BOGUS_PASSWORD"
    index => "test-rabbitmq-logstash-pipeline"
  }
  stdout { codec => rubydebug { metadata => true } }
}

```

Thanks very much for you help.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 12, 2024, 3:01am UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711/5 "2024-02-12T03:01:29Z")

</div>

I would argue that this is a bug. Processing the common options add\_field / add\_tag / remove\_field / remove\_tag is called decoration. The input [calls decorate event](https://github.com/logstash-plugins/logstash-input-rabbitmq/blob/04e48c218368cc7c318dfc8d934a9dd9c6fc4fb5/lib/logstash/inputs/rabbitmq.rb#L252) before it adds the rabbitmq metadata. I think that is the wrong way around.

As you have discovered, the workaround is to create the email fields using a mutate filter.

---

<div class="post-metadata">

**Author:** ![Big-Edd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/big-edd/32/131503_2.png) [@Big-Edd](https://discuss.elastic.co/u/Big-Edd)\
**Post date:** [February 16, 2024, 1:46am UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711/6 "2024-02-16T01:46:37Z")

</div>

Thanks for clarifying.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 15, 2024, 1:47am UTC](https://discuss.elastic.co/t/amqp-metadata-from-rabbitmq-input-plugin/352711/7 "2024-03-15T01:47:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.