# Sending @metadata from logstash to elastic search

**URL:** <https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890>\
**Category:** Logstash\
**Created:** [July 12, 2017, 10:20pm UTC](https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890 "2017-07-12T22:20:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![darraghjones](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darraghjones/32/20017_2.png) [@darraghjones](https://discuss.elastic.co/u/darraghjones)\
**Post date:** [July 12, 2017, 10:20pm UTC](https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890/1 "2017-07-12T22:20:00Z")

</div>

I've got the following logstash config, and I'm trying to send the RabbitMQ headers (which are stored in the @metadata field) to ElasticSearch

```
input {
    rabbitmq {
        auto_delete => false
        durable => false
        host => "my_host"
        port => 5672
        queue => "my_queue"
        key => "#"      
        threads => 1
        codec => "plain"
        user => "user"
        password => "pass"
        metadata_enabled => true
    }
}

filter {
    ???
}

output {
    stdout { codec => rubydebug {metadata => true} }
    elasticsearch { hosts => localhost }
}

```

I can see the headers in the std output

```
{
    "@timestamp" => 2017-07-11T15:53:28.629Z,
     "@metadata" => {
           "rabbitmq_headers" => { "My_Header" => "My_value"
        },
        "rabbitmq_properties" => {
            "content-encoding" => "utf-8",
              "correlation-id" => "785901df-e954-4735-a9cf-868088fdac87",
                "content-type" => "application/json",
                    "exchange" => "My_Exchange",
                 "routing-key" => "123-456",
                "consumer-tag" => "amq.ctag-ZtX3L_9Zsz96aakkSGYzGA"
        }
    },
      "@version" => "1",
       "message" => "{...}"

```

Is there some filter (grok, mutate, kv, etc.) which can copy these values to Tags in the message sent to ElasticSearch?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 13, 2017, 6:47am UTC](https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890/2 "2017-07-13T06:47:10Z")

</div>

You can use a mutate filter to rename the fields you want to keep. Note the syntax for how to reference nested fields.

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename)  
[https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references)

---

<div class="post-metadata">

**Author:** ![darraghjones](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darraghjones/32/20017_2.png) [@darraghjones](https://discuss.elastic.co/u/darraghjones)\
**Post date:** [July 17, 2017, 1:37pm UTC](https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890/3 "2017-07-17T13:37:20Z")

</div>

I'd like to keep all the metadata fields...so I've tried:

```
filter {
	mutate {
		rename => {"@metadata" => "meta"}
	}
}

```

But it's not working.

---

<div class="post-metadata">

**Author:** ![josephjohney](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@josephjohney](https://discuss.elastic.co/u/josephjohney)\
**Post date:** [July 17, 2017, 2:40pm UTC](https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890/4 "2017-07-17T14:40:40Z")

</div>

The contents of the @metadata field only exist in Logstash and are not part of any events sent from Logstash.

We can however use mutate to create/retrieve fields from @metadata within logstash pipeline.

like mutate { add\_field =\> { "[@metadata][test]" =\> "Hello" } }

and use it as below  
output {  
if [@metadata][test] == "Hello" {  
stdout { codec =\> rubydebug }  
}  
}

You can also create new fields using the existing metadata information. But you have to use it similar to [@metadata][test] where you define the metadata attribute name as well.

Regards,  
Joseph

---

<div class="post-metadata">

**Author:** ![darraghjones](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darraghjones/32/20017_2.png) [@darraghjones](https://discuss.elastic.co/u/darraghjones)\
**Post date:** [July 17, 2017, 3:15pm UTC](https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890/5 "2017-07-17T15:15:08Z")

</div>

Ended up doing this:

```
ruby {
	code => 'event.get("[@metadata][rabbitmq_headers]").each {|k,v| event.set(k, v)}'
}

```

Let me know if there's a more 'declarative' way.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2017, 3:15pm UTC](https://discuss.elastic.co/t/sending-metadata-from-logstash-to-elastic-search/92890/6 "2017-08-14T15:15:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.