# @metadata get lost in logstash redis output/input

**URL:** <https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080>\
**Category:** Logstash\
**Created:** [December 11, 2017, 12:07pm UTC](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080 "2017-12-11T12:07:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fabian](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fabian](https://discuss.elastic.co/u/fabian)\
**Post date:** [December 11, 2017, 12:07pm UTC](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080/1 "2017-12-11T12:07:05Z")

</div>

Hey there, I'm trying to give the filebeat modules a try but stuck in process the @metadata field. My setup looks like

filebeat \> logstash forwarder \> redis \> logstash processor \> elasticsearch

> **Config logstash forwarder:**
>
> input {  
> beats {  
> port =\> 5044  
> }  
> }  
> output {  
> redis {  
> host =\> ["redis1", "redis2", "redis3"]  
> data\_type =\> "list"  
> key =\> "logstash"  
> codec =\> json  
> }

> **Config logstash processor:**
>
> input {  
> redis {  
> host =\> "redis1"  
> key =\> "logstash"  
> data\_type =\> "list"  
> codec =\> json  
> }  
> redis {  
> host =\> "redis2"  
> key =\> "logstash"  
> data\_type =\> "list"  
> codec =\> json  
> }  
> redis {  
> host =\> "redis3"  
> key =\> "logstash"  
> data\_type =\> "list"  
> codec =\> json  
> }  
> }  
> filter {  
> if [fileset][module] == "system" {  
> if [fileset][name] == "auth" {  
> grok {  
> match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",  
> "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",  
> "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped\_ip]}",  
> "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: \s\*%{DATA:[system][auth][user]} ☹ %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]}  
> ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",  
> "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",  
> "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:[%{POSINT:[system][auth][pid]}])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",  
> "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid]}])?: %{GREEDYMULTILINE:[system][auth][message]}"] }  
> pattern\_definitions =\> {  
> "GREEDYMULTILINE"=\> "(.|\n)_"  
> }  
> remove\_field =\> "message"  
> }  
> date {  
> match =\> ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]  
> }  
> geoip {  
> source =\> "[system][auth][ssh][ip]"  
> target =\> "[system][auth][ssh][geoip]"  
> }  
> }  
> else if [fileset][name] == "syslog" {  
> grok {  
> match =\> { "message" =\> ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }  
> pattern\_definitions =\> { "GREEDYMULTILINE" =\> "(.|\n)_" }  
> remove\_field =\> "message"  
> }  
> date {  
> match =\> ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]  
> }  
> }  
> }  
> }  
> output {  
> elasticsearch {  
> hosts =\> localhost  
> manage\_template =\> false  
> index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  
> }  
> }

The index created is %{[@metadata][beat]}-%{[@metadata][version]}-2012-12-11.

If I change the output in logstash forwarder to stdout { codec =\> rubydebug {metadata =\> true } } exept of redis the @metadata fields exists. But when I route it through redis and change output in logstash processor to stdout { codec =\> rubydebug {metadata =\> true } } there are no @metadata fields anymore.

Is there a way to use redis output/input and keep @metadata fields?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 11, 2017, 12:25pm UTC](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080/2 "2017-12-11T12:25:58Z")

</div>

> Is there a way to use redis output/input and keep @metadata fields?

No, you'll have to rename the fields. `@metadata` is, by design, ignored by output (at least one exception exists; the rubydebug codec has a `metadata` option that causes `@metadata` to be included).

---

<div class="post-metadata">

**Author:** ![fabian](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fabian](https://discuss.elastic.co/u/fabian)\
**Post date:** [December 11, 2017, 12:56pm UTC](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080/3 "2017-12-11T12:56:44Z")

</div>

Thanks for your fast reply. Is there a any reason why it is designed like that?

What I did now is to put an additional field in filebeat an create my indexes this way. For anyone who is interested:

> **filebeat config**
>
> ```
> ...
> fields_under_root: true
> fields:
> beattype: filebeat
> ...
> 
> ```

> **logstash output**
>
> ```
> output {
> elasticsearch {
> hosts => localhost
> manage_template => false
> index => "%{[beattype]}-%{[beat][version]}-%{+YYYY.MM.dd}"
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 11, 2017, 1:21pm UTC](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080/4 "2017-12-11T13:21:57Z")

</div>

> Is there a any reason why it is designed like that?

Metadata fields are supposed to be used as helpers fields (like local variables) within a Logstash instance with no risk of leaking the fields to outputs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 8, 2018, 1:22pm UTC](https://discuss.elastic.co/t/metadata-get-lost-in-logstash-redis-output-input/111080/5 "2018-01-08T13:22:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.