# Passing to output index field but getting \[@metadata\]\[index\]

**URL:** <https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650>\
**Category:** Logstash\
**Created:** [March 15, 2020, 9:27am UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650 "2020-03-15T09:27:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Burga](https://avatars.discourse-cdn.com/v4/letter/b/9d8465/32.png) [@Burga](https://discuss.elastic.co/u/Burga)\
**Post date:** [March 15, 2020, 9:27am UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/1 "2020-03-15T09:27:14Z")

</div>

Hi , i'm creating index name dynamically but when I'm passing the field to output

i'm getting the variable name ([@metadata][index]):

```
 mutate {
  add_field => { "[@metadata][index_prefix]" => "system-secure" }
}

```

output :

output {  
elasticsearch {  
hosts =\> ["il-infra-es1", "il-infra-es2", "il-infra-es3"]  
manage\_template =\> false  
index =\> "%[@metadata][index\_prefix]-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> line {format =\> "Rate: %[@metadata][index\_prefix]"} }  
}

and getting --\> Rate: %[@metadata][index\_prefix]

please assist .

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 15, 2020, 9:33am UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/2 "2020-03-15T09:33:21Z")

</div>

You are missing braces. `%[@metadata][index_prefix]` should be `%{[@metadata][index_prefix]}`.

Rob

[![GitHub](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f8ae834f16b1a02d31607317669716807844d84.png)](https://github.com/robcowart) [![YouTube](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43b9b81a8c93786219985aeb5335c1c323449053.png)](https://www.youtube.com/channel/UCivWvTx1DwrWNcDLV58kmOg) [![LinkedIn](https://us1.discourse-cdn.com/elastic/original/3X/6/7/674f3370d0f0542ddc5e408516beb1b7edd6c1bf.png)](https://www.linkedin.com/in/robertcowart/)  
**[How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations](https://www.youtube.com/watch?v=gZb7HpVOges)**  
**[What is the best storage technology for Elasticsearch?](https://www.youtube.com/watch?v=nKUpfJCBiS4)**

---

<div class="post-metadata">

**Author:** ![Burga](https://avatars.discourse-cdn.com/v4/letter/b/9d8465/32.png) [@Burga](https://discuss.elastic.co/u/Burga)\
**Post date:** [March 15, 2020, 11:40am UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/3 "2020-03-15T11:40:12Z")

</div>

> [@rcowart](#):
>
> %{[@metadata][index\_prefix]}

after the change I'm getting the same result 🙂  
stdout { codec =\> line {format =\> "%{[@metadata][index\_prefix]}"} }

result:

Mar 15 13:39:06 il-infra-logs2 logstash[17753]: %{[@metadata][index\_prefix]}

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 15, 2020, 12:01pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/4 "2020-03-15T12:01:56Z")

</div>

Usually that happens when the field isn't set. To better see what is included in the event, add an output with the `rubydebug` codec and enable inclusion of `metadata`.

```auto
output {
  stdout {
    codec => rubydebug {
      metadata => "true"
    }
  }
}

```

BTW, please enclose your examples in a proper code block, as I did above, so it displays easier. You do this by putting three back ticks on the lines before and after your code.

Rob

[![GitHub](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f8ae834f16b1a02d31607317669716807844d84.png)](https://github.com/robcowart) [![YouTube](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43b9b81a8c93786219985aeb5335c1c323449053.png)](https://www.youtube.com/channel/UCivWvTx1DwrWNcDLV58kmOg) [![LinkedIn](https://us1.discourse-cdn.com/elastic/original/3X/6/7/674f3370d0f0542ddc5e408516beb1b7edd6c1bf.png)](https://www.linkedin.com/in/robertcowart/)  
**[How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations](https://www.youtube.com/watch?v=gZb7HpVOges)**  
**[What is the best storage technology for Elasticsearch?](https://www.youtube.com/watch?v=nKUpfJCBiS4)**

---

<div class="post-metadata">

**Author:** ![Burga](https://avatars.discourse-cdn.com/v4/letter/b/9d8465/32.png) [@Burga](https://discuss.elastic.co/u/Burga)\
**Post date:** [March 15, 2020, 12:35pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/5 "2020-03-15T12:35:14Z")

</div>

yes the field wasn't set , somehow it didn't get into the

```auto

if [fileset][module] == "system" {

```

and I don't know why when I put the line right after the filter it sets the field but after the

```auto
 [fileset][module] == "system" it doesn't 

```

it doesn't

My code is

```auto
filter {

    
	   mutate {
      add_field => { "[@metadata][index_prefix]" => "filter" }
    }

  if [fileset][module] == "system" {
  
  
	   mutate {
      add_field => { "[@metadata][index_prefix]" => "system" }
    }
  
    if [fileset][name] == "auth" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
        pattern_definitions => {
          "GREEDYMULTILINE"=> "(.|\n)*"
        }
        remove_field => "message"
      }
      date {
        match => ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
      geoip {
        source => "[system][auth][ssh][ip]"
        target => "[system][auth][ssh][geoip]"
      }
	 
	  
	  
    }
    else if [fileset][name] == "syslog" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
        pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
        remove_field => "message"
      }
      date {
        match => ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
	  
	  
    }
  }

output {
  elasticsearch {
    hosts => ["il-infra-es1", "il-infra-es2", "il-infra-es3"]
    manage_template => false
    index => "%[@metadata][index_prefix]-%{+YYYY.MM.dd}"
  }
stdout { codec => line {format => "%{[@metadata][index_prefix]}"} }
}

```

In my original code I'm trying to create different indexes to different types (syslog,auth )

the second day I'm sitting on this issue , can you please direct me what I'm doing wrong

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 15, 2020, 1:19pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/6 "2020-03-15T13:19:45Z")

</div>

Did you output using `rubydebug`, including `metadata => true`, as I suggested? I think it would help to see what is going on here.

Let's reduce this to a minimum...

```auto
mutate {
  add_field => { "[@metadata][index_prefix]" => "filter" }
}

if [fileset][module] == "system" {
  mutate {
    add_field => { "[@metadata][index_prefix]" => "system" }
  }

  ...

}

```

That first `mutate` will set `[@metadata][index_prefix]` to `filter`.

If `[fileset][module]` equals `system` the second `mutate` will _add another_ value to `[@metadata][index_prefix]`. The result being an _array_ of two values, `[filter, system]`. You should be able to see this if you output the values as I suggested.

In this case when refer to `%{[@metadata][index_prefix]}` Logstash doesn't know which value to use, and you just get the variable name.

If you intend this second `mutate` to replace the original value, you need to use `replace` instead of `add_field`. In fact, if I am setting a field's value using `mutate` and I know I never want that field to hold more than a single value, I will always use `replace`. If the field doesn't already exist, it will be created. So there is no downside to using `replace`.

BTW, don't forget to change `index => "%[@metadata][index_prefix]-%{+YYYY.MM.dd}"` to include braces `index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"`.

Rob

[![GitHub](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f8ae834f16b1a02d31607317669716807844d84.png)](https://github.com/robcowart) [![YouTube](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43b9b81a8c93786219985aeb5335c1c323449053.png)](https://www.youtube.com/channel/UCivWvTx1DwrWNcDLV58kmOg) [![LinkedIn](https://us1.discourse-cdn.com/elastic/original/3X/6/7/674f3370d0f0542ddc5e408516beb1b7edd6c1bf.png)](https://www.linkedin.com/in/robertcowart/)  
**[How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations](https://www.youtube.com/watch?v=gZb7HpVOges)**  
**[What is the best storage technology for Elasticsearch?](https://www.youtube.com/watch?v=nKUpfJCBiS4)**

---

<div class="post-metadata">

**Author:** ![Burga](https://avatars.discourse-cdn.com/v4/letter/b/9d8465/32.png) [@Burga](https://discuss.elastic.co/u/Burga)\
**Post date:** [March 15, 2020, 1:44pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/7 "2020-03-15T13:44:20Z")

</div>

> [@rcowart](#):
>
> stdout { codec =\> rubydebug { metadata =\> "true" } }

Yes I ran it as you suggested and I get no value set for this field when

it is placed inside the if

```auto
if [fileset][module] == "system" {
  
  
	   mutate {
      add_field => { "[@metadata][index_prefix]" => "system" }
    }

```

when it outside it has the value of "filter" , I suppose it means that it simply doesn't get inside because the

```auto
[fileset][module] == "system"

```

is false ?

this is what i get :

Mar 15 15:36:24 il-infra-logs2 logstash[758]: "log" =\> {  
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "file" =\> {  
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "path" =\> "/var/log/messages"  
Mar 15 15:36:24 il-infra-logs2 logstash[758]: },  
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "offset" =\> 18645146  
Mar 15 15:36:24 il-infra-logs2 logstash[758]: },  
**Mar 15 15:36:24 il-infra-logs2 logstash[758]: "@metadata" =\> {**  
**Mar 15 15:36:24 il-infra-logs2 logstash[758]: "version" =\> "7.6.1",**  
**Mar 15 15:36:24 il-infra-logs2 logstash[758]: "ip\_address" =\> "192.168.127.230",**  
**Mar 15 15:36:24 il-infra-logs2 logstash[758]: "beat" =\> "filebeat",**  
**Mar 15 15:36:24 il-infra-logs2 logstash[758]: "type" =\> "\_doc"**  
Mar 15 15:36:24 il-infra-logs2 logstash[758]: },

how can I check what is the value of [fileset][module] ?

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 15, 2020, 1:58pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/8 "2020-03-15T13:58:10Z")

</div>

If you don't see it in the `rubydebug` output, then it doesn't exist.

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 15, 2020, 2:02pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/9 "2020-03-15T14:02:13Z")

</div>

I think what you are looking for in current versions of Beats is `event.module`.

---

<div class="post-metadata">

**Author:** ![Burga](https://avatars.discourse-cdn.com/v4/letter/b/9d8465/32.png) [@Burga](https://discuss.elastic.co/u/Burga)\
**Post date:** [March 15, 2020, 5:50pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/10 "2020-03-15T17:50:12Z")

</div>

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2020, 5:50pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650/11 "2020-04-12T17:50:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.