# Multiple Fields substitution doesn't work in logstash

**URL:** <https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119>\
**Category:** Logstash\
**Created:** [April 12, 2017, 8:48am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119 "2017-04-12T08:48:18Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![vramakrishnan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vramakrishnan/32/31750_2.png) [@vramakrishnan](https://discuss.elastic.co/u/vramakrishnan)\
**Post date:** [April 12, 2017, 8:48am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/1 "2017-04-12T08:48:18Z")

</div>

I have annotated filebeat to add these fields.

"fields" =\> {  
"provider" =\> "foobar",  
"resourcetype" =\> "ipmi",  
"customer" =\> "customer001"  
},

And in logstash conf file, I want to use these fields in combination to generate unique elastic index.  
I tried these format and they are not expanding correctly.

Please let me know what is the right syntax.

output {  
elasticsearch {  
**index =\> "%{[fields][resourcetype]}.%{[fields][customer]}.%{[fields.provider]}.%{+YYYY.MM.dd}"**

```
 **index generated by log stash is : %{[fields][resourcetype].[fields][customer].[fields.provider]}.2017.04.12**  

```

Using single field as listed below works, but multiple fields doesn't work.  
index =\> "%{[fields][resourcetype]}.%{+YYYY.MM.dd}"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 12, 2017, 8:59am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/2 "2017-04-12T08:59:51Z")

</div>

The index name indicates that your configuration looks like

```
index => "%{[fields][resourcetype].[fields][customer].[fields.provider]}.%{+YYYY.MM.dd}"

```

when it should look like this:

```
index => "%{[fields][resourcetype]}.%{[fields][customer]}.%{[fields.provider]}.%{+YYYY.MM.dd}"
```

---

<div class="post-metadata">

**Author:** ![vramakrishnan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vramakrishnan/32/31750_2.png) [@vramakrishnan](https://discuss.elastic.co/u/vramakrishnan)\
**Post date:** [April 12, 2017, 4:21pm UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/3 "2017-04-12T16:21:20Z")

</div>

This format below worked.  
index =\> "%{[fields][resourcetype]}.%{[fields][customer]}.%{[fields][provider]}.%{+YYYY.MM.dd}"

However, when trying as follows, I am not able to get it working:

if [type] == "kafka-input" {  
mutate {  
**add\_field =\> { "elasticindex" =\> "%{[fields][resourcetype]}.%{[fields][customer]}.%{[fields][provider]}" }**  
}  
}

output {  
elasticsearch {  
**index =\> "%{elasticindex}.%{+YYYY.MM.dd}"**

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 13, 2017, 7:18am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/4 "2017-04-13T07:18:24Z")

</div>

Please show an example event where this didn't work, e.g. by showing the result of a `stdout { codec => rubydebug }` output or by copying/pasting from the JSON tab in Kibana.

---

<div class="post-metadata">

**Author:** ![vramakrishnan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vramakrishnan/32/31750_2.png) [@vramakrishnan](https://discuss.elastic.co/u/vramakrishnan)\
**Post date:** [April 14, 2017, 3:34am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/5 "2017-04-14T03:34:02Z")

</div>

Json log from Kibana

{  
**"\_index": "%{elasticindex}.2017.04.14",**  
"\_type": "log",  
"\_id": "AVtqhK88H217HTENKGJN",  
"\_score": null,  
"\_source": {  
"@timestamp": "2017-04-14T03:32:09.809Z",  
"offset": 5686,  
"beat": {  
"hostname": "3495c2daccca",  
**"name": "FileBeat",**  
\*\* "version": "5.2.0"\*\*  
},  
"input\_type": "log",  
"@version": "1",  
"source": "/Users/xyz/work/cplane/testoutput/everest.test.INFO",  
**"fields": {**  
\*\* "provider": "provider1",\*\*  
\*\* "service": "everest",\*\*  
\*\* "resourcetype": "csos",\*\*  
\*\* "customer": "infra"\*\*  
\*\* },\*\*  
"message": "I0414 03:32:05.056421 15270 election.go:249] service/everestd/leader: get CreateIndex 3310 ModifyIndex 8115 LockIndex 6 Session 1037e02c-cb1b-e391-61c7-ed08e74b3b67 Value [ohknauzv.everest.provider1.com](http://ohknauzv.everest.provider1.com)",  
"type": "log"  
},  
"fields": {  
"@timestamp": [  
1492140729809  
]

---

<div class="post-metadata">

**Author:** ![vramakrishnan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vramakrishnan/32/31750_2.png) [@vramakrishnan](https://discuss.elastic.co/u/vramakrishnan)\
**Post date:** [April 14, 2017, 6:05am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/6 "2017-04-14T06:05:00Z")

</div>

I tried a simple add\_field with value as a static string and that substitution is also not working with logstash 5.2 or 5.3 for the logstash conf shown below

**Logstash.conf**

input {

kafka {  
bootstrap\_servers =\> ["10.1.1.1:9092"]  
type =\> "kafka"  
topics =\> ["csoslogs.infra", "ipmi.customer001", "ipmi.customer002"]  
codec =\> "json"  
decorate\_events =\> true  
}

}

filter {

if [type] == "syslog" {  
mutate {  
add\_field =\> { "elasticindex" =\> "%{host}" }  
}  
}

if [type] == "kafka" {  
mutate {  
**add\_field =\> { "elasticindex" =\> "10.6.6.1" }**  
}  
}

output {  
stdout { codec =\> rubydebug }  
}

output {  
elasticsearch {  
hosts =\> ["104.42.236.238:9200"]  
user =\> "es\_admin"  
password =\> "Cloudsimple123!"  
**index =\> "%{elasticindex}.%{+YYYY.MM.dd}"**  
}  
}

**JSON seen in kibana :**

{  
**"\_index": "%{elasticindex}.2017.04.14",**  
"\_type": "log",  
"\_id": "AVtrBCiSH217HTENKISm",  
"\_score": null,  
"\_source": {  
"@timestamp": "2017-04-14T05:51:24.476Z",  
"offset": 71259,  
"kafka": {  
"consumer\_group": "logstash",  
"partition": 0,  
"offset": 5910,  
"topic": "csoslogs.infra",  
"key": null  
},  
"beat": {  
"hostname": "3495c2daccca",  
"name": "FileBeat",  
"version": "5.2.0"  
},  
"input\_type": "log",  
"@version": "1",  
"source": "/Users/vramakrishnan/work/cplane/testoutput/coordinator.test.INFO",  
"fields": {  
"podip": "10.6.6.1",  
"provider": "provider1",  
"service": "coordinator",  
"resourcetype": "csos",  
"customer": "infra"  
},  
"message": "I0414 05:51:18.736575 27476 consul\_helper.go:286] 0: created consul session 79c34ca4-a1f6-326b-9d5d-43eb62626f3c, so agent client-0 is initialized",  
"type": "log"  
},  
"fields": {  
"@timestamp": [  
1492149084476  
]  
},  
"sort": [  
1492149084476  
]  
}

---

<div class="post-metadata">

**Author:** ![vramakrishnan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vramakrishnan/32/31750_2.png) [@vramakrishnan](https://discuss.elastic.co/u/vramakrishnan)\
**Post date:** [April 16, 2017, 5:38am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/7 "2017-04-16T05:38:31Z")

</div>

Any workaround or suggestion for this issue is greatly appreciated.

This has turned into a stopper for us and kindly help.

---

<div class="post-metadata">

**Author:** ![vramakrishnan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vramakrishnan/32/31750_2.png) [@vramakrishnan](https://discuss.elastic.co/u/vramakrishnan)\
**Post date:** [April 17, 2017, 3:02am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/8 "2017-04-17T03:02:07Z")

</div>

It turns out the "type == "kafka" check in logstash filter is not working and hence not entering inside the if-block.

This used to work in logstash 5.1.1 and is use of type in filter deprecated ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 18, 2017, 5:31am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/9 "2017-04-18T05:31:57Z")

</div>

> It turns out the "type == "kafka" check in logstash filter is not working and hence not entering inside the if-block.

Yeah, the event's `type` field contains "log" and not "kafka".

> This used to work in logstash 5.1.1 and is use of type in filter deprecated ?

No, but it looks like `type => "kafka"` in the input doesn't work. Maybe it's because Filebeat already sets the `type` field to "log" and the kafka input won't override that value? I suggest you set the event type as close to the source as possible, in this case in Filebeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 16, 2017, 5:45am UTC](https://discuss.elastic.co/t/multiple-fields-substitution-doesnt-work-in-logstash/82119/10 "2017-05-16T05:45:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.