# Badly formatted index, after interpolation still contains placeholder **URL:** \ **Category:** Logstash\ **Tags:** ilm-index-lifecycle-management, datastreams\ **Created:** [October 17, 2023, 12:12pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205 "2023-10-17T12:12:13Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![John\_paul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_paul/32/126594_2.png) [@John\_paul](https://discuss.elastic.co/u/John_paul)\ **Post date:** [October 17, 2023, 12:12pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/1 "2023-10-17T12:12:13Z")
Hey guys, facing this below issue, was working fine while using indexes directly but started failing when configured to use data streams, ```auto ERROR: elasticsearch - Badly formatted index, after interpolation still contains placeholder: [logs-ssc-misc-%{[instance_name]}-%{[instance_IP]}-nonprod] ``` ```auto EVENT: { "agent""=>"{ "hostname""=>""my-node", "name""=>""my-node", "id""=>""yyy", "type""=>""filebeat", "ephemeral_id""=>""c44-c44-c44-4c4-4c4", "version""=>""7.8.1" }, "instance_name""=>""my_node_logs", "log""=>"{ "file""=>"{ "path""=>""/usr/share/filebeat/a.log" }, "offset"=>0 }, "level""=>""DEBUG", "@metadata""=>"{ "version""=>""7.8.1", "beat""=>""filebeat", "input""=>"{ "beats""=>"{ "host""=>"{ "ip""=>""0.0.0.0" } } }, "type""=>""_doc" }, "logger""=>""{org.apache}", "instance_IP""=>""0.0.0.0", "message""=>yy" } ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 17, 2023, 12:20pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/2 "2023-10-17T12:20:48Z")
This event you shared is before or after it runs through logstash? The error you got means that the fields `instance_name` and `instance_IP` does not exist when the event arrives at the output level, this can happen if you are not parsing your original message. What is your source? Please share your Logstash configuration. Also, if possible adds a `stdout` or `file` output and share the event that logstash is sending to the outputs. ---
**Author:** ![John\_paul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_paul/32/126594_2.png) [@John\_paul](https://discuss.elastic.co/u/John_paul)\ **Post date:** [October 17, 2023, 12:35pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/3 "2023-10-17T12:35:26Z")
```auto input { beats { port => 5044 } } filter { mutate { gsub => [ # Replace hyphens with underscores in instance_name "instance_name", "-", "_" ] } } output { elasticsearch { hosts => ["${ELASTICSEARCH_HOST1}", "${ELASTICSEARCH_HOST2}", "${ELASTICSEARCH_HOST3}"] user => "${ELASTICSEARCH_USERNAME}" password => "${ELASTICSEARCH_PASSWORD}" cacert => "/usr/share/logstash/certs/ca.pem" data_stream => "true" data_stream_type => "logs" data_stream_dataset => "ssc-misc-%{[instance_name]}-%{[instance_IP]}" data_stream_namespace => "nonprod" } } ``` ---
**Author:** ![John\_paul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_paul/32/126594_2.png) [@John\_paul](https://discuss.elastic.co/u/John_paul)\ **Post date:** [October 17, 2023, 12:39pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/4 "2023-10-17T12:39:05Z")
The event i have posted above is taken from the logstash logs .i.e. the event logstash received. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 17, 2023, 12:41pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/5 "2023-10-17T12:41:00Z")
> [@John\_paul](#): > > the event logstash received. You need to share the event that logstash is ending to see if it needs any parse or not. Add a file output temporarily and share some lines: ```auto file { path => "/tmp/temp-output-logstash.json" } ``` ---
**Author:** ![John\_paul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_paul/32/126594_2.png) [@John\_paul](https://discuss.elastic.co/u/John_paul)\ **Post date:** [October 17, 2023, 2:55pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/6 "2023-10-17T14:55:26Z")
this is the json content ```auto { "@version":"1", "level":"DEBUG", "message":[ "a", "b" ], "@timestamp":"2023-10-17T14:41:08.497Z", "tags":[ "beats_input_codec_plain_applied" ], "input":{ "type":"log" }, "tid":"-1234", "log":{ "offset":5052393, "file":{ "path":"/usr/share/filebeat/a.log" } }, "instance_IP":"0.0.0.0", "timestamp":"2023-10-17 14:41:08,497", "logger":"{org.apache.synapse.transport.http.wire}", "agent":{ "name":"my-node", "version":"7.8.1", "hostname":"my-node", "id":"1234", "ephemeral_id":"1234", "type":"filebeat" }, "ecs":{ "version":"1.5.0" }, "event":{ }, "instance_name":"my_node_logs", "host":{ "name":"my_node" }, "type":"mynode" } ``` ---
**Author:** ![John\_paul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_paul/32/126594_2.png) [@John\_paul](https://discuss.elastic.co/u/John_paul)\ **Post date:** [October 17, 2023, 3:07pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/7 "2023-10-17T15:07:16Z")
This is the index template pattern mapped to ILM policy with DS enabled, ```auto logs-ssc-misc-*-*-nonprod* ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 17, 2023, 4:06pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/8 "2023-10-17T16:06:34Z")
> [@John\_paul](#): > > this is the json content This is the content of the `file` output in Logstash? Not sure what the issue is, the fields are present in the document. Does this happens for every document? ---
**Author:** ![John\_paul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_paul/32/126594_2.png) [@John\_paul](https://discuss.elastic.co/u/John_paul)\ **Post date:** [October 17, 2023, 4:07pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/9 "2023-10-17T16:07:11Z")
yes, all events are failing. I have other outputs, which has static names, that works fine. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 17, 2023, 4:14pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/10 "2023-10-17T16:14:26Z")
Just found a similar issue with a solution, check this [post](https://discuss.elastic.co/t/manage-several-data-stream-s-in-the-elasitcsearch-output-with-interpolation/332981/2). You will need to some `data_stream` fields, basically you will need to create the fields: - `datas_tream.type` - `data_stream.dataset` - `data_stream.namespace` You need this in your filter block: ```auto mutate { add_field => { "[data_stream][type]" => "logs" "[data_stream][dataset]" => "ssc-misc-%{[instance_name]}-%{[instance_IP]}" "[data_stream][namespace]" => "nonprod" } } ``` And remove those settings from the output. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 17, 2023, 4:28pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/11 "2023-10-17T16:28:05Z")
@leandrojmp @John_paul > [@leandrojmp](#): > > `"[data_stream][dataset]" => "ssc-misc-%{[instance_name]}-%{[instance_IP]}"` Caution: the dataset _ **can not** _ have `-` dashes in it ... dashes can _ **only** _ be used to separate the `type`, `dataset` and `namespace` So, the above is invalid. > **[Data streams | Fleet and Elastic Agent Guide \[8.10\] | Elastic](https://www.elastic.co/guide/en/fleet/8.10/data-streams.html#data-streams-naming-scheme)** > **[An introduction to the Elastic data stream naming scheme](https://www.elastic.co/blog/an-introduction-to-the-elastic-data-stream-naming-scheme)** > > Steaming data into Elasticsearch? In this blog post, we'll give an overview of the Elastic data stream naming scheme and how it works. > These three parts are combined by a “-” and result in data streams like `logs-nginx.access-production` . In all three parts, the “-” character is not allowed. This means all data streams are named in the following way: In short use `.` dots within the fields ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 17, 2023, 4:37pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/12 "2023-10-17T16:37:43Z")
Thanks @stephenb , didn't know about that. But this limitation only exists if you want to use the `data_stream` settings in the Elasticsearch output I think since it will validate the value of the fields. I'm using custom data streams names, so I can not use those `data_stream` settings and do not have this limitation. ---
**Author:** ![John\_paul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john_paul/32/126594_2.png) [@John\_paul](https://discuss.elastic.co/u/John_paul)\ **Post date:** [October 17, 2023, 4:39pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/13 "2023-10-17T16:39:08Z")
Worked like a charm. Thanks @leandrojmp . Life saver. yeah @stephenb i read the documentation, the hyphens are not allowed as DS will use hyphens to create the index names internally But it works without any issues even with hyphens. What i noticed is only in scenarios of creating dynamic names, during that interpolation its not allowing to use hyphens. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 17, 2023, 4:57pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/14 "2023-10-17T16:57:31Z")
I think it is more than just using those settings... To be clear, you can use custom names but more data stream functionality (free good stuff) is coming that will depend on the proper naming, like automatic routing, automatic custom pipelines based on the names, routing of mapping exceptions etc ... these are some of the things that I understand are on the future roadmap .. (I am playing with some of them now...) They will not be interpreted correctly in downstream or dependent operations. Proceed at your own risk / caution. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 17, 2023, 5:01pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/15 "2023-10-17T17:01:21Z")
Yeah, while I disagree with some things, I understand that approach. By custom data stream names I mean using something where the type is not `logs`, `metrics`, `traces` or `synthetics`. If you want to use something like `appname-prod` as a data stream name, you cannot use the `data_stream` settings in Logstash and need to index it as a normal indice pointing to the data stream name. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 14, 2023, 5:01pm UTC](https://discuss.elastic.co/t/badly-formatted-index-after-interpolation-still-contains-placeholder/345205/16 "2023-11-14T17:01:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.