# Filebeat error when setting up data stream with working index

**URL:** <https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 21, 2024, 7:19am UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254 "2024-08-21T07:19:19Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tom\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_n/32/136135_2.png) [@Tom\_N](https://discuss.elastic.co/u/Tom_N)\
**Post date:** [August 21, 2024, 7:19am UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/1 "2024-08-21T07:19:19Z")

</div>

I am using filebeat version 7.17.22.

At the start, my filebeat works and sends data to Elasticsearch, which can be seen using Kibana. It sends data to the index.

I have been trying to setup a datastream, so I did so by creating the datastream with the default index pattern filebeat created using the below command (learn from [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-data-stream.html))

```auto
PUT _data_stream/filebeat-7.17.22-

```

Then, I tried to aim my filebeat output to this datastream by putting the `output.elasticsearch.index` as `filebeat-7.17.22-`.

However, I got the error:

```auto
Exiting: error loading template: failed to load template: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"composable template [filebeat-7.17.22] with index patterns [filebeat-7.17.22-*], priority [150] and no data stream configuration would cause data streams [filebeat-7.17.22-] to no longer match a data stream template"}]

```

I tried searching it up online but couldn't really find anything that solved the issue.

My filebeat.yml files:

```auto
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /path/to/logs/*.txt

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.dashboards.enabled: false

setup.kibana:
  host: "http://localhost:5601"
  username: "USERNAMEHERE"
  password: "PASSWORDHERE"

output.elasticsearch:
  hosts: ["https://localhost:9200/"]
  preset: balanced
  protocol: "https"
  username: "USERNAMEHERE"
  password: "PASSWORDHERE"
  pipeline: "filebeat-pipeline"
  index: "filebeat-%{[agent.version]}-"

  ssl:
    enabled: true
    certificate_authorities: /path/to/ca_cert

setup.template.enabled: true
setup.template.name: "filebeat-%{[agent.version]}"
setup.template.pattern: "filebeat-%{[agent.version]}-*"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded

logging.level: debug

setup.ilm.enabled: false
setup.ilm.check_exists: true

```

Some links that I have been looking at to troubleshoot the issue: [url1](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325), [url2](https://discuss.elastic.co/t/does-filebeat-plan-to-support-data-streams/296851), [url3](https://discuss.elastic.co/t/filebeat-setup-error-loading-template-failed-to-put-data-stream-no-matching-index-template-found-for-data-stream/307789)

I also tried to migrate the index to a data stream according to [Migrate to data stream API](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-migrate-to-data-stream.html), but just got the error:

```auto
"type": "illegal_argument_exception",
        "reason": "no matching index template found for data stream [filebeat-7.17.22]"

```

when running the command:

```auto
POST /_data_stream/_migrate/filebeat-7.17.22

```

---

<div class="post-metadata">

**Author:** ![Trevor\_Blackford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trevor_blackford/32/120087_2.png) [@Trevor\_Blackford](https://discuss.elastic.co/u/Trevor_Blackford)\
**Post date:** [August 21, 2024, 12:59pm UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/2 "2024-08-21T12:59:32Z")

</div>

I think your error might be due to the setup.template parameters.  
%{[agent.version]} is automatically added to the name and pattern, so you are applying it twice, which no longer matches the output.

> **[Configure Elasticsearch index template loading | Filebeat Reference \[7.17\] |...](https://www.elastic.co/guide/en/beats/filebeat/7.17/configuration-template.html)**

---

<div class="post-metadata">

**Author:** ![Tom\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_n/32/136135_2.png) [@Tom\_N](https://discuss.elastic.co/u/Tom_N)\
**Post date:** [August 21, 2024, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/3 "2024-08-21T14:49:19Z")

</div>

Thank you for your reply! According to the documentation you gave me:

> The Filebeat version is always appended to the given name, so the final name is `filebeat-%{[agent.version]}`.

So, I believe that means that I can just get rid of the agent.version part and it will work?

Something like this?

```auto
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"

```

I will try this and get back to you if it works. Thank you!

---

<div class="post-metadata">

**Author:** ![Tom\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_n/32/136135_2.png) [@Tom\_N](https://discuss.elastic.co/u/Tom_N)\
**Post date:** [August 22, 2024, 3:19am UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/4 "2024-08-22T03:19:40Z")

</div>

Hi @Trevor_Blackford, tried what you suggested and no, it doesn't seem like filebeat automatically applies the `%{[agent.version]}` when you just have the default template name and pattern set.

I decided to make a new index template called `filebeat-7.17.22-test` with the same index pattern `filebeat-7.17.22-*`. I made a data stream called `filebeat-7.17.22-test` using the command:  
`PUT _data_stream/filebeat-7.17.22-test`.

Then, in my filebeat.yml file, I configured it as such:

```auto
output.elasticsearch:
  index: "filebeat-%{[agent.version]}-test"

setup.template.enabled: true
setup.template.name: "filebeat-%{[agent.version]}-test"
setup.template.pattern: "filebeat-%{[agent.version]}-*"

```

This did not work and I got the same error as before. I decided to try your advice again, but accidentally made a typo, naming the `setup.template.pattern` as `"filebea-*"`, as seen below

```auto
output.elasticsearch:
  index: "filebeat-%{[agent.version]}-test"

setup.template.enabled: true
setup.template.name: "filebeat-%{[agent.version]}-test"
setup.template.pattern: "filebea-*"

```

And somehow, this worked? The index template loaded and the data stream `filebeat-7.17.22-test` is taking the filebeat output. However, when I checked my data stream on Kibana, my data stream was not using the index template I had specified, but the default one filebeat creates, which is `filebeat-7.17.22`, seen below

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af0ef78cda6e5757c5cf6fcc88e5cf0f4212e3d2.png)

When I checked my index templates, `filebeat-7.17.22-test` did not even have the Data stream portion ticked, even though I had ticked it before?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5e3ecf101fb9630acb3c53cfbd98c51dc8b60d9d.png)

I tried changing the `setup.template.name` to `filebeat-7.17.22`, but that gave me the same error again.

I was wondering if you had any insight into how this is presumably working (Logs are being outputted with correct processing)?

---

<div class="post-metadata">

**Author:** ![Trevor\_Blackford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trevor_blackford/32/120087_2.png) [@Trevor\_Blackford](https://discuss.elastic.co/u/Trevor_Blackford)\
**Post date:** [August 22, 2024, 7:53pm UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/5 "2024-08-22T19:53:04Z")

</div>

Sorry to mislead you with the defaults. I can see that's the default setting that supplied with the default config.

The problem here indeed has to do with data streams, which must have certain mappings in their index template in order to work.

> **[Data streams | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html)**

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 22, 2024, 10:44pm UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/6 "2024-08-22T22:44:17Z")

</div>

@Tom_N Here is an easy way to get started

Clone the The Existing 7.17.22 Indext Template and rename it and set the data stream. It needs to not match the existing index pattern OR you will need to remove the default... that is all I changed and I save it (you should actually go in later and clean up the aliase etc.

 ![Screenshot 2024-08-22 at 3.36.16 PM](https://us1.discourse-cdn.com/elastic/original/3X/7/6/76bd5485c4dd7fdf4e1ba049e196302719f55cc8.png)

Create the data stream

`PUT _data_stream/filebeat-datastream-7.17.3`

My Entire working filebeat

```auto

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
# ======================= Elasticsearch template setting =======================
setup.ilm.enabled: false
setup.template.enabled: false

setup.kibana:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: filebeat-datastream-%{[agent.version]}

```

Then

`./filebeat -e`

There you go...

 ![Screenshot 2024-08-22 at 3.39.02 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/9/89434f38a484f33283d8a2f806e33b7b13fcfae4.png)

 ![Screenshot 2024-08-22 at 3.39.11 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/a/ea28cc7f408b479f61ede30ff50690bfef2491f5.png)

---

<div class="post-metadata">

**Author:** ![Tom\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_n/32/136135_2.png) [@Tom\_N](https://discuss.elastic.co/u/Tom_N)\
**Post date:** [August 23, 2024, 2:15am UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/7 "2024-08-23T02:15:38Z")

</div>

No worries, thank you for your help nonetheless!

---

<div class="post-metadata">

**Author:** ![Tom\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_n/32/136135_2.png) [@Tom\_N](https://discuss.elastic.co/u/Tom_N)\
**Post date:** [August 23, 2024, 2:23am UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/8 "2024-08-23T02:23:48Z")

</div>

Hi @stephenb, this worked, thank you!  
I want to ask though why does the filebeat.reference.yml file say

```auto
#In case you modify this pattern you must update setup.template.name and setup.template.pattern accordingly.
  output.elasticsearch.index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

```

```auto
#The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
#setup.template.name: "filebeat-%{[agent.version]}"

```

I originally assumed you had to change the setup.template name and pattern if you specified a specific index/data stream. But does it just mean that if you are using your own template for the specified index, then update accordingly?

Also, does migrating to a data stream change how Filebeat processors or ingest pipelines work? Upon closer inspection, there is 1 missing field which is `user_id` (`_source.user_id`). Instead, the value now appears under the field `user_id.#text`. The `user_id` field is created by processing the xml in Filebeat with decode\_xml and then using ingest pipeline processors to filter it. **Is there a way I can extract 'user\_id.#text` under user\_id instead?**  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/7/177f4e92440fa66d7edbef8458e2bc51dac4dcd8.png)

I suspect this issue is caused by migrating to a data stream and is an error with elasticsearch side, and might be an issue with my ingest pipeline. I am sure my Grok processor works, the rename processor might not be working though

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/d/0d5264defffec9490c12471d71e43de68c2ae771.png)

Below is my filebeat.yml config for parsers

```auto
parsers:
    - multiline:
        type: pattern
        pattern: '([0-9]+(\.[0-9]+)+)\s([0-9]+(:[0-9]+)+)'
        negate: true
        match: after
  processors:
    - dissect:
        tokenizer: "%{header}\n\n%{xmlmsg}"
        field: "message"
        target_prefix: ""
        trim_values: "left"
        trim_chars: " \t"
    - script:
        lang: javascript
        source: >
          function process(event) {
            var xmlmsg = event.Get("xmlmsg");
            event.Put("xmlmsg", xmlmsg.trim());
          }
    - decode_xml:
        field: xmlmsg
        target_field: xml_data
        overwrite_keys: true

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 23, 2024, 2:29pm UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/9 "2024-08-23T14:29:38Z")

</div>

> [@Tom\_N](#):
>
> ```auto
> #In case you modify this pattern you must update setup.template.name and setup.template.pattern accordingly.
> output.elasticsearch.index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
> 
> ```
> 
> ```auto
> #The template name and pattern has to be set in case the Elasticsearch index pattern is modified.
> #setup.template.name: "filebeat-%{[agent.version]}"
> 
> ```
> 
> I originally assumed you had to change the setup.template name and pattern if you specified a specific index/data stream. But does it just mean that if you are using your own template for the specified index, then update accordingly?

These all related to if you are using indices... and the various ways they can be setup.

> [@Tom\_N](#):
>
> Also, does migrating to a data stream change how Filebeat processors or ingest pipelines

No. But there are additional ways they can be specific in the index templates etc... Please ask a separate question if you are interested in this.

> [@Tom\_N](#):
>
> he `user_id` field is created by processing the xml in Filebeat with decode\_xml and then using ingest pipeline processors to filter it. **Is there a way I can extract 'user\_id.#text` under user\_id instead?**

You will need to rename it using a filebeat [rename](https://www.elastic.co/guide/en/beats/filebeat/current/rename-fields.html) processor or in an ingest pipeline. IMPORTANT it also depends if `user_id` is just a single fields or and object are there other `user_id.other_field`

And while we are at it the proper [user field name per ECS](https://www.elastic.co/guide/en/beats/filebeat/current/rename-fields.html) would be `user.id`

If you have additional questions, please open a specific topic.

---

<div class="post-metadata">

**Author:** ![Tom\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_n/32/136135_2.png) [@Tom\_N](https://discuss.elastic.co/u/Tom_N)\
**Post date:** [August 24, 2024, 3:06pm UTC](https://discuss.elastic.co/t/filebeat-error-when-setting-up-data-stream-with-working-index/365254/10 "2024-08-24T15:06:26Z")

</div>

Thank you for your reply and help Stephen!