# Implement filebeat with ingest pipelies

**URL:** <https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482>\
**Category:** Elasticsearch\
**Created:** [May 23, 2025, 8:58pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482 "2025-05-23T20:58:29Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 23, 2025, 8:58pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/1 "2025-05-23T20:58:29Z")

</div>

Hello,  
I have ELK instance which consists of elasticsearch, logstash and kibana.

I would like to implement filebeat with ingest pipelines - meaning filebeat sends logs with tags and ingest pipelines recognize it and create index.

Please correct me if I'm wrong and if such implementation will work.

I created filebeat with config:

```auto
    output:
      elasticsearch:
        enabled: true
        hosts: ["http://es1:9200"]
        timeout: 60
        #ssl.certificate_authorities: ["/etc/ssl/certs/logstash-cert-ca.tchaws.amadeus.net.pem"]
        #ssl.certificate: "/etc/ssl/certs/logstash-cert.tchaws.amadeus.net.pem"
        #ssl.key: "/etc/ssl/certs/logstash-key.tchaws.amadeus.net.key"      
    filebeat.inputs:
      - type: log
        tags: [dpkg]
        enabled: true
        paths:
          - /opt/dpkg.log

```

and filebeat has output defined to elasticsearch and sends logs with tags.

Inside elasticsearch I created such pipeline (with grok as preprocessor)

 ![grok](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a8b0c12535e5fc0a9c90e6c2724e5ab69a08b8e.png)

This is test grok pattern, so for all works I decided to use only greedydata.

But I don't know how can I create index from this pipeline.

In my understanding,  
I have filebeat which sends logs with tags, ingest pipeline has pipeline with preprocessor with grok for logs.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 23, 2025, 11:23pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/2 "2025-05-23T23:23:27Z")

</div>

Hi @dominbdg

Use the `pipeline` configuration

> [@dominbdg](#):
>
> ```auto
> output:
> elasticsearch:
> enabled: true
> hosts: ["http://es1:9200"]
> timeout: 60
> pipeline: <the_pipeline_name_you_created> 
> 
> ```

> **[Configure the Elasticsearch output | Elastic Documentation](https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#pipeline-option-es)**
>
> The Elasticsearch output sends events directly to Elasticsearch using the Elasticsearch HTTP API. Example configuration: When sending data to a secured...

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 24, 2025, 11:03am UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/3 "2025-05-24T11:03:44Z")

</div>

I implemented filebeat output as You said :

```auto
    output:
      elasticsearch:
        enabled: true
        hosts: ["http://es1:9200"]
        timeout: 60
        pipeline: "filebeat-1"
        #ssl.certificate_authorities: ["/etc/ssl/certs/logstash-cert-ca.tchaws.amadeus.net.pem"]
        #ssl.certificate: "/etc/ssl/certs/logstash-cert.tchaws.amadeus.net.pem"
        #ssl.key: "/etc/ssl/certs/logstash-key.tchaws.amadeus.net.key"
    filebeat.inputs:
      - type: log
        tags: [dpkg]
        enabled: true
        paths:
          - /opt/dpkg.log

```

but I still don't have this index created in elasticsearch.  
I checked connection and filebeat is connected to elasticsearch.

I still don't understand how using ingest pipelines I can create indexes.  
Maybe I'm not going to do at right way, because in my understanding,  
Filebeat has pipeline name, it's point to elasticsearch, and on ingest pipeline I have created grok.

In my environment in logstash, I have input (listening to filebeat), grok parrern and output.  
I cannot find anything to output (create index) in ingest pipelines

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 24, 2025, 2:18pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/4 "2025-05-24T14:18:12Z")

</div>

Ok lets back up a bit.

What version are you on?

How did you install filebeat?

> [@dominbdg](#):
>
> ingest pipelines I can create indexes

Ingest pipelines do not create indices... They are used to transform data before writing them to an index .. ingest pipeline live in elasticsearch

The default datastream (collection ofindices) from filebeat will be something like filebeat-8.17..4

> [@dominbdg](#):
>
> In my environment in logstash, I have input (listening to filebeat), grok parrern and output.

Ok well your filebeat is not configured to point to logstash it is pointing to Elasticsearch  
So that is another issue.  
The Ingest pipeline you showed above is not in Logstash that is in Elasticsearch

Ingest pipelines are in Elasticsearch  
Logstash pipelines are in Logstash  
The both transform data, but are 2 different architectural patterns.  
Why are you using Logstash, do you need to? It adds complexity if you are just getting started.

So you need to decide if you want

A) Filebeat -\> Elasticsearch (with ingest pipleine)

B) Filebeat -\> Logstash (logstash pipeline) -\> Elasticsearch

C) Filebeat -\> Logstash (passthrough) -\> Elasticsearch (with ingest pipleine)

If you are just getting started A is the simplest.

So what are you trying to do? and why?

Have you looked at the Filebeat quickstart?

> **[Filebeat quick start: installation and configuration | Elastic Documentation](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-installation-configuration)**
>
> This guide describes how to get started quickly with log collection. You’ll learn how to: install Filebeat on each system you want to monitor, specify...

There are commands to test the configurations as well.

Also once filebeat reads a file it will not read it again because it keeps track of what it ready so if you want to test and re-read the same file over again you will need to clean up the data registry

> **[Directory layout | Elastic Documentation](https://www.elastic.co/docs/reference/beats/filebeat/directory-layout)**
>
> The directory layout of an installation is as follows: You can change these settings by using CLI flags or setting path options in the configuration file...

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 24, 2025, 2:39pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/5 "2025-05-24T14:39:41Z")

</div>

I'm happy that accidentally founded that I have in datastream data from filebeat side.  
But I don't know how to name this data stream.  
I have "filebeat-8.11.1" which is version of my filebeat.

I founded that automatically is it created template.  
How can I name this data stream - probably from filebeat side but I couldn't find anything.  
In logstash I had indexes created weekly or monthly. I don't know if it is possible to use such settings here.

But basically I'm happy to have data stream from this implementation.

In my config of filebeat I have pipeline "filebeat-1" and I expected that this will be the same name of data stream.

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 24, 2025, 2:49pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/6 "2025-05-24T14:49:25Z")

</div>

ok sorry for that late - but my basic explanation.  
I'm not starting my adventure with ELK but I'm quite deeply skilled in ELK,  
but my current environment are with elastic+logstash+kibana.

I have concept of to get rid ofn logstash, and I wanted to go with elastic with ingest pipelines and kibana.

Someone before me installed such environment and probably this guy didn't have experience with ingest pipelines.

This is my first shot with ingest pipelines and I'm working to adapt it to my environment.  
For me pipeline in elastic is much more clear than in logstash.  
More about that logstash is much more complicated speaking about grok patterns and requires in my case a lot of resources.

Right now I'm learning about ingest pipelines but I see that looks much more better than logstash.

In my case logstash is used mostly with flat log files (not json files) but when I putted very simple grok like greedydata - ingest pipelines are also work with that kind of files.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 24, 2025, 3:38pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/7 "2025-05-24T15:38:47Z")

</div>

Okay in general though...

Ingest pipelines do not set the name of the index or data.... They are used to transform data.

I would read about data streams  
How frequently and how big the backing data indexes are are governed by the index life cycle management policy applied to the data stream

There is a default ILM policy for filebeat

Also, 8.11 is pretty old. You should consider upgrading at some point

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [May 24, 2025, 4:43pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/8 "2025-05-24T16:43:12Z")

</div>

Most likely your /opt/dpkg.log log has been read, recorded in the file registry `log.json` in the `/var/lib/filebeat/` directory.

How create index by using ingest pipelines?  
As you started, create the ingest pipeline, point in FB to ES, check data in Kibana.

1. Update filebeat.yml, add:

```auto
- type: filestream # log is obsolete 
  enabled: true
  tags: [dpkg]
  paths:
    - /opt/dpkg.log

setup.template:
  name: "template-test"
  pattern: "template-test*"
  enabled: true

output.elasticsearch:
  enabled: true
  hosts: ["http://es1:9200"]
  index: "test-%{[agent.version]}"# change how you call it
  timeout: 60
  pipeline: "name-test" # change how you call it

output.console: # useful to make sure FB read log, if you use it, disable output.elasticsearch
  pretty: true
  enabled: false

```

If you not set the index name, your data will go to filebeat-%{agentversion}.

1. Make the grok or any other processor, should look like this:

```auto
PUT _ingest/pipeline/name-test
{
  "description": "Test Pipeline",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{GREEDYDATA:msg}"], 
        "pattern_definitions": { "" }
      }
    }
  ]
}

```

Use Kibana UI if is easier. You can also set only 2-3 fields timestamp and greedy msg just to make sure the grok is parsing. Make the grok pattern outside IPipeline, can be easier.

1. Make a test index and insert test data

```auto
 POST /test/_doc/
{
    "message": "<your line from the log>"
}

```

1. Create an index template under Index Management, no need to specify fields, ES will do it for you, just name the template and pattern as you set in filebeat.yml.

2. Test filebeat.yml

```auto
filebeat.exe test config

```

1. Run it:  
`filebeat.exe -e`
2. If there is an error will be on screen and log. You can set [log level](https://www.elastic.co/docs/reference/beats/filebeat/configuration-logging) to debug if you wish, but enough is -e.
3. Create a data view in Kibana, should point on `test-*` index, or how you name it.
4. Check data in Discovery
5. Improve your grok to parse data in fields.

Remember, logs are your best friends, and RTFM.

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 24, 2025, 6:45pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/9 "2025-05-24T18:45:51Z")

</div>

Many thanks for that long reply.  
This time I have issue, meaning - filebeat is sending logs:

> {"log.level":"warn","@timestamp":"2025-05-24T18:33:58.262Z","log.logger":"elasticsearch","log.origin":{"function":"[github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3","file.name":"elasticsearch/client.go","file.line":175},"message":"Failed](http://github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3%22,%22file.name%22:%22elasticsearch/client.go%22,%22file.line%22:175%7D,%22message%22:%22Failed) to index 1600 events in last 10s: events were dropped! Look at the event log to view the event and cause.","service.name":"filebeat","ecs.version":"1.6.0"}

In the logs I founded only:

> "message":"Set settings.index.lifecycle.name in template to filebeat as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}

But the thing is that I have this configured:

```auto
{
  "index": {
    "lifecycle": {
      "name": "filebeat"
    },

```

I don't really know what is the issue with that.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 24, 2025, 9:40pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/10 "2025-05-24T21:40:48Z")

</div>

> [@dominbdg](#):
>
> [:"Failed](http://github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3%22,%22file.name%22:%22elasticsearch/client.go%22,%22file.line%22:175%7D,%22message%22:%22Failed) to index 1600 events in last 10s: events were dropped! Look at the event log to view the event and

You should have a directory `events` in the filebeat logs directory.. those ofs will show the specific error which is likely a mapping error.

> [@dominbdg](#):
>
> In the logs I founded only:
> 
> > "message":"Set settings.index.lifecycle.name in template to filebeat as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
> 
> But the thing is that I have this configured:
> 
> ```auto
> {
> "index": {
> "lifecycle": {
> "name": "filebeat"
> },
> 
> ```
> 
> I don't really know what is the issue with that.

That all looks fine to me

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [May 24, 2025, 11:21pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/11 "2025-05-24T23:21:39Z")

</div>

> [@dominbdg](#):
>
> But the thing is that I have this configured:
> 
> ```auto
> {
> "index": {
> "lifecycle": {
> "name": "filebeat"
> },
> 
> ```
> 
> I don't really know what is the issue with that.

If you don't want to use ILM, you can exclude on the root, above setup.template:

```auto

setup.ilm.enabled: false

setup.template:
...

```

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 25, 2025, 3:40pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/12 "2025-05-25T15:40:44Z")

</div>

Hello,  
Thanks for Your answer.  
I solved issue by delete template for it and recreate pipeline with other name.

I have a question because I don't understand that. On one time I have created index, and on another implementation of filebeat I have created datastream

I don't know why is that happening

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 25, 2025, 3:55pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/13 "2025-05-25T15:55:45Z")

</div>

> [@dominbdg](#):
>
> I have a question because I don't understand that. On one time I have created index, and on another implementation of filebeat I have created datastream

I'm not sure we're going to be able to give you a specific answer because it's not really clear what the exact steps you did each time.

This is why I suggest starting with defaults and getting the default behavior and then adjusting.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [May 25, 2025, 4:16pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/14 "2025-05-25T16:16:02Z")

</div>

As Stephen point, not clear feedback will not provide useful help.

> From Qantas airline repair logs:  
> Pilot: Aircraft handles funny.  
> Maintenance engineers: Aircraft warned to straighten up, fly right, and be serious.

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 25, 2025, 4:48pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/15 "2025-05-25T16:48:40Z")

</div>

ok I will try to be more clear.

I have filebeat config for dpkg log:

```auto
    output:
      elasticsearch:
        enabled: true
        hosts: ["http://es1:9200"]
        timeout: 60
        index: "dpkg-%{+YYYY.MM}"
        pipeline: "filebeat-dpkg"
        action: create
    #manage_template:
    setup.template:
      name: "filebeat-dpkg"
      pattern: "dpkg-*"
    enabled: true
    setup.ilm.enabled: false
    #index.lifecycle.name: "filebeat"
    filebeat.inputs:
      - type: filestream
        id: "filebeat-dpkg"
        tags: [dpkg]
        enabled: true
        paths:
          - /opt/dpkg.log

```

I created pipeline for it with below grok pattern:

`filebeat-dpkgProcessors [{ "grok": { "field": "message", "patterns": [ "%{GREEDYDATA}"], "tag": "dpkg" } } ]`

removed template - will be created automatically.  
and I have "dpkg-2025.05" index in datastream.

I have configuration for sddm.log:

```auto
    output:
      elasticsearch:
        enabled: true
        hosts: ["http://es1:9200"]
        timeout: 60
        index: "sddm-%{+YYYY.MM}"
        pipeline: "filebeat-sddm"
        action: create
    setup.template:
      name: "filebeat-sddm"
      pattern: "filebeat-sddm-*"
    enabled: true
    setup.ilm.enabled: false
    #index.lifecycle.name: "93-Day-Retention-Policy"
    filebeat.inputs:
      - type: filestream
        id: "filebeat-sddm"
        tags: [sddm]
        enabled: true
        paths:
          - /opt/sddm2.log

```

the same - removed indec template, created pipeline as 'filebeat-sddm' for it  
and I have index 'sddm-2025.05' not datastream.

I completely cannot figure why one log I have in index and another in datastream.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 25, 2025, 4:55pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/16 "2025-05-25T16:55:11Z")

</div>

First do not add the Date to the data stream name that is anti-pattern and does not make sense since the backing indices will have the the date.

> [@dominbdg](#):
>
> ` index: "sddm-%{+YYYY.MM}"`

should be something like

> [@dominbdg](#):
>
> ` index: "sddm-prod"`

I believe The reason for your issue is

```auto
    output:
      elasticsearch:
        enabled: true
        hosts: ["http://es1:9200"]
        timeout: 60
        index: "dpkg-%{+YYYY.MM}". <<<< THIS HERE 
        pipeline: "filebeat-dpkg"
        action: create
    #manage_template:
    setup.template:
      name: "filebeat-dpkg"
      pattern: "dpkg-*" <<<< MATCHES THIS PATTERN SO TEMPLATE IS APPLIED 

```

But here

```auto
    output:
      elasticsearch:
        enabled: true
        hosts: ["http://es1:9200"]
        timeout: 60
        index: "sddm-%{+YYYY.MM}" <<< THIS HERE 
        pipeline: "filebeat-sddm"
        action: create
    setup.template:
      name: "filebeat-sddm"
      pattern: "filebeat-sddm-*" <<<< DOES NOT MATCHES THIS PATTERN SO TEMPLATE IS ***NOT*** APPLIED AND SO A SIMPLE DEFAULT INDEX IS CREATED  
    enabled: true

```

Details 🙂

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 25, 2025, 9:17pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/17 "2025-05-25T21:17:04Z")

</div>

Many thanks for that  
Sometimes I'm blind on such details.

Ok, corrected my configuration:

```auto
    output:
      elasticsearch:
        hosts: ["http://es1:9200"]
        timeout: 60
        index: "dpkg"
        pipeline: "filebeat-dpkg-5"
        #action: create
        #data_stream: false
        #manage_template:
    setup.template:
      name: "filebeat-dpkg-5"
      pattern: "dpkg-*"
    enabled: true
    setup.ilm.enabled: false
        #index.lifecycle.name: "filebeat"
    filebeat.inputs:
      - type: filestream
        #id: "filebeat-dpkg-5"
        tags: [dpkg]
        enabled: true
        paths:
          - /opt/dpkg.log

```

I don't know why it is creating index to me not datastream.  
But funny thing when I will change it to something like "dpkg-prod" it is creating datastream.  
It is somewhere remembers this index dpkg but I don't know where.

I removed old pipeline for it, and removed filebeat from ILM because I'm not using it.  
I don't know where to look other to not have index but like the rest - datastreams

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 25, 2025, 10:10pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/18 "2025-05-25T22:10:03Z")

</div>

> [@dominbdg](#):
>
> `index: "dpkg"`. \<\< DATA STREAM NAME

> [@dominbdg](#):
>
> `pattern: "dpkg-*"` \<\<\<\< DOES NOT MATCH ABOVE EXPECTING -\*

The index name does not match the pattern....do not match

Try

`index: "dpkg-prod"`

The pattern is to match the data stream name not the backing indices... So your pattern has a `-*` but the data stream name you're setting it does not...

Yes it's a little confusing that the file beats still says index but it's really what you're writing to which in this case is a data stream

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 25, 2025, 10:20pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/19 "2025-05-25T22:20:41Z")

</div>

You might also want to look at this...

Probably the way I would do this to ingest into a logs data stream then add the pipeline

> [@Filebeat Elasticsearch output data stream?](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562/3):
>
> Example: Can be set per input filebeat.inputs: - type: filestream id: my-filestream-id enabled: true paths: - /var/log/\*.log fields\_under\_root: true fields: data\_stream.type: logs data\_stream.dataset: my.app data\_stream.namespace: prod event.dataset: my.app setup.ilm.enabled: false setup.template.enabled: false # setup.template.settings: # index.number\_of\_shards: 1 setup.kibana: output.elasticsearch: hosts: ["http://localhost:9200"] index: "%{[data\_stream.…

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [May 25, 2025, 10:40pm UTC](https://discuss.elastic.co/t/implement-filebeat-with-ingest-pipelies/378482/20 "2025-05-25T22:40:53Z")

</div>

I will try it - many thanks for that, indeed my filebeat.inputs is not completed as this example