# Filebeat -\> ElasticSearch for logrus running on k8s

**URL:** <https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 18, 2022, 10:32pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416 "2022-08-18T22:32:06Z")\
**Posts on this page:** 9\
**Page:** 2

<div class="post-metadata">

**Author:** ![Austin\_ES\_Questions](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/austin_es_questions/32/98119_2.png) [@Austin\_ES\_Questions](https://discuss.elastic.co/u/Austin_ES_Questions)\
**Post date:** [August 24, 2022, 4:18am UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/21 "2022-08-24T04:18:13Z")

</div>

Gotcha makes sense. Excitingly I am seeing promising results from trying your exact pipeline, so thank you for that!

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/b/0b40881770371da25a227fe1cd9aeec64bd9e45a.jpeg)

I will have to do some analysis of filebeat logs themselves to see where we are failing as I see a few `Cannot index event publisher.Event` for some of the more esoteric logs. We will then have to decide bewteen going live with `filebeat-*` patterns and no configured ILM policies or trying to get that all wrapped up first

Re:

> [@stephenb](#):
>
> You can do a most of this through the UI.. with the exception of creating the initial managed index see [here](https://www.elastic.co/guide/en/elasticsearch/reference/8.3/set-up-lifecycle-policy.html#create-initial-index)

part of why I'm asking where to set index configurations is to avoid doing it through the UI - we have a strong preference to use committed files for infrastructure (e.g., terraform), since we frequently spin-up / spin-down different environments, and being able to correctly provision an ES cluster with all settings configured is the "holy grail" for our CI/CD workflow.

Do you think it's fair to try and do all of your referenced ILM policy configuration through the elasticsearch resources at [Terraform Registry](https://registry.terraform.io/providers/elastic/elasticstack/latest/docs) ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 24, 2022, 4:19am UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/22 "2022-08-24T04:19:17Z")

</div>

if you used the filebeat settings your filebeat.yml could be this simple..

```auto
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log

setup.kibana:

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "discuss-mixed-container"

processors:
  - add_tags:
      tags: [monitor-logs]

```

if you set up all your templates and own ILM etc...

It could be this simple

```auto
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log

setup.kibana:

# Must have these if you want to set your own index name!!!
setup.template.enabled: false
setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "discuss-mixed-container"
  index: my-applogs

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 24, 2022, 4:22am UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/23 "2022-08-24T04:22:44Z")

</div>

> [@Austin\_ES\_Questions](#):
>
> part of why I'm asking where to set index configurations is to avoid doing it through the UI - we have a strong preference to use committed files for infrastructure (e.g., terraform), since we frequently spin-up / spin-down different environments, and being able to correctly provision an ES cluster with all settings configured is the "holy grail" for our CI/CD workflow.

Ahhh GOOD!! Yes my most succesful teams do this all through the API and CI/CD! Good to hear

> [@Austin\_ES\_Questions](#):
>
> Do you think it's fair to try and do all of your referenced ILM policy configuration through the elasticsearch resources at [Terraform Registry](https://registry.terraform.io/providers/elastic/elasticstack/latest/docs) ?

Yes I think so it is really Template, ILM and Initial Managed Index which is basically and empty index with a write alias ~~I think the example is even that~~

A write alias is the base pattern that filebeat always writes to say  
`log-app-frontend` that is what would be the index setting in filebeat

But it really points to

`log-app-frontend-2022.08.23-000001` for instance ...

then it rolls over filebeat keeps writing to `log-app-frontend` but the backing index is

`log-app-frontend-2022.08.24-000002`

That is how write alias and rollover work to give you ILM ...

> [@Austin\_ES\_Questions](#):
>
> I will have to do some analysis of filebeat logs themselves to see where we are failing as I see a few `Cannot index event publisher.Event` for some of the more esoteric logs. We will then have to decide bewteen going live with `filebeat-*` patterns and no configured ILM policies or trying to get that all wrapped up first

Yes you can just iterate to clean that all up... there is some more advanced patterns .. that you can do in pipeline if there are errors to just revert to the raw log line and pump them to a separate index or some.. but best thing is to just iterate... and get 95%+ of logs and then work from there.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 24, 2022, 4:34am UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/24 "2022-08-24T04:34:05Z")

</div>

> [@Austin\_ES\_Questions](#):
>
> `filebeat-*` patterns and no configured ILM policies or trying to get that all wrapped up first

Filebeat comes with and ILM policy... you can just edit to your purpose.. that is the easy part. I would do it through the UI.. the `GET` the policy afterwards so you can see what a wellformed ILM policy looks like..

Ohh cool that you got it working  
Add those App Tags to filter on...  
Then you can switch to seperate index name in a bit once you get it all going .. little change for your customer... depending on the volume...

_You might find the like it... hey I can see /search the app log and front end log in the same search... oooo they did not think of that! 🙂 .... i.e. the equivalent of `log-*` for every then just turn on / off tag filters.. not have to select different index patterns_

Ok stepping away... keep moving! Good Thread!

---

<div class="post-metadata">

**Author:** ![Austin\_ES\_Questions](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/austin_es_questions/32/98119_2.png) [@Austin\_ES\_Questions](https://discuss.elastic.co/u/Austin_ES_Questions)\
**Post date:** [August 29, 2022, 7:46pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/25 "2022-08-29T19:46:33Z")

</div>

Hi Stephen,

I'm returning to setting up indices. We prefer to completely define our configuration rather than try to rely on defaults.

My goal is to do whatever the recommended best practice is for managing different datasets with different retention windows, through indices, ILM, templates, aliases, and data streams

Towards that end I'm experimenting with defining index lifecycles and templates in terraform like so:

```auto
resource "elasticstack_elasticsearch_index_lifecycle" "hot_warm_delete_10_30" {
  name = "hot_warm_delete_10_30"

  hot {
    min_age = "1h"
    rollover {
      max_age = "1d"
      max_primary_shard_size = "30gb"
    }
  }

  warm {
    min_age = "10d"
    readonly {
      enabled = true
    }
    forcemerge {
      max_num_segments = 1
    }
    shrink {
      number_of_shards = 1
    }
  }

  delete {
    min_age = "30d"
    delete {
      delete_searchable_snapshot = true
    }
  }
}

resource "elasticstack_elasticsearch_index_template" "logs_app" {
  name = "logs_app"

  priority = 2022

  index_patterns = [
    "logs-app-*"
  ]

  template {
    settings = jsonencode({
      "lifecycle.name" = elasticstack_elasticsearch_index_lifecycle.hot_warm_delete_10_30.name
    })
  }
}

```

with the idea that I will be able to configure additional lifecycles like `hot_delete_10` and `hot_warm_cold_delete_10_30_90` to manage migration and retention of different data sources

However, I'm uncertain how to connect the data I'm writing from filebeat to these template configurations.

From the filebeat logs it looks like to set an index like

```auto
index: "logs-%{kubernetes.container.name}-%{kubernetes.labels.app_kubernetes_io/name}-%{+yyyy.MM.dd}"

```

(where the kubernetes fields are populated by processors.add\_kubernetes\_metadata)

I would also have to configure `setup.template.name` and `setup.template.pattern`, which are fields defined not per `filebeat.inputs` but generally for the filebeat configuration

I am hoping to avoid defining datastreams explicitly in terraform - I would like to infer them from kubernetes fields so that new services don't need to modify the elasticstack or filebeat configurations to start logging

Would you be able to describe the right way to do this configuration? It's a little unclear to me how to completely wire sending data from filebeat to different datastreams conditionally on the source/fields of data

Thank you!  
Austin

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 29, 2022, 8:28pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/26 "2022-08-29T20:28:16Z")

</div>

You are asking a lot @Austin_ES_Questions 🙂

This is how my mature customer work so it is a great approach... You will just need to understand the concepts and their relationships. You will do 90% via your automation and very little will actually go in filebeat.yml

So a couple things before we get started.

This is important

1. You mention data streams a few times are you staying with 7.17.X or are you moving to 8.x. Data Streams are the default in 8.X so if you want to head there / learn that I would do that now...

**What version are you going to stick with and do you want indices or datastreams?** they are not the same and it seems you are kinda using them interchangebly they are not...

**Take a read [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html) and let me know which / what you want to do**... because it is 2 different approaches... the future is datastreams

1. [quote="Austin\_ES\_Questions, post:25, topic:312416"]  
I would also have to configure `setup.template.name` and `setup.template.pattern`, which are fields defined not per `filebeat.inputs` but generally for the filebeat configuration  
[/quote]

Since you are going to do all these wonderfull things via automation you will not be setting anything up via filebeat 🙂 You filebeat.yml will get much simpler.

> [@Austin\_ES\_Questions](#):
>
> From the filebeat logs it looks like to set an index like
> 
> ```auto
> index: "logs-%{kubernetes.container.name}-%{kubernetes.labels.app_kubernetes_io/name-%{+yyyy.MM.dd}"
> 
> ```

no that is not correct... I will explain later... if you look above I spoke about write aliases or data streams...

---

<div class="post-metadata">

**Author:** ![Austin\_ES\_Questions](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/austin_es_questions/32/98119_2.png) [@Austin\_ES\_Questions](https://discuss.elastic.co/u/Austin_ES_Questions)\
**Post date:** [August 29, 2022, 8:37pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/27 "2022-08-29T20:37:54Z")

</div>

> [@stephenb](#):
>
> 1. You mention data streams a few times are you staying with 7.17.X or are you moving to 8.x. Data Streams are the default in 8.X so if you want to head there / learn that I would do that now...

We are using the helm charts provided by elastic: [https://artifacthub.io/packages/helm/elastic/elasticsearch](https://artifacthub.io/packages/helm/elastic/elasticsearch)

and the elastic terraform providers:  
[https://registry.terraform.io/namespaces/elastic](https://registry.terraform.io/namespaces/elastic)

So our elasticsearch cluster is version 8.3.3 but filebeat is 7.17.3

We do not have a preference for indices/aliases/ILM or datastreams - whatever the officially supported solution is for deploying Elasticsearch + Filebeat from Elastic on kubernetes is

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 29, 2022, 9:41pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/28 "2022-08-29T21:41:32Z")

</div>

Assume you will move to filebeat 8.X when Helm available?

I am **not** a helms person so I can not help with that.

I would probably pick a datastreams as they are the path forward with timeseries data.

So you would follow these steps on [this page](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-a-data-stream.html) and [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/use-a-data-stream.html)

The good thing is if you use component template and your logs are mostly the same then just the index template will be different for each... so most everything will be reusable....

Start with one of your data streams like `log-app-frontend-sdk` and go through the steps... when you have all that then the filebeat stuff should be is easy...

you will disable template setting and just set the data stream name as the `index` field in the elasticsearch output.

Ohhh maybe I am still missing somethings major here like this is all running in K8s and you magically want to write to different datastreams.... so yes you will need to figure out what your datastream name will be based on the meta data ...

Not sure exactly what this will look like but it will need to match your datastream

`logs-%{kubernetes.container.name}-%{kubernetes.labels.app_kubernetes_io/name}`

```auto
setup.template.enabled: false
setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: "discuss-mixed-container"
  index: `logs-%{kubernetes.container.name}-%{kubernetes.labels.app_kubernetes_io/name}`

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 26, 2022, 11:42pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416/29 "2022-09-26T23:42:11Z")

</div>

Closed on September 26, 2022, 11:42pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/filebeat-elasticsearch-for-logrus-running-on-k8s/312416.md?page=1)