# Filebeat writing to its own index

**URL:** <https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 28, 2022, 9:09am UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842 "2022-07-28T09:09:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 28, 2022, 9:09am UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/1 "2022-07-28T09:09:55Z")

</div>

```auto
    ###################### Filebeat Configuration #########################

# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat inputs =============================

filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
      - /var/log/myindex-app/*.log

# matching on this type 2022-07-20 10:56:29,393
  multiline:
    pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
    negate: true
    match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================
setup.template:
  #name: "myindex-%{[agent.version]}"
  #pattern: "myindex-%{[agent.version]}-*"
  overwrite: true
  settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

#========================== Modules configuration =============================
filebeat.modules:
#-------------------------------- Nginx Module --------------------------------
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

    # Input configuration (advanced). Any input configuration option
    # can be added under this section.
    #input:

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]

    # Input configuration (advanced). Any input configuration option
    # can be added under this section.
    #input:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  #ingress_controller:
  # enabled: false
  #
  # # Set custom paths for the log files. If left empty,
  # # Filebeat will choose the paths depending on your OS.
  # #var.paths:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.  
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["${ELASTIC_URL}"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Certificate for SSL client authentication

  # Client Certificate Key
  
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  
  # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
  index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"

```

I am trying to get my logs to go into their own index, which then uses ILM. I have got ILM to set up correctly, but cannot get filebeat to write to the index that I have tried to define.

I have tried so many varieties of config but not yet found one that works. On the odd occasion I can an error, but the majority of the time I am not getting errors just the logging stops appearing in kibana.

Can someone please explain what is wrong with my config. I am two weeks into this, so am really losing the will...

Thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 28, 2022, 11:21pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/2 "2022-07-28T23:21:52Z")

</div>

> [@tractor\_boy](#):
>
> ```auto
> index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
> 
> ```

You need to write to the ILM alias, whatever you have set that as in the policy.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [July 29, 2022, 1:30am UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/3 "2022-07-29T01:30:35Z")

</div>

And one index needs to have is\_write\_alias true.

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 10:08am UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/4 "2022-07-29T10:08:36Z")

</div>

Have I set it? If not how would I set it?

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 10:12am UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/5 "2022-07-29T10:12:44Z")

</div>

How would this be added into the config?

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 12:03pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/6 "2022-07-29T12:03:09Z")

</div>

Also may not be relevant but I am getting two ILM policies created each time, one lower case the other upper case. All the config in that area is lower case.

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 12:18pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/7 "2022-07-29T12:18:49Z")

</div>

```auto

    ###################### Filebeat Configuration #########################

    # You can find the full configuration reference here:
    # https://www.elastic.co/guide/en/beats/filebeat/index.html

    #=========================== Filebeat inputs =============================

    filebeat.inputs:
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.

    - type: log

      # Change to true to enable this input configuration.
      enabled: true

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
          - /var/log/myindex-app/*.log
   
    # matching on this type 2022-07-20 10:56:29,393
      multiline:
        pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
        negate: true
        match: after

    #============================= Filebeat modules ===============================

    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml

      # Set to true to enable config reloading
      reload.enabled: false

      # Period on which files under path should be checked for changes
      #reload.period: 10s

    #==================== Elasticsearch template setting ==========================
    setup.template:
      name: "myindex-%{[agent.version]}"
      pattern: "myindex-%{[agent.version]}-*"
      alias: "myindex"
      overwrite: true
      settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false
    
    

    #========================== Modules configuration =============================
    filebeat.modules:
    #-------------------------------- Nginx Module --------------------------------
    - module: nginx
      # Access logs
      access:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/access.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Error logs
      error:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/error.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
      #ingress_controller:
      # enabled: false
      #
      # # Set custom paths for the log files. If left empty,
      # # Filebeat will choose the paths depending on your OS.
      # #var.paths:

    #================================ Outputs =====================================

    # Configure what output to use when sending the data collected by the beat.  
    # ---------------------------- Elasticsearch Output ----------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["${ELASTIC_URL}"]

      # Protocol - either `http` (default) or `https`.
      protocol: "https"

      # Certificate for SSL client authentication

      # Client Certificate Key
      
      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: ${ELASTIC_USERNAME}
      password: ${ELASTIC_PASSWORD}
      
      # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
      indices:
      #index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
      index: "myindex": {
      "is_write_index": true
                   }
    

    setup.ilm:
      enabled: true
      policy_name: "myindex"
      overwrite: true
      rollover_alias: "myindex-%{[agent.version]}"
      pattern: "{now/d}-0000001"
      policy_file: "/usr/share/filebeat/config/myindex.policy.json"
      

    #================================ Processors =====================================

    # Configure processors to enhance or manipulate events generated by the beat.

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

```

This is my current config, which still doesn't work, and causes two ilm policies to be created.

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 12:31pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/8 "2022-07-29T12:31:06Z")

</div>

so logs are being written, but they are currently not getting into elastic, so getting lost somewhere within filebeat and elastic.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [July 29, 2022, 4:11pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/9 "2022-07-29T16:11:43Z")

</div>

What is the content of

> [@tractor\_boy](#):
>
> `/usr/share/filebeat/config/myindex.policy.json`

What version of beats & elastic?

This is a working config for filebeat 8.x setting up it's template and index:

```auto
filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
setup.template.name: "filebeat-%{[beat.version]}-elastic-internal"
setup.template.pattern: "filebeat-%{[beat.version]}-elastic-internal*"

setup.ilm.enabled: true

setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-elastic-internal"

setup.ilm.policy_name: "roll_daily_delete_30"

setup.kibana:
  host: "https://xxxxxx:5601"

output.elasticsearch:
  hosts: ["https://xxxxxxxx:9200"]
  index: "filebeat-%{[agent.version]}-elastic-internal"
  username: "xxxxx"
  password: "yyyyy"
  ssl.certificate_authorities: ["/etc/filebeat/certs/https_interm.cer"]
  ssl.certificate: "/etc/filebeat/certs/https_cert.cer"
  ssl.key: "/etc/filebeat/certs/https_stack.key"

```

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 4:21pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/10 "2022-07-29T16:21:02Z")

</div>

I will try and replicate, although the two look pretty much the same.

In a previous post there was a mention of an alias being required. Yours doesn't have that. Also is\_write\_index, which again is not in yours.

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 4:22pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/11 "2022-07-29T16:22:50Z")

</div>

the ilm policy file is the setup for the ilm policy. This part of the config seems to be working, although an upper case version keeps appearing. on the upper case one - I changed the config and only the lower case one is changing, so the upper case seems to be appearing for no apparent reason.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [July 29, 2022, 4:43pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/12 "2022-07-29T16:43:17Z")

</div>

That is an attribute of an index, not a filebeat config option.

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 4:43pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/13 "2022-07-29T16:43:44Z")

</div>

```auto
 ###################### Filebeat Configuration #########################

    # You can find the full configuration reference here:
    # https://www.elastic.co/guide/en/beats/filebeat/index.html

    #=========================== Filebeat inputs =============================

    filebeat.inputs:
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.

    - type: log

      # Change to true to enable this input configuration.
      enabled: true

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
          - /var/log/myindex-app/*.log
   
    # matching on this type 2022-07-20 10:56:29,393
      multiline:
        pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
        negate: true
        match: after

    #============================= Filebeat modules ===============================

    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml

      # Set to true to enable config reloading
      reload.enabled: false

      # Period on which files under path should be checked for changes
      #reload.period: 10s

    #==================== Elasticsearch template setting ==========================
    setup.template:
      name: "myindex-%{[agent.version]}"
      pattern: "myindex-%{[agent.version]}-*"
      alias: "myindex"
      overwrite: true
      settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false
    
    

    #========================== Modules configuration =============================
    filebeat.modules:
    #-------------------------------- Nginx Module --------------------------------
    - module: nginx
      # Access logs
      access:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/access.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Error logs
      error:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/error.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
      #ingress_controller:
      # enabled: false
      #
      # # Set custom paths for the log files. If left empty,
      # # Filebeat will choose the paths depending on your OS.
      # #var.paths:

    #================================ Outputs =====================================

    # Configure what output to use when sending the data collected by the beat.  
    # ---------------------------- Elasticsearch Output ----------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["${ELASTIC_URL}"]

      # Protocol - either `http` (default) or `https`.
      protocol: "https"

      # Certificate for SSL client authentication

      # Client Certificate Key
      
      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: ${ELASTIC_USERNAME}
      password: ${ELASTIC_PASSWORD}
      
      # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
      #indices:
      #index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
      index: "myindex-%{[agent.version]}"
      #: {
      #"is_write_index": true
      # }
    

    setup.ilm:
      enabled: true
      policy_name: "myindex"
      overwrite: true
      rollover_alias: "myindex-%{[agent.version]}"
      pattern: "{now/d}-0000001"
      policy_file: "/usr/share/filebeat/config/myindex.policy.json"
      

    #================================ Processors =====================================

    # Configure processors to enhance or manipulate events generated by the beat.

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

```

Above is what I think matches your example, but with my specific index name. This still however is not working.

What I would like to know is what filebeat is actually doing such that I can possibly understand what is wrong.

Are you able to advise what is wrong with my config by providing suitable changes that I can replicate?

thanks

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 4:44pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/14 "2022-07-29T16:44:42Z")

</div>

What do I put in the config file to implement the alias?

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [July 29, 2022, 5:10pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/15 "2022-07-29T17:10:52Z")

</div>

I am going to have to log off for the weekend, but anything that you can suggest is gratefully received as I am at my wits end having spent 2 solid weeks not making any progress.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [July 29, 2022, 5:47pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/16 "2022-07-29T17:47:56Z")

</div>

When something like this happens, remove all your customization, delete all templates, indices, ILM policies, everything. Start over letting filebeat use it's default config, default names, default everything. When that works, change one thing at a time working toward your custom config.

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [August 1, 2022, 7:41am UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/17 "2022-08-01T07:41:58Z")

</div>

Effectively been doing that for the last 2 weeks! I have tried so many combinations just not found one that actually works.

I seem to have got the ilm piece sorted, so not able to get filebeat to write to myindex, but also not able to find any logs anywhere to attempt to understand what is going wrong.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 1, 2022, 8:46am UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/18 "2022-08-01T08:46:03Z")

</div>

> [@tractor\_boy](#):
>
> `rollover_alias: "myindex-%{[agent.version]}"`

I do not think you can have this dynamic as `agent.version` is an event field that is not available on setup.

> [@tractor\_boy](#):
>
> `index: "myindex-%{[agent.version]}"`

This must be the rollover alias and can not be dynamic.

---

<div class="post-metadata">

**Author:** ![tractor\_boy](https://avatars.discourse-cdn.com/v4/letter/t/278dde/32.png) [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\
**Post date:** [August 1, 2022, 3:09pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/19 "2022-08-01T15:09:33Z")

</div>

the standard filebeat alias is filebeat-7.0.1 which suggests it can be dynamic.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 1, 2022, 3:19pm UTC](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/20 "2022-08-01T15:19:13Z")

</div>

> [@tractor\_boy](#):
>
> filebeat-7.0.1

Which version of the stack are you using? If it is version 7.0.1, it is very old.

I would recommend always stating which version you are using as the answer can vary a lot based on this.

[Next page](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842.md?page=2)