Filebeat writing to its own index

Elastic Stack Beats
1 
    ###################### Filebeat Configuration #########################

# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat inputs =============================

filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
      - /var/log/myindex-app/*.log

# matching on this type 2022-07-20 10:56:29,393
  multiline:
    pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
    negate: true
    match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================
setup.template:
  #name: "myindex-%{[agent.version]}"
  #pattern: "myindex-%{[agent.version]}-*"
  overwrite: true
  settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false



#==========================  Modules configuration =============================
filebeat.modules:
#-------------------------------- Nginx Module --------------------------------
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

    # Input configuration (advanced). Any input configuration option
    # can be added under this section.
    #input:

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]

    # Input configuration (advanced). Any input configuration option
    # can be added under this section.
    #input:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  #ingress_controller:
  #  enabled: false
  #
  #  # Set custom paths for the log files. If left empty,
  #  # Filebeat will choose the paths depending on your OS.
  #  #var.paths:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.  
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["${ELASTIC_URL}"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Certificate for SSL client authentication

  # Client Certificate Key
  
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  
  # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
  index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"

I am trying to get my logs to go into their own index, which then uses ILM. I have got ILM to set up correctly, but cannot get filebeat to write to the index that I have tried to define.

I have tried so many varieties of config but not yet found one that works. On the odd occasion I can an error, but the majority of the time I am not getting errors just the logging stops appearing in kibana.

Can someone please explain what is wrong with my config. I am two weeks into this, so am really losing the will...

Thanks

2

You need to write to the ILM alias, whatever you have set that as in the policy.

3

And one index needs to have is_write_alias true.

4

Have I set it? If not how would I set it?

5

How would this be added into the config?

6

Also may not be relevant but I am getting two ILM policies created each time, one lower case the other upper case. All the config in that area is lower case.

7 

    ###################### Filebeat Configuration #########################

    # You can find the full configuration reference here:
    # https://www.elastic.co/guide/en/beats/filebeat/index.html

    #=========================== Filebeat inputs =============================

    filebeat.inputs:
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.

    - type: log

      # Change to true to enable this input configuration.
      enabled: true

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
          - /var/log/myindex-app/*.log
   
    # matching on this type 2022-07-20 10:56:29,393
      multiline:
        pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
        negate: true
        match: after

    #============================= Filebeat modules ===============================

    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml

      # Set to true to enable config reloading
      reload.enabled: false

      # Period on which files under path should be checked for changes
      #reload.period: 10s

    #==================== Elasticsearch template setting ==========================
    setup.template:
      name: "myindex-%{[agent.version]}"
      pattern: "myindex-%{[agent.version]}-*"
      alias: "myindex"
      overwrite: true
      settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false
    
    

    #==========================  Modules configuration =============================
    filebeat.modules:
    #-------------------------------- Nginx Module --------------------------------
    - module: nginx
      # Access logs
      access:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/access.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Error logs
      error:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/error.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
      #ingress_controller:
      #  enabled: false
      #
      #  # Set custom paths for the log files. If left empty,
      #  # Filebeat will choose the paths depending on your OS.
      #  #var.paths:

    #================================ Outputs =====================================

    # Configure what output to use when sending the data collected by the beat.  
    # ---------------------------- Elasticsearch Output ----------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["${ELASTIC_URL}"]

      # Protocol - either `http` (default) or `https`.
      protocol: "https"

      # Certificate for SSL client authentication

      # Client Certificate Key
      
      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: ${ELASTIC_USERNAME}
      password: ${ELASTIC_PASSWORD}
      
      # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
      indices:
      #index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
      index: "myindex": {
      "is_write_index": true
                   }
    

    setup.ilm:
      enabled: true
      policy_name: "myindex"
      overwrite: true
      rollover_alias: "myindex-%{[agent.version]}"
      pattern: "{now/d}-0000001"
      policy_file: "/usr/share/filebeat/config/myindex.policy.json"
      

    #================================ Processors =====================================

    # Configure processors to enhance or manipulate events generated by the beat.

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

This is my current config, which still doesn't work, and causes two ilm policies to be created.

8

so logs are being written, but they are currently not getting into elastic, so getting lost somewhere within filebeat and elastic.

9

What is the content of

What version of beats & elastic?

This is a working config for filebeat 8.x setting up it's template and index:

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
setup.template.name: "filebeat-%{[beat.version]}-elastic-internal"
setup.template.pattern: "filebeat-%{[beat.version]}-elastic-internal*"

setup.ilm.enabled: true

setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-elastic-internal"

setup.ilm.policy_name: "roll_daily_delete_30"

setup.kibana:
  host: "https://xxxxxx:5601"

output.elasticsearch:
  hosts: ["https://xxxxxxxx:9200"]
  index: "filebeat-%{[agent.version]}-elastic-internal"
  username: "xxxxx"
  password: "yyyyy"
  ssl.certificate_authorities: ["/etc/filebeat/certs/https_interm.cer"]
  ssl.certificate: "/etc/filebeat/certs/https_cert.cer"
  ssl.key: "/etc/filebeat/certs/https_stack.key"
10

I will try and replicate, although the two look pretty much the same.

In a previous post there was a mention of an alias being required. Yours doesn't have that. Also is_write_index, which again is not in yours.

11

the ilm policy file is the setup for the ilm policy. This part of the config seems to be working, although an upper case version keeps appearing. on the upper case one - I changed the config and only the lower case one is changing, so the upper case seems to be appearing for no apparent reason.

12

That is an attribute of an index, not a filebeat config option.

13 
 ###################### Filebeat Configuration #########################

    # You can find the full configuration reference here:
    # https://www.elastic.co/guide/en/beats/filebeat/index.html

    #=========================== Filebeat inputs =============================

    filebeat.inputs:
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.

    - type: log

      # Change to true to enable this input configuration.
      enabled: true

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
          - /var/log/myindex-app/*.log
   
    # matching on this type 2022-07-20 10:56:29,393
      multiline:
        pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
        negate: true
        match: after

    #============================= Filebeat modules ===============================

    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml

      # Set to true to enable config reloading
      reload.enabled: false

      # Period on which files under path should be checked for changes
      #reload.period: 10s

    #==================== Elasticsearch template setting ==========================
    setup.template:
      name: "myindex-%{[agent.version]}"
      pattern: "myindex-%{[agent.version]}-*"
      alias: "myindex"
      overwrite: true
      settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false
    
    

    #==========================  Modules configuration =============================
    filebeat.modules:
    #-------------------------------- Nginx Module --------------------------------
    - module: nginx
      # Access logs
      access:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/access.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Error logs
      error:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/error.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
      #ingress_controller:
      #  enabled: false
      #
      #  # Set custom paths for the log files. If left empty,
      #  # Filebeat will choose the paths depending on your OS.
      #  #var.paths:

    #================================ Outputs =====================================

    # Configure what output to use when sending the data collected by the beat.  
    # ---------------------------- Elasticsearch Output ----------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["${ELASTIC_URL}"]

      # Protocol - either `http` (default) or `https`.
      protocol: "https"

      # Certificate for SSL client authentication

      # Client Certificate Key
      
      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: ${ELASTIC_USERNAME}
      password: ${ELASTIC_PASSWORD}
      
      # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
      #indices:
      #index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
      index: "myindex-%{[agent.version]}"
      #: {
      #"is_write_index": true
      #             }
    

    setup.ilm:
      enabled: true
      policy_name: "myindex"
      overwrite: true
      rollover_alias: "myindex-%{[agent.version]}"
      pattern: "{now/d}-0000001"
      policy_file: "/usr/share/filebeat/config/myindex.policy.json"
      

    #================================ Processors =====================================

    # Configure processors to enhance or manipulate events generated by the beat.

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

Above is what I think matches your example, but with my specific index name. This still however is not working.

What I would like to know is what filebeat is actually doing such that I can possibly understand what is wrong.

Are you able to advise what is wrong with my config by providing suitable changes that I can replicate?

thanks

14

What do I put in the config file to implement the alias?

15

I am going to have to log off for the weekend, but anything that you can suggest is gratefully received as I am at my wits end having spent 2 solid weeks not making any progress.

16

When something like this happens, remove all your customization, delete all templates, indices, ILM policies, everything. Start over letting filebeat use it's default config, default names, default everything. When that works, change one thing at a time working toward your custom config.

1 Like
17

Effectively been doing that for the last 2 weeks! I have tried so many combinations just not found one that actually works.

I seem to have got the ilm piece sorted, so not able to get filebeat to write to myindex, but also not able to find any logs anywhere to attempt to understand what is going wrong.

18

I do not think you can have this dynamic as agent.version is an event field that is not available on setup.

This must be the rollover alias and can not be dynamic.

19

the standard filebeat alias is filebeat-7.0.1 which suggests it can be dynamic.

20

Which version of the stack are you using? If it is version 7.0.1, it is very old.

I would recommend always stating which version you are using as the answer can vary a lot based on this.