Filebeat writing to its own index

tractor_boy · July 28, 2022, 9:09am

    ###################### Filebeat Configuration #########################

# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat inputs =============================

filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
      - /var/log/myindex-app/*.log

# matching on this type 2022-07-20 10:56:29,393
  multiline:
    pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
    negate: true
    match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================
setup.template:
  #name: "myindex-%{[agent.version]}"
  #pattern: "myindex-%{[agent.version]}-*"
  overwrite: true
  settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false



#==========================  Modules configuration =============================
filebeat.modules:
#-------------------------------- Nginx Module --------------------------------
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/access.log"]

    # Input configuration (advanced). Any input configuration option
    # can be added under this section.
    #input:

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/error.log"]

    # Input configuration (advanced). Any input configuration option
    # can be added under this section.
    #input:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  #ingress_controller:
  #  enabled: false
  #
  #  # Set custom paths for the log files. If left empty,
  #  # Filebeat will choose the paths depending on your OS.
  #  #var.paths:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.  
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["${ELASTIC_URL}"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Certificate for SSL client authentication

  # Client Certificate Key
  
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  
  # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
  index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"

I am trying to get my logs to go into their own index, which then uses ILM. I have got ILM to set up correctly, but cannot get filebeat to write to the index that I have tried to define.

I have tried so many varieties of config but not yet found one that works. On the odd occasion I can an error, but the majority of the time I am not getting errors just the logging stops appearing in kibana.

Can someone please explain what is wrong with my config. I am two weeks into this, so am really losing the will...

Thanks

warkolm · July 28, 2022, 11:21pm

You need to write to the ILM alias, whatever you have set that as in the policy.

rugenl · July 29, 2022, 1:30am

And one index needs to have is_write_alias true.

tractor_boy · July 29, 2022, 10:08am

Have I set it? If not how would I set it?

tractor_boy · July 29, 2022, 10:12am

How would this be added into the config?

tractor_boy · July 29, 2022, 12:03pm

Also may not be relevant but I am getting two ILM policies created each time, one lower case the other upper case. All the config in that area is lower case.

tractor_boy · July 29, 2022, 12:18pm


    ###################### Filebeat Configuration #########################

    # You can find the full configuration reference here:
    # https://www.elastic.co/guide/en/beats/filebeat/index.html

    #=========================== Filebeat inputs =============================

    filebeat.inputs:
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.

    - type: log

      # Change to true to enable this input configuration.
      enabled: true

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
          - /var/log/myindex-app/*.log
   
    # matching on this type 2022-07-20 10:56:29,393
      multiline:
        pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
        negate: true
        match: after

    #============================= Filebeat modules ===============================

    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml

      # Set to true to enable config reloading
      reload.enabled: false

      # Period on which files under path should be checked for changes
      #reload.period: 10s

    #==================== Elasticsearch template setting ==========================
    setup.template:
      name: "myindex-%{[agent.version]}"
      pattern: "myindex-%{[agent.version]}-*"
      alias: "myindex"
      overwrite: true
      settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false
    
    

    #==========================  Modules configuration =============================
    filebeat.modules:
    #-------------------------------- Nginx Module --------------------------------
    - module: nginx
      # Access logs
      access:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/access.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Error logs
      error:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/error.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
      #ingress_controller:
      #  enabled: false
      #
      #  # Set custom paths for the log files. If left empty,
      #  # Filebeat will choose the paths depending on your OS.
      #  #var.paths:

    #================================ Outputs =====================================

    # Configure what output to use when sending the data collected by the beat.  
    # ---------------------------- Elasticsearch Output ----------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["${ELASTIC_URL}"]

      # Protocol - either `http` (default) or `https`.
      protocol: "https"

      # Certificate for SSL client authentication

      # Client Certificate Key
      
      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: ${ELASTIC_USERNAME}
      password: ${ELASTIC_PASSWORD}
      
      # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
      indices:
      #index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
      index: "myindex": {
      "is_write_index": true
                   }
    

    setup.ilm:
      enabled: true
      policy_name: "myindex"
      overwrite: true
      rollover_alias: "myindex-%{[agent.version]}"
      pattern: "{now/d}-0000001"
      policy_file: "/usr/share/filebeat/config/myindex.policy.json"
      

    #================================ Processors =====================================

    # Configure processors to enhance or manipulate events generated by the beat.

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

This is my current config, which still doesn't work, and causes two ilm policies to be created.

tractor_boy · July 29, 2022, 12:31pm

so logs are being written, but they are currently not getting into elastic, so getting lost somewhere within filebeat and elastic.

rugenl · July 29, 2022, 4:11pm

What is the content of

What version of beats & elastic?

This is a working config for filebeat 8.x setting up it's template and index:

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
setup.template.name: "filebeat-%{[beat.version]}-elastic-internal"
setup.template.pattern: "filebeat-%{[beat.version]}-elastic-internal*"

setup.ilm.enabled: true

setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-elastic-internal"

setup.ilm.policy_name: "roll_daily_delete_30"

setup.kibana:
  host: "https://xxxxxx:5601"

output.elasticsearch:
  hosts: ["https://xxxxxxxx:9200"]
  index: "filebeat-%{[agent.version]}-elastic-internal"
  username: "xxxxx"
  password: "yyyyy"
  ssl.certificate_authorities: ["/etc/filebeat/certs/https_interm.cer"]
  ssl.certificate: "/etc/filebeat/certs/https_cert.cer"
  ssl.key: "/etc/filebeat/certs/https_stack.key"

tractor_boy · July 29, 2022, 4:21pm

I will try and replicate, although the two look pretty much the same.

In a previous post there was a mention of an alias being required. Yours doesn't have that. Also is_write_index, which again is not in yours.

tractor_boy · July 29, 2022, 4:22pm

the ilm policy file is the setup for the ilm policy. This part of the config seems to be working, although an upper case version keeps appearing. on the upper case one - I changed the config and only the lower case one is changing, so the upper case seems to be appearing for no apparent reason.

rugenl · July 29, 2022, 4:43pm

That is an attribute of an index, not a filebeat config option.

tractor_boy · July 29, 2022, 4:43pm

 ###################### Filebeat Configuration #########################

    # You can find the full configuration reference here:
    # https://www.elastic.co/guide/en/beats/filebeat/index.html

    #=========================== Filebeat inputs =============================

    filebeat.inputs:
    # Each - is an input. Most options can be set at the input level, so
    # you can use different inputs for various configurations.
    # Below are the input specific configurations.

    - type: log

      # Change to true to enable this input configuration.
      enabled: true

      # Paths that should be crawled and fetched. Glob based paths.
      paths:
          - /var/log/myindex-app/*.log
   
    # matching on this type 2022-07-20 10:56:29,393
      multiline:
        pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}'
        negate: true
        match: after

    #============================= Filebeat modules ===============================

    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml

      # Set to true to enable config reloading
      reload.enabled: false

      # Period on which files under path should be checked for changes
      #reload.period: 10s

    #==================== Elasticsearch template setting ==========================
    setup.template:
      name: "myindex-%{[agent.version]}"
      pattern: "myindex-%{[agent.version]}-*"
      alias: "myindex"
      overwrite: true
      settings:
      index.number_of_shards: 1
      #index.codec: best_compression
      #_source.enabled: false
    
    

    #==========================  Modules configuration =============================
    filebeat.modules:
    #-------------------------------- Nginx Module --------------------------------
    - module: nginx
      # Access logs
      access:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/access.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Error logs
      error:
        enabled: true

        # Set custom paths for the log files. If left empty,
        # Filebeat will choose the paths depending on your OS.
        var.paths: ["/var/log/nginx/error.log"]

        # Input configuration (advanced). Any input configuration option
        # can be added under this section.
        #input:

      # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
      #ingress_controller:
      #  enabled: false
      #
      #  # Set custom paths for the log files. If left empty,
      #  # Filebeat will choose the paths depending on your OS.
      #  #var.paths:

    #================================ Outputs =====================================

    # Configure what output to use when sending the data collected by the beat.  
    # ---------------------------- Elasticsearch Output ----------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["${ELASTIC_URL}"]

      # Protocol - either `http` (default) or `https`.
      protocol: "https"

      # Certificate for SSL client authentication

      # Client Certificate Key
      
      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: ${ELASTIC_USERNAME}
      password: ${ELASTIC_PASSWORD}
      
      # %{[fileset.module]}-%{[fileset.name]} to be added as an option - TBC
      #indices:
      #index: "myindex-%{[agent.version]}-%{+yyyy.MM.dd}"
      index: "myindex-%{[agent.version]}"
      #: {
      #"is_write_index": true
      #             }
    

    setup.ilm:
      enabled: true
      policy_name: "myindex"
      overwrite: true
      rollover_alias: "myindex-%{[agent.version]}"
      pattern: "{now/d}-0000001"
      policy_file: "/usr/share/filebeat/config/myindex.policy.json"
      

    #================================ Processors =====================================

    # Configure processors to enhance or manipulate events generated by the beat.

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

Above is what I think matches your example, but with my specific index name. This still however is not working.

What I would like to know is what filebeat is actually doing such that I can possibly understand what is wrong.

Are you able to advise what is wrong with my config by providing suitable changes that I can replicate?

thanks

tractor_boy · July 29, 2022, 4:44pm

What do I put in the config file to implement the alias?

tractor_boy · July 29, 2022, 5:10pm

I am going to have to log off for the weekend, but anything that you can suggest is gratefully received as I am at my wits end having spent 2 solid weeks not making any progress.

rugenl · July 29, 2022, 5:47pm

When something like this happens, remove all your customization, delete all templates, indices, ILM policies, everything. Start over letting filebeat use it's default config, default names, default everything. When that works, change one thing at a time working toward your custom config.

tractor_boy · August 1, 2022, 7:41am

Effectively been doing that for the last 2 weeks! I have tried so many combinations just not found one that actually works.

I seem to have got the ilm piece sorted, so not able to get filebeat to write to myindex, but also not able to find any logs anywhere to attempt to understand what is going wrong.

Christian_Dahlqvist · August 1, 2022, 8:46am

I do not think you can have this dynamic as agent.version is an event field that is not available on setup.

This must be the rollover alias and can not be dynamic.

tractor_boy · August 1, 2022, 3:09pm

the standard filebeat alias is filebeat-7.0.1 which suggests it can be dynamic.

Christian_Dahlqvist · August 1, 2022, 3:19pm

Which version of the stack are you using? If it is version 7.0.1, it is very old.

I would recommend always stating which version you are using as the answer can vary a lot based on this.

Topic		Replies	Views
No filebeat index into elasticsearch indices Beats filebeat	6	336	August 2, 2018
Filebeat configuration with output elasticsearch Issue Beats filebeat	3	481	March 18, 2019
The filebeat default index cannot be changed Beats filebeat	2	645	August 17, 2018
How to set cutom index in filebeat Beats filebeat	4	249	June 16, 2022
Dynamic index name not working with Filebeat Beats filebeat	4	1875	November 11, 2020

Filebeat writing to its own index

Related topics