Filebeat - seems to be not picking up when using the inputs.d directory

Elastic Stack Beats
1

Hello,

When having a play around with Filebeat and the input capabilities of the agent.
Below is the config from the filebeat.yml and an example of a yml that is in the inputs.d directory.

filebeat.yml - Works when config like this

#============================= Filebeat inputs ===============================

#filebeat.config:
#  inputs:
#    enabled: true
# Glob pattern for configuration loading
#    path: /etc/filebeat/inputs.d/*.yml

# Set to true to enable config reloading
#    reload.enabled: true

# Period on which files under path should be checked for changes
#    reload.period: 10s

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: s3
  enabled: true
  queue_url: https://thisisalink.to.a.place.com
  tags: ["this-is-a-tag"]

filebeat.yml - seems to struggle when only using the inputs config

#============================= Filebeat inputs ===============================

filebeat.config:
  inputs:
    enabled: true
# Glob pattern for configuration loading
    path: /etc/filebeat/inputs.d/*.yml

# Set to true to enable config reloading
    reload.enabled: true

# Period on which files under path should be checked for changes
    reload.period: 10s

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: s3
  enabled: false
  queue_url: https://thisisalink.to.a.place.com
  tags: ["this-is-a-tag"]

Whereas, this is the the yml file which is in the inputs.d directory (same as when put in the filebeat.yml)

filebeat.inputs:

- type: s3
  enabled: true
  queue_url: https://thisisalink.to.a.place.com
  tags: ["this-is-a-tag"]

With this set up, it seems that Filebeat is struggling to pick up the needed data sets/ logs that are intended to be monitored. Where as, when I put everything in the filebeat.yml, i.e. when putting the config in the filebeat.input section. You are able to see the data flow through great.

I am a little stuck here as I am unsure why this is. Would this be a permissions thing? However this has the same set up and permission setting as the normal modules.d directory has.

OR would this be a case that I am missing something small or just having a slow moment.

Thank you!

2

Hey @Apollyon,

When loading external configuration files with filebeat.config.inputs, the content of the files should be only a list of inputs. In your case you would need to remove the filebeat.inputs: line, so your file in the inputs.d directory looks like that:

- type: s3
  enabled: true
  queue_url: https://thisisalink.to.a.place.com
  tags: ["this-is-a-tag"]
3

Hey @jsoriano,

ahhhh, this makes sense!

Thank you very much for that!!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.