Filebeat 6.4.3 config issue

[sunny1 (shah)]

I am installing filebeat 6.4.3. For yml file i have custom index name. When i start filebeat it says setup.template.name and setup.template.patter have to be set if index name is modified. I am not sure how to pass these fields in yml file. Below is the yml config for me: Can someone please help me fix this.

#=========================== Filebeat inputs =============================

filebeat.inputs:


- type: log
  json.keys_under_root: true
  json.add_error_key: true


  enabled: true

  paths:
    - C:\LMSTestResults\LMS_Results\SunnyTestResults*.json

  fields:
   document_type: "sunny_mongo_results"
   

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false


#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  host: "localhost:5601"


#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "%{[fields][document_type]}"
  setup.template.enabled: false
  setup.template.name: ""
  setup.template.pattern: "*"





#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug

[steffens (Steffen Siering)]

You have to add name and pattern to the setup.template namespace. See the filebeat.reference.yml for examples.

[sunny1 (shah)]

Thanks for ur reply, but i am not able to get what i need to set for pattern and template. I looked at examples but most examples had constant for index which i dont have so not able to get correct syntax.

[steffens (Steffen Siering)]

Filebeat provisions the template mapping on startup. That is, they indeed quire some constant prefix.

Alternatively you can use filebeat export config, modify the template mapping (json file), and use the setup.template.json... settings to have filebeat load you template.

See Elasticsearch Index Templates docs to learn more about templates.
Also check out docs on manual loading the template.

[sunny1 (shah)]

Thanks Steffens.

I tried to give constant for my index as below but still same error to set name and pattern.

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "SG-%{[fields][document_type]}"
  setup.template.enabled: false
  setup.template.name: "SG"
  setup.template.pattern: "SG-*"

[sunny1 (shah)]

I also tried to set template manually using below command but same error

.\filebeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

[steffens (Steffen Siering)]

Do you have any error message(s) to share with us?

[sunny1 (shah)]

Sorry no errors, but all it ask is to set pattern and name which i already did.

[steffens (Steffen Siering)]

The indentation seems wrong. The setup settings are not part of the elasticsearch output:

setup.template.enabled: false
setup.template.name: "SG"
setup.template.pattern: "SG-*"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "SG-%{[fields][document_type]}"

[sunny1 (shah)]

Thanks Steffens, some of the examples i had seen had name and pattern being part of elasticsearch output.

But as you mentioned did the change and it helped.
Thanks again,
Sunny