Filebeat does not create index template

Elastic Stack Beats
1

Hello,

as said in the object, i'm using filebeat but for some reason its index template wasn't created on ES, the strange thing is that with the same configuration metricbeat's template was created.

image
image979×416 31.1 KB

I report settings configured in filebeat.yml:

#==================== Elasticsearch template setting ==========================

setup.template.enabled: true

setup.template.settings:

  index.number_of_shards: 1

  index.codec: best_compression

Any suggestion on this?

Thanks
KR
Simone

2

Hi @snolfi

Could you share your output configuration?
An other option is to export the index template manually and import into Elasticsearch with curl.

3

Hi @fgjensen,

here what you asked:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

#  # Array of hosts to connect to.

#  hosts: ["{{ elk_host }}:9200"]

  # Optional protocol and basic auth credentials.

  #protocol: "https"

  #username: "elastic"

  #password: "changeme"

#----------------------------- Logstash output --------------------------------

output.logstash:

  # The Logstash hosts

  hosts: ["{{ elk_host }}:{{ jboss_pl_port }}"]

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

#----------------------------- Console output --------------------------------

#

#output.file:

#  path: "/tmp/filebeat"

#  filename: filebeat

Where could i find filebeat index template if it's not present in kibana console?

Thanks
Simone

4

When you are using environment variables in the filebeat configuration file the syntax is {var}, like this in your case hosts: ["{elk_host }:${jboss_pl_port}"]

However, you cannot export the filebeat template and ingest pipelines, when you have enabled the logstash output. So you have to disable the logstash output temporary and enable the elasticsearch output, while you export the filebeat index template and the ingest pipelines.

An other option is to export the template and pipelines manually to a file and import the objects into Elasticsearch. If you do not see the index template in Kibana it is not present i Elasticsearch.

BR
Flemming

5

Hi Flemming,

there was a mistake, in this line:

hosts: ["{{ elk_host }}:{{ jboss_pl_port }}"]

I used parethesis to define a variable, in my yaml the real entry is:

hosts: ["10.17.46.152:5044"]

I enabled elasticsearch output but nothing changed

KR
Simone

6

with the logstash output commented out, and the elastic output active, what is the result of running filebeat -e -c config.yml setup template?

7

Following

[root@hostname ~]# filebeat -e -c /usr/share/filebeat/module/mssql/log/config/config.yml setup template
Exiting: error loading config file: yaml: line 3: did not find expected key


[root@hostname ~]# cat /usr/share/filebeat/module/mssql/log/config/config.yml
type: log
paths:
  {{ range $i, $path := .paths }}
- {{$path}}
  {{ end }}
exclude_files: [".gz$"]

multiline.pattern: '^\d\d'
multiline.negate: true
multiline.match: after

processors:
- add_locale: ~

I ran these commands for the unique config.yml present on the host

8

Hi Simone I do not think you can use the range condition when specifying the paths to be crawled, since its a condition in a processor. Try to break down the problem into two parts e.g. 1) load the index template and 2) fix the configuration for the logs. Filebeat loads the index template automatically when you have enabled the Elasticsearch output and disabled the Logstash output. After loading the index template you can concentrate on the configuration with the selftest as pointed out by @tjfred. Yet, an other option is to set the filebeat log level to debug. This often gives the hits required to fix the configuration.

9

Hello Flemming,

using the suggested configuration on filebeat.yml, today the index-template has been created, so it worked. I exported my index-template and imported it, using curl in environments where it was needed, thanks a lot.

Back to the configurations of the log, could you please tell me how to fix the configuration?

Regards
Simone

10

Glad to hear you succeeded with the index template.

According to the Filebeat mssql module documentation the module "Sets the default paths to the log files". Could you, please, explain in more details what your path configuration should expand into? I would expect a configuration like /path/to/log//.log

11

Hello Flemming,

there was a little mistake working on this module, in reality we are not using this module so there's no need to fix anything.

I'm satisfied we solved index-template issue, thanks a lot!!

KR
Simone

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.