Filebeat is unable to create custom index

Hi team,
Im trying to create custom index with Filebeat and ive read the official docs and disabled the ilm (setup.ilm.enabled: false) and also configured the template name and pattern but now index template is getting loaded as i can see it while running filebeat setup -e but the index is not getting created, it gives me - ILM policy and write alias loading not enabled. Can i get some expert advice?

I guess you are trying to setup connection with Kibana from filebeat, probably for in-store dashboards, etc. Can you please resolve that issue first and share the full logs for filebeat startup ?

Also, if index template has ben created, the index based out of it will only be created when you ingest the first event for that index.

Hi Ayush,
Im actually taking data from 1883 port(ActiveMQ broker) and using filebeat to fetch the topic-data from broker and send it to Elasticsearch.
filebeat.yml

filebeat.inputs:

  • type: mqtt
    hosts:
    • 'tcp://127.0.0.1:1883'
      topics:
    • sensor-data
      qos: 2
      client_id: filebeat_mqtt
      username: system
      password: manager
      tags:
    • mqtt
    • iot
    • filebeat
    • broker
    • gps
      enabled: true
      fields_under_root: true
      fields:
      event.dataset: sensor-data
      event.module: filebeat-mqtt
      event.outcome: iot-sensors
      event.category: sensor

setup.template.settings:
index.number_of_shards: 1
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.ilm.enabled: false
setup.template.overwrite: true

output.elasticsearch:
hosts: using my cloud_id
index: "sensor-data-%{+yyyy-MM-dd}"
protocol: "https"

and what ive concluded so far is that my filebeat is not accepting the input details like topic, qos, client_id, etc because the error is - starting input, keys present on the config: [filebeat.inputs.0.client_id filebeat.inputs.0.enabled filebeat.inputs.0.fields.event.category filebeat.inputs.0.fields.event.dataset filebeat.inputs.0.fields.event.module filebeat.inputs.0.fields.event.outcome filebeat.inputs.0.fields_under_root filebeat.inputs.0.hosts.0 filebeat.inputs.0.password filebeat.inputs.0.qos filebeat.inputs.0.tags.0 filebeat.inputs.0.tags.1 filebeat.inputs.0.tags.2 filebeat.inputs.0.tags.3 filebeat.inputs.0.tags.4 filebeat.inputs.0.topics.0 filebeat.inputs.0.type filebeat.inputs.0.username]

Can you please format the YAML using "Preformatted text </>" option, its hard to read it without proper indentation.

oh yeah sure,

filebeat.inputs:
  - type: mqtt
    hosts:
      - 'tcp://127.0.0.1:1883'
    topics:
      - sensor-data
    qos: 2
    client_id: filebeat_mqtt
    username: system
    password: manager
    tags:
      - mqtt
      - iot
      - filebeat
      - broker
      - gps
    enabled: true
    fields_under_root: true
    fields:
      event.dataset: sensor-data
      event.module: filebeat-mqtt
      event.outcome: iot-sensors
      event.category: sensor
setup.template.settings:
index.number_of_shards: 1
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.ilm.enabled: false
setup.template.overwrite: true

output.elasticsearch:
hosts: using my cloud_id
index: "sensor-data-%{+yyyy-MM-dd}"
protocol: "https"

setup.template.settings:
  index.number_of_shards: 1

please mind the spaces in yaml. Other than this, your config looks ok. Can you capture filebeat logs by running in debug mode for 5-10 minutes and corresponding ES logs ?

Hi, the indentation is correct as you mentioned, and please check the logs of filebeat & sorry i wont be able to provide es logs as im using hosted services and dont have access to es.

>filebeat -e
2023-01-10T21:26:19.412+0530    INFO    instance/beat.go:685    Home path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4] Config path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4] Data path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data] Logs path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\logs] Hostfs Path: [/]
2023-01-10T21:26:19.412+0530    INFO    instance/beat.go:693    Beat ID: bae03a30-3777-4afd-9bd1-74ab1973fc8d
2023-01-10T21:26:19.427+0530    WARN    [add_cloud_metadata]    add_cloud_metadata/provider_aws_ec2.go:79       read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.. No token in the metadata request will be used.
2023-01-10T21:26:19.428+0530    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.
2023-01-10T21:26:19.428+0530    INFO    [beat]  instance/beat.go:1039   Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "data": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\data", "home": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "logs": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\logs"}, "type": "filebeat", "uuid": "bae03a30-3777-4afd-9bd1-74ab1973fc8d"}}}
2023-01-10T21:26:19.428+0530    INFO    [beat]  instance/beat.go:1048   Build info      {"system_info": {"build": {"commit": "ea28c0419dc4ede9318c4b34a732ce11b03482b7", "libbeat": "7.17.4", "time": "2022-05-18T16:46:57.000Z", "version": "7.17.4"}}}
2023-01-10T21:26:19.428+0530    INFO    [beat]  instance/beat.go:1051   Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.17.9"}}}
2023-01-10T21:26:19.436+0530    INFO    [beat]  instance/beat.go:1055   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-01-03T21:33:17.94+05:30","name":"abc","ip":["10.255.43.76/32","169.254.52.79/16","169.254.83.6/16","192.168.43.201/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.19041.2364 (WinBuild.160101.0800)","mac":["02:05:85:7f:eb:80","c4:23:60:f1:48:26","c6:23:60:f1:48:25","c4:23:60:f1:48:25"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"19044.2364"},"timezone":"IST","timezone_offset_sec":19800,"id":"63d1da06-50be-4ef0-ab77-99fd496e16af"}}}
2023-01-10T21:26:19.436+0530    INFO    [beat]  instance/beat.go:1084   Process info    {"system_info": {"process": {"cwd": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "exe": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\filebeat.exe", "name": "filebeat.exe", "pid": 7416, "ppid": 11708, "start_time": "2023-01-10T21:26:16.664+0530"}}}
2023-01-10T21:26:19.436+0530    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.4
2023-01-10T21:26:19.436+0530    WARN    [cfgwarn]       tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2023-01-10T21:26:19.437+0530    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: "cloud-url"
2023-01-10T21:26:19.440+0530    INFO    [publisher]     pipeline/module.go:113  Beat name: abc
2023-01-10T21:26:19.441+0530    INFO    instance/beat.go:492    filebeat start running.
2023-01-10T21:26:19.442+0530    INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2023-01-10T21:26:19.442+0530    INFO    memlog/store.go:119     Loading data file of 'C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data\registry\filebeat' succeeded. Active transaction id=0
2023-01-10T21:26:19.442+0530    INFO    memlog/store.go:124     Finished loading transaction log file for 'C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data\registry\filebeat'. Active transaction id=0
2023-01-10T21:26:19.443+0530    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2023-01-10T21:26:19.443+0530    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 1
2023-01-10T21:26:19.443+0530    INFO    [crawler]       beater/crawler.go:117   starting input, keys present on the config: [filebeat.inputs.0.client_id filebeat.inputs.0.enabled filebeat.inputs.0.fields.event.category filebeat.inputs.0.fields.event.dataset filebeat.inputs.0.fields.event.module filebeat.inputs.0.fields.event.outcome filebeat.inputs.0.fields_under_root filebeat.inputs.0.hosts.0 filebeat.inputs.0.password filebeat.inputs.0.qos filebeat.inputs.0.tags.0 filebeat.inputs.0.tags.1 filebeat.inputs.0.tags.2 filebeat.inputs.0.tags.3 filebeat.inputs.0.tags.4 filebeat.inputs.0.topics.0 filebeat.inputs.0.type filebeat.inputs.0.username]
2023-01-10T21:26:19.443+0530    INFO    [crawler]       beater/crawler.go:148   Starting input (ID: 14062064499327290867)
2023-01-10T21:26:19.443+0530    INFO    [crawler]       beater/crawler.go:106   Loading and starting Inputs completed. Enabled inputs: 1
2023-01-10T21:26:19.458+0530    WARN    [libmqtt]       paho.mqtt.golang@v1.2.1-0.20200121105743-0d940dd29fd2/memstore.go:137   [store]   memorystore wiped
2023-01-10T21:26:49.487+0530    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":140}},"total":{"ticks":250,"time":{"ms":265},"value":250},"user":{"ticks":125,"time":{"ms":125}}},"handles":{"open":254},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":32293},"version":"7.17.4"},"memstats":{"gc_next":18611648,"memory_alloc":9858408,"memory_sys":37399976,"memory_total":55882632,"rss":60829696},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8}}}}}
2023-01-10T21:27:19.476+0530    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":31}},"total":{"ticks":281,"time":{"ms":31},"value":281},"user":{"ticks":125}},"handles":{"open":252},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":62282},"version":"7.17.4"},"memstats":{"gc_next":18611648,"memory_alloc":10135864,"memory_total":56160088,"rss":60899328},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2023-01-10T21:28:24.188+0530    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":187,"time":{"ms":32}},"total":{"ticks":312,"time":{"ms":32},"value":312},"user":{"ticks":125}},"handles":{"open":260},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":126993},"version":"7.17.4"},"memstats":{"gc_next":19221104,"memory_alloc":9615408,"memory_total":56441128,"rss":58691584},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}


Cool, so filebeat is up and running without any errors. Are you sure there are events to be ready from mqtt? Also, you logs doesn't show if filebeat was able to connect with ES successfully to send the events, that's another open ended thingy.
Essentially the index is created when the first event is sent from filebeat and stored in ES for that index, so please make sure the data is being ingested into ES. Probably check from GET _cat/indices?v and see if docs_count is increasing for your current index and that your cluster is green and doesn't have any resource pressure.

Yes, the topics are ready in mqtt broker, and im able to connect to ES because if i remove the custom index and template, the default empty index filebeat-7.14.1-2023.01.10-000004 gets created and as soon as i apply template settings for creating custom index, no index is getting created.

Try with:

setup.template.settings:
  index.number_of_shards: 1
setup.ilm.enabled: false
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.template.overwrite: true
setup.template.enabled: false

the hell! it worked :dizzy_face:
Thanks a lot mate!

Good to know :slight_smile: