Hi team,
Im trying to create custom index with Filebeat and ive read the official docs and disabled the ilm (setup.ilm.enabled: false) and also configured the template name and pattern but now index template is getting loaded as i can see it while running filebeat setup -e but the index is not getting created, it gives me - ILM policy and write alias loading not enabled. Can i get some expert advice?
I guess you are trying to setup connection with Kibana from filebeat, probably for in-store dashboards, etc. Can you please resolve that issue first and share the full logs for filebeat startup ?
Also, if index template has ben created, the index based out of it will only be created when you ingest the first event for that index.
Hi Ayush,
Im actually taking data from 1883 port(ActiveMQ broker) and using filebeat to fetch the topic-data from broker and send it to Elasticsearch.
filebeat.yml
filebeat.inputs:
- type: mqtt
hosts:- 'tcp://127.0.0.1:1883'
topics: - sensor-data
qos: 2
client_id: filebeat_mqtt
username: system
password: manager
tags: - mqtt
- iot
- filebeat
- broker
- gps
enabled: true
fields_under_root: true
fields:
event.dataset: sensor-data
event.module: filebeat-mqtt
event.outcome: iot-sensors
event.category: sensor
- 'tcp://127.0.0.1:1883'
setup.template.settings:
index.number_of_shards: 1
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.ilm.enabled: false
setup.template.overwrite: true
output.elasticsearch:
hosts: using my cloud_id
index: "sensor-data-%{+yyyy-MM-dd}"
protocol: "https"
and what ive concluded so far is that my filebeat is not accepting the input details like topic, qos, client_id, etc because the error is - starting input, keys present on the config: [filebeat.inputs.0.client_id filebeat.inputs.0.enabled filebeat.inputs.0.fields.event.category filebeat.inputs.0.fields.event.dataset filebeat.inputs.0.fields.event.module filebeat.inputs.0.fields.event.outcome filebeat.inputs.0.fields_under_root filebeat.inputs.0.hosts.0 filebeat.inputs.0.password filebeat.inputs.0.qos filebeat.inputs.0.tags.0 filebeat.inputs.0.tags.1 filebeat.inputs.0.tags.2 filebeat.inputs.0.tags.3 filebeat.inputs.0.tags.4 filebeat.inputs.0.topics.0 filebeat.inputs.0.type filebeat.inputs.0.username]
Can you please format the YAML using "Preformatted text </>" option, its hard to read it without proper indentation.
oh yeah sure,
filebeat.inputs:
- type: mqtt
hosts:
- 'tcp://127.0.0.1:1883'
topics:
- sensor-data
qos: 2
client_id: filebeat_mqtt
username: system
password: manager
tags:
- mqtt
- iot
- filebeat
- broker
- gps
enabled: true
fields_under_root: true
fields:
event.dataset: sensor-data
event.module: filebeat-mqtt
event.outcome: iot-sensors
event.category: sensor
setup.template.settings:
index.number_of_shards: 1
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.ilm.enabled: false
setup.template.overwrite: true
output.elasticsearch:
hosts: using my cloud_id
index: "sensor-data-%{+yyyy-MM-dd}"
protocol: "https"
setup.template.settings:
index.number_of_shards: 1
please mind the spaces in yaml. Other than this, your config looks ok. Can you capture filebeat logs by running in debug mode for 5-10 minutes and corresponding ES logs ?
Hi, the indentation is correct as you mentioned, and please check the logs of filebeat & sorry i wont be able to provide es logs as im using hosted services and dont have access to es.
>filebeat -e
2023-01-10T21:26:19.412+0530 INFO instance/beat.go:685 Home path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4] Config path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4] Data path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data] Logs path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\logs] Hostfs Path: [/]
2023-01-10T21:26:19.412+0530 INFO instance/beat.go:693 Beat ID: bae03a30-3777-4afd-9bd1-74ab1973fc8d
2023-01-10T21:26:19.427+0530 WARN [add_cloud_metadata] add_cloud_metadata/provider_aws_ec2.go:79 read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.. No token in the metadata request will be used.
2023-01-10T21:26:19.428+0530 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2023-01-10T21:26:19.428+0530 INFO [beat] instance/beat.go:1039 Beat info {"system_info": {"beat": {"path": {"config": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "data": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\data", "home": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "logs": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\logs"}, "type": "filebeat", "uuid": "bae03a30-3777-4afd-9bd1-74ab1973fc8d"}}}
2023-01-10T21:26:19.428+0530 INFO [beat] instance/beat.go:1048 Build info {"system_info": {"build": {"commit": "ea28c0419dc4ede9318c4b34a732ce11b03482b7", "libbeat": "7.17.4", "time": "2022-05-18T16:46:57.000Z", "version": "7.17.4"}}}
2023-01-10T21:26:19.428+0530 INFO [beat] instance/beat.go:1051 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.17.9"}}}
2023-01-10T21:26:19.436+0530 INFO [beat] instance/beat.go:1055 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-01-03T21:33:17.94+05:30","name":"abc","ip":["10.255.43.76/32","169.254.52.79/16","169.254.83.6/16","192.168.43.201/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.19041.2364 (WinBuild.160101.0800)","mac":["02:05:85:7f:eb:80","c4:23:60:f1:48:26","c6:23:60:f1:48:25","c4:23:60:f1:48:25"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"19044.2364"},"timezone":"IST","timezone_offset_sec":19800,"id":"63d1da06-50be-4ef0-ab77-99fd496e16af"}}}
2023-01-10T21:26:19.436+0530 INFO [beat] instance/beat.go:1084 Process info {"system_info": {"process": {"cwd": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "exe": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\filebeat.exe", "name": "filebeat.exe", "pid": 7416, "ppid": 11708, "start_time": "2023-01-10T21:26:16.664+0530"}}}
2023-01-10T21:26:19.436+0530 INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.4
2023-01-10T21:26:19.436+0530 WARN [cfgwarn] tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2023-01-10T21:26:19.437+0530 INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: "cloud-url"
2023-01-10T21:26:19.440+0530 INFO [publisher] pipeline/module.go:113 Beat name: abc
2023-01-10T21:26:19.441+0530 INFO instance/beat.go:492 filebeat start running.
2023-01-10T21:26:19.442+0530 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2023-01-10T21:26:19.442+0530 INFO memlog/store.go:119 Loading data file of 'C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data\registry\filebeat' succeeded. Active transaction id=0
2023-01-10T21:26:19.442+0530 INFO memlog/store.go:124 Finished loading transaction log file for 'C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data\registry\filebeat'. Active transaction id=0
2023-01-10T21:26:19.443+0530 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:117 starting input, keys present on the config: [filebeat.inputs.0.client_id filebeat.inputs.0.enabled filebeat.inputs.0.fields.event.category filebeat.inputs.0.fields.event.dataset filebeat.inputs.0.fields.event.module filebeat.inputs.0.fields.event.outcome filebeat.inputs.0.fields_under_root filebeat.inputs.0.hosts.0 filebeat.inputs.0.password filebeat.inputs.0.qos filebeat.inputs.0.tags.0 filebeat.inputs.0.tags.1 filebeat.inputs.0.tags.2 filebeat.inputs.0.tags.3 filebeat.inputs.0.tags.4 filebeat.inputs.0.topics.0 filebeat.inputs.0.type filebeat.inputs.0.username]
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:148 Starting input (ID: 14062064499327290867)
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2023-01-10T21:26:19.458+0530 WARN [libmqtt] paho.mqtt.golang@v1.2.1-0.20200121105743-0d940dd29fd2/memstore.go:137 [store] memorystore wiped
2023-01-10T21:26:49.487+0530 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":140}},"total":{"ticks":250,"time":{"ms":265},"value":250},"user":{"ticks":125,"time":{"ms":125}}},"handles":{"open":254},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":32293},"version":"7.17.4"},"memstats":{"gc_next":18611648,"memory_alloc":9858408,"memory_sys":37399976,"memory_total":55882632,"rss":60829696},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8}}}}}
2023-01-10T21:27:19.476+0530 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":31}},"total":{"ticks":281,"time":{"ms":31},"value":281},"user":{"ticks":125}},"handles":{"open":252},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":62282},"version":"7.17.4"},"memstats":{"gc_next":18611648,"memory_alloc":10135864,"memory_total":56160088,"rss":60899328},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2023-01-10T21:28:24.188+0530 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":187,"time":{"ms":32}},"total":{"ticks":312,"time":{"ms":32},"value":312},"user":{"ticks":125}},"handles":{"open":260},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":126993},"version":"7.17.4"},"memstats":{"gc_next":19221104,"memory_alloc":9615408,"memory_total":56441128,"rss":58691584},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
Cool, so filebeat is up and running without any errors. Are you sure there are events to be ready from mqtt? Also, you logs doesn't show if filebeat was able to connect with ES successfully to send the events, that's another open ended thingy.
Essentially the index is created when the first event is sent from filebeat and stored in ES for that index, so please make sure the data is being ingested into ES. Probably check from GET _cat/indices?v and see if docs_count is increasing for your current index and that your cluster is green and doesn't have any resource pressure.
Yes, the topics are ready in mqtt broker, and im able to connect to ES because if i remove the custom index and template, the default empty index filebeat-7.14.1-2023.01.10-000004 gets created and as soon as i apply template settings for creating custom index, no index is getting created.
Try with:
setup.template.settings:
index.number_of_shards: 1
setup.ilm.enabled: false
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.template.overwrite: true
setup.template.enabled: false
the hell! it worked 
Thanks a lot mate!
Good to know 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.