# Filebeat is unable to create custom index

**URL:** <https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 9, 2023, 5:55pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755 "2023-01-09T17:55:28Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![AdityaKhajuria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adityakhajuria/32/101150_2.png) [@AdityaKhajuria](https://discuss.elastic.co/u/AdityaKhajuria)\
**Post date:** [January 9, 2023, 5:55pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/1 "2023-01-09T17:55:28Z")

</div>

Hi team,  
Im trying to create custom index with Filebeat and ive read the official docs and disabled the ilm (setup.ilm.enabled: false) and also configured the template name and pattern but now index template is getting loaded as i can see it while running filebeat setup -e but the index is not getting created, it gives me - ILM policy and write alias loading not enabled. Can i get some expert advice?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/3/632871f0344450ce56e785e7f6ecc33ce582b40d.png)

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [January 10, 2023, 2:08pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/2 "2023-01-10T14:08:43Z")

</div>

I guess you are trying to setup connection with Kibana from filebeat, probably for in-store dashboards, etc. Can you please resolve that issue first and share the full logs for filebeat startup ?

Also, if index template has ben created, the index based out of it will **only** be created when you ingest the first event for that index.

---

<div class="post-metadata">

**Author:** ![AdityaKhajuria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adityakhajuria/32/101150_2.png) [@AdityaKhajuria](https://discuss.elastic.co/u/AdityaKhajuria)\
**Post date:** [January 10, 2023, 2:39pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/3 "2023-01-10T14:39:23Z")

</div>

Hi Ayush,  
Im actually taking data from 1883 port(ActiveMQ broker) and using filebeat to fetch the topic-data from broker and send it to Elasticsearch.  
**filebeat.yml**

filebeat.inputs:

- type: mqtt  
hosts:
  - 'tcp://127.0.0.1:1883'  
topics:
  - sensor-data  
qos: 2  
client\_id: filebeat\_mqtt  
username: system  
password: manager  
tags:
  - mqtt
  - iot
  - filebeat
  - broker
  - gps  
enabled: true  
fields\_under\_root: true  
fields:  
event.dataset: sensor-data  
event.module: filebeat-mqtt  
event.outcome: iot-sensors  
event.category: sensor

setup.template.settings:  
index.number\_of\_shards: 1  
setup.template.name: "sensor-data"  
setup.template.pattern: "sensor-data-\*"  
setup.ilm.enabled: false  
setup.template.overwrite: true

output.elasticsearch:  
hosts: using my cloud\_id  
index: "sensor-data-%{+yyyy-MM-dd}"  
protocol: "https"

and what ive concluded so far is that my filebeat is not accepting the input details like topic, qos, client\_id, etc because the error is - starting input, keys present on the config: [filebeat.inputs.0.client\_id filebeat.inputs.0.enabled filebeat.inputs.0.fields.event.category filebeat.inputs.0.fields.event.dataset filebeat.inputs.0.fields.event.module filebeat.inputs.0.fields.event.outcome filebeat.inputs.0.fields\_under\_root filebeat.inputs.0.hosts.0 filebeat.inputs.0.password filebeat.inputs.0.qos filebeat.inputs.0.tags.0 filebeat.inputs.0.tags.1 filebeat.inputs.0.tags.2 filebeat.inputs.0.tags.3 filebeat.inputs.0.tags.4 filebeat.inputs.0.topics.0 filebeat.inputs.0.type filebeat.inputs.0.username]

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [January 10, 2023, 3:02pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/4 "2023-01-10T15:02:14Z")

</div>

Can you please format the YAML using "Preformatted text \</\>" option, its hard to read it without proper indentation.

---

<div class="post-metadata">

**Author:** ![AdityaKhajuria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adityakhajuria/32/101150_2.png) [@AdityaKhajuria](https://discuss.elastic.co/u/AdityaKhajuria)\
**Post date:** [January 10, 2023, 3:23pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/5 "2023-01-10T15:23:21Z")

</div>

> [@Ayush\_Mathur](#):
>
> Can you please format the YAML using "Preformatted text \</\>" option, its hard to read it without proper indentation.

oh yeah sure,

```auto
filebeat.inputs:
  - type: mqtt
    hosts:
      - 'tcp://127.0.0.1:1883'
    topics:
      - sensor-data
    qos: 2
    client_id: filebeat_mqtt
    username: system
    password: manager
    tags:
      - mqtt
      - iot
      - filebeat
      - broker
      - gps
    enabled: true
    fields_under_root: true
    fields:
      event.dataset: sensor-data
      event.module: filebeat-mqtt
      event.outcome: iot-sensors
      event.category: sensor
setup.template.settings:
index.number_of_shards: 1
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.ilm.enabled: false
setup.template.overwrite: true

output.elasticsearch:
hosts: using my cloud_id
index: "sensor-data-%{+yyyy-MM-dd}"
protocol: "https"

```

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [January 10, 2023, 3:43pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/6 "2023-01-10T15:43:47Z")

</div>

```auto
setup.template.settings:
  index.number_of_shards: 1

```

please mind the spaces in yaml. Other than this, your config looks ok. Can you capture filebeat logs by running in debug mode for 5-10 minutes and corresponding ES logs ?

---

<div class="post-metadata">

**Author:** ![AdityaKhajuria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adityakhajuria/32/101150_2.png) [@AdityaKhajuria](https://discuss.elastic.co/u/AdityaKhajuria)\
**Post date:** [January 10, 2023, 4:05pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/7 "2023-01-10T16:05:56Z")

</div>

Hi, the indentation is correct as you mentioned, and please check the logs of filebeat & sorry i wont be able to provide es logs as im using hosted services and dont have access to es.

```auto
>filebeat -e
2023-01-10T21:26:19.412+0530 INFO instance/beat.go:685 Home path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4] Config path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4] Data path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data] Logs path: [C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\logs] Hostfs Path: [/]
2023-01-10T21:26:19.412+0530 INFO instance/beat.go:693 Beat ID: bae03a30-3777-4afd-9bd1-74ab1973fc8d
2023-01-10T21:26:19.427+0530 WARN [add_cloud_metadata] add_cloud_metadata/provider_aws_ec2.go:79 read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.. No token in the metadata request will be used.
2023-01-10T21:26:19.428+0530 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2023-01-10T21:26:19.428+0530 INFO [beat] instance/beat.go:1039 Beat info {"system_info": {"beat": {"path": {"config": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "data": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\data", "home": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "logs": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\logs"}, "type": "filebeat", "uuid": "bae03a30-3777-4afd-9bd1-74ab1973fc8d"}}}
2023-01-10T21:26:19.428+0530 INFO [beat] instance/beat.go:1048 Build info {"system_info": {"build": {"commit": "ea28c0419dc4ede9318c4b34a732ce11b03482b7", "libbeat": "7.17.4", "time": "2022-05-18T16:46:57.000Z", "version": "7.17.4"}}}
2023-01-10T21:26:19.428+0530 INFO [beat] instance/beat.go:1051 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.17.9"}}}
2023-01-10T21:26:19.436+0530 INFO [beat] instance/beat.go:1055 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-01-03T21:33:17.94+05:30","name":"abc","ip":["10.255.43.76/32","169.254.52.79/16","169.254.83.6/16","192.168.43.201/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.19041.2364 (WinBuild.160101.0800)","mac":["02:05:85:7f:eb:80","c4:23:60:f1:48:26","c6:23:60:f1:48:25","c4:23:60:f1:48:25"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"19044.2364"},"timezone":"IST","timezone_offset_sec":19800,"id":"63d1da06-50be-4ef0-ab77-99fd496e16af"}}}
2023-01-10T21:26:19.436+0530 INFO [beat] instance/beat.go:1084 Process info {"system_info": {"process": {"cwd": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4", "exe": "C:\\Users\\AKHAJURI\\Desktop\\filebeat-7.17.4\\filebeat.exe", "name": "filebeat.exe", "pid": 7416, "ppid": 11708, "start_time": "2023-01-10T21:26:16.664+0530"}}}
2023-01-10T21:26:19.436+0530 INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.4
2023-01-10T21:26:19.436+0530 WARN [cfgwarn] tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2023-01-10T21:26:19.437+0530 INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: "cloud-url"
2023-01-10T21:26:19.440+0530 INFO [publisher] pipeline/module.go:113 Beat name: abc
2023-01-10T21:26:19.441+0530 INFO instance/beat.go:492 filebeat start running.
2023-01-10T21:26:19.442+0530 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2023-01-10T21:26:19.442+0530 INFO memlog/store.go:119 Loading data file of 'C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data\registry\filebeat' succeeded. Active transaction id=0
2023-01-10T21:26:19.442+0530 INFO memlog/store.go:124 Finished loading transaction log file for 'C:\Users\AKHAJURI\Desktop\filebeat-7.17.4\data\registry\filebeat'. Active transaction id=0
2023-01-10T21:26:19.443+0530 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:117 starting input, keys present on the config: [filebeat.inputs.0.client_id filebeat.inputs.0.enabled filebeat.inputs.0.fields.event.category filebeat.inputs.0.fields.event.dataset filebeat.inputs.0.fields.event.module filebeat.inputs.0.fields.event.outcome filebeat.inputs.0.fields_under_root filebeat.inputs.0.hosts.0 filebeat.inputs.0.password filebeat.inputs.0.qos filebeat.inputs.0.tags.0 filebeat.inputs.0.tags.1 filebeat.inputs.0.tags.2 filebeat.inputs.0.tags.3 filebeat.inputs.0.tags.4 filebeat.inputs.0.topics.0 filebeat.inputs.0.type filebeat.inputs.0.username]
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:148 Starting input (ID: 14062064499327290867)
2023-01-10T21:26:19.443+0530 INFO [crawler] beater/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2023-01-10T21:26:19.458+0530 WARN [libmqtt] paho.mqtt.golang@v1.2.1-0.20200121105743-0d940dd29fd2/memstore.go:137 [store] memorystore wiped
2023-01-10T21:26:49.487+0530 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":140}},"total":{"ticks":250,"time":{"ms":265},"value":250},"user":{"ticks":125,"time":{"ms":125}}},"handles":{"open":254},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":32293},"version":"7.17.4"},"memstats":{"gc_next":18611648,"memory_alloc":9858408,"memory_sys":37399976,"memory_total":55882632,"rss":60829696},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8}}}}}
2023-01-10T21:27:19.476+0530 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":31}},"total":{"ticks":281,"time":{"ms":31},"value":281},"user":{"ticks":125}},"handles":{"open":252},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":62282},"version":"7.17.4"},"memstats":{"gc_next":18611648,"memory_alloc":10135864,"memory_total":56160088,"rss":60899328},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2023-01-10T21:28:24.188+0530 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":187,"time":{"ms":32}},"total":{"ticks":312,"time":{"ms":32},"value":312},"user":{"ticks":125}},"handles":{"open":260},"info":{"ephemeral_id":"be4e9d19-9c40-42db-9e43-c747043bd22c","uptime":{"ms":126993},"version":"7.17.4"},"memstats":{"gc_next":19221104,"memory_alloc":9615408,"memory_total":56441128,"rss":58691584},"runtime":{"goroutines":39}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}

```

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [January 11, 2023, 8:03am UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/8 "2023-01-11T08:03:33Z")

</div>

Cool, so filebeat is up and running without any errors. Are you sure there are events to be ready from mqtt? Also, you logs doesn't show if filebeat was able to connect with ES successfully to send the events, that's another open ended thingy.  
Essentially the index is created when the first event is sent from filebeat and stored in ES for that index, so please make sure the data is being ingested into ES. Probably check from `GET _cat/indices?v` and see if docs\_count is increasing for your current index and that your cluster is green and doesn't have any resource pressure.

---

<div class="post-metadata">

**Author:** ![AdityaKhajuria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adityakhajuria/32/101150_2.png) [@AdityaKhajuria](https://discuss.elastic.co/u/AdityaKhajuria)\
**Post date:** [January 11, 2023, 9:59am UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/9 "2023-01-11T09:59:40Z")

</div>

Yes, the topics are ready in mqtt broker, and im able to connect to ES because if i remove the custom index and template, the default empty index filebeat-7.14.1-2023.01.10-000004 gets created and as soon as i apply template settings for creating custom index, no index is getting created.

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [January 11, 2023, 10:03am UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/10 "2023-01-11T10:03:01Z")

</div>

Try with:

```auto
setup.template.settings:
  index.number_of_shards: 1
setup.ilm.enabled: false
setup.template.name: "sensor-data"
setup.template.pattern: "sensor-data-*"
setup.template.overwrite: true
setup.template.enabled: false

```

---

<div class="post-metadata">

**Author:** ![AdityaKhajuria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adityakhajuria/32/101150_2.png) [@AdityaKhajuria](https://discuss.elastic.co/u/AdityaKhajuria)\
**Post date:** [January 11, 2023, 10:42am UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/12 "2023-01-11T10:42:07Z")

</div>

the hell! it worked 😵  
Thanks a lot mate!

---

<div class="post-metadata">

**Author:** ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)\
**Post date:** [January 11, 2023, 11:19am UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/13 "2023-01-11T11:19:55Z")

</div>

Good to know 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2023, 1:20pm UTC](https://discuss.elastic.co/t/filebeat-is-unable-to-create-custom-index/322755/14 "2023-02-08T13:20:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.