# Multiple indexes not being created for multiple log files

**URL:** <https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 6, 2024, 11:23am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817 "2024-05-06T11:23:55Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [May 6, 2024, 11:23am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/1 "2024-05-06T11:23:55Z")

</div>

Hello,

I've been trying to create multiple indexes for multiple log files that get created within a server. I'm quite new to filebeat so wanted to know what I might be doing wrong within the code. The use case requires me to use filebeat to capture and send logs to elasticsearch.

Can someone please help. I'm currently testing the below code with filebeat 8.x

```auto
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: filestream
  id: test-index1-logs
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - {{ log_path_index1 }}
  fields:
    name: "index1"

- type: filestream
  id: test-index2-logs
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - {{ log_path_index2 }}
  fields:
    name: "index2"

# ======================= Elasticsearch template setting =======================
setup.template.enabled: true
setup.template.name: "test-%{[fields.name]}"
setup.template.pattern: "test-%{[fields.name]}-*"
setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: true
  allow_older_versions: true
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
  index: test-%{[fields.name]}-%{+yyyy.MM.dd}"

```

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [May 9, 2024, 10:19am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/2 "2024-05-09T10:19:56Z")

</div>

Can someone please help in this?

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [May 9, 2024, 6:56pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/3 "2024-05-09T18:56:28Z")

</div>

Hi @developer_cloud , Welcome to the Elastic community. The configuration file looks good to me. Any error you getting?

You can try [debugging steps](https://www.elastic.co/guide/en/beats/filebeat/current/enable-filebeat-debugging.html) and check if you getting any error or not.

You can paste error or warning if you getting any.

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [May 30, 2024, 9:04pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/4 "2024-05-30T21:04:23Z")

</div>

Hi @ashishtiwari1993 , I checked through the logs at /var/log/filebeat/\* and found an error stating `Connection marked as failed because the onConnect callback failed: error loading template: error creating template instance: key not found"`. This seems to be happening for Fields.name I believe. I'm currently using filebeat 8.x. Would there be any version related conflicts causing this issue?

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [June 3, 2024, 10:48am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/5 "2024-06-03T10:48:05Z")

</div>

> [@developer\_cloud](#):
>
> Connection marked as failed because the onConnect callback failed: error loading template: error creating template instance: key not found

Yes it could be the reason. May i what is the version of Filebeat and Elasticsearch?

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 3, 2024, 11:08am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/7 "2024-06-03T11:08:55Z")

</div>

I was using filebeat 8.x. However, when I switched back to filebeat 7.x, I did not encounter this error. Even then, the streams were not being created. Not sure what I might be doing wrong.

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [June 3, 2024, 11:12am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/8 "2024-06-03T11:12:05Z")

</div>

What it showing when you trying to load [template manually](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually) ?

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 3, 2024, 11:41am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/9 "2024-06-03T11:41:07Z")

</div>

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.  
Index setup finished.  
Loading dashboards (Kibana must be running and reachable)  
Skipping loading dashboards, Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/bin/kibana: No directory /usr/share/filebeat/bin/kibana/7  
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.  
See more: [Machine Learning in the Elastic Stack [8.13] | Elastic](https://www.elastic.co/guide/en/machine-learning/current/index.html)  
It is not possble to load ML jobs into an Elasticsearch 8.0.0 or newer using the Beat.  
Exiting: 1 error: Error setting up ML for apache\_ecs: 10 errors: ; ; ; ; ; ; ; ; ;

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 3, 2024, 11:55am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/10 "2024-06-03T11:55:31Z")

</div>

filebeat version: 7.17.21

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [June 3, 2024, 11:56am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/11 "2024-06-03T11:56:21Z")

</div>

This seems your parsing also breaking. Could you try specifying static index name and template. Just verify whether its parsing issue or not.

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 3, 2024, 12:16pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/12 "2024-06-03T12:16:27Z")

</div>

could you elaborate on that?  
currently I'm starting the filebeat and then executing the below command  
./filebeat test config -c /etc/filebeat/filebeat.yml #checking any config errors  
./filebeat test output -c /etc/filebeat/filebeat.yml #check for connectivity  
./filebeat setup -c /etc/filebeat/filebeat.yml

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 3, 2024, 12:18pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/13 "2024-06-03T12:18:32Z")

</div>

Also, I'm currently executing this on filebeat version 7.17. Not sure if multi indexing is function in this version.

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [June 3, 2024, 2:33pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/14 "2024-06-03T14:33:15Z")

</div>

Your filebeat and other elastic stacks (elasticsearch, kibana etc.) should be same version. I would recommend if you can do fresh installation and just try to read file without changing yaml file. Try to run simple example of [file reading](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html). Once it successfully done, try to change according to your use case.

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 5, 2024, 11:30am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/15 "2024-06-05T11:30:02Z")

</div>

Hi @ashishtiwari1993, I synched my filebeat version to the version of the elastic and ran filebeat with the default configuration with just the changes in the log file path. This seems to work fine for one log file. However when I'm trying to execute the same while adding one more input I'm not getting anything.

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [June 5, 2024, 11:32am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/16 "2024-06-05T11:32:52Z")

</div>

Could you share your multi input configurations?

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 5, 2024, 11:46am UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/18 "2024-06-05T11:46:20Z")

</div>

@ashishtiwari1993 I'm not getting errors in /var/log/filebeat as well. Not sure why the indices are not being created.

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 5, 2024, 12:23pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/19 "2024-06-05T12:23:15Z")

</div>

updated ilm policy setup to false and this seemed to have worked since the log was only being sent to filebeat-7.17.\* without the index name being updated. However, the problem for multi indexing still exists.

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 5, 2024, 1:04pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/20 "2024-06-05T13:04:32Z")

</div>

got it. Thanks for your help and time @ashishtiwari1993 😊

---

<div class="post-metadata">

**Author:** ![developer\_cloud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/developer_cloud/32/134045_2.png) [@developer\_cloud](https://discuss.elastic.co/u/developer_cloud)\
**Post date:** [June 12, 2024, 2:50pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/21 "2024-06-12T14:50:22Z")

</div>

@ashishtiwari1993 , just a quick question, is there a way to attach the multiple indices created to a single ilm policy via filebeat.yml. Currently ilm.setup is set to false. When set to true, it attaches a different index. The indices created via the above filebeat.yml do not get attached to the ilm. Any insights on the same?

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [June 17, 2024, 2:02pm UTC](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817/22 "2024-06-17T14:02:44Z")

</div>

@developer_cloud everything you need to control via ILM policy. Here what worked for me with custom index name -

1. [Create an ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/example-using-index-lifecycle-policy.html)

So I create a

- `test-index-1` assigned policy `test-policy-1`

1. Added index name in filebeat like below -

```auto
output.elasticsearch.index: "test-index-1"
setup.template.name: "filebeat"
setup.template.pattern: "filebeat"

```

Which means it will use all fields from [filebeat template only](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually-alternate).

[Next page](https://discuss.elastic.co/t/multiple-indexes-not-being-created-for-multiple-log-files/358817.md?page=2)