# Multiple Log Paths With Multiple Index **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 21, 2024, 6:58am UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253 "2024-08-21T06:58:49Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![oseker](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oseker](https://discuss.elastic.co/u/oseker)\ **Post date:** [August 21, 2024, 6:58am UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/1 "2024-08-21T06:58:49Z")
Many log files are stored on the system in various locations. I'm attempting to use filestream input type to read log files and then send them straight to Elasticsearch. "Error loading template: error creating template instance: key not found" is what I'm seeing when I try to create an index for every log path on Kibana. Whats wrong with the configuration? Filebeat version is 8.14.3 ```auto filebeat.inputs: - type: filestream id: smx json.keys_under_root: true json.add_error_key: true fields: custom-index: smx scan_frequency: 30s harvester_limit: 100 close.on_state_change.inactive: 30m close.on_state_change.removed: true clean_removed: true encoding: utf-8 paths: - /home/app/smx/logs/app-info.log* - type: filestream id: client json.keys_under_root: true json.add_error_key: true fields: custom-index: client scan_frequency: 30s harvester_limit: 100 close.on_state_change.inactive: 30m close.on_state_change.removed: true clean_removed: true encoding: utf-8 paths: - /home/app/client/log/app-info.log* - type: filestream id: anotherapp json.keys_under_root: true json.add_error_key: true fields: custom-index: anotherapp scan_frequency: 30s harvester_limit: 100 close.on_state_change.inactive: 30m close.on_state_change.removed: true clean_removed: true encoding: utf-8 paths: - /home/app/anotherapp/log/app-info.log* output.elasticsearch: hosts: ["http://myelastic"] username: "user" password: 'pass' index: "%{[fields.custom-index]}" setup.template.enabled: true setup.template.overwrite: true setup.template.name: "%{[fields.custom-index]}" setup.template.pattern: "%{[fields.custom-index]}-*" setup.ilm.enabled: true setup.ilm.policy_name: "7-days@lifecycle" setup.ilm.rollover_alias: "%{[fields.custom-index]}" ``` ---
**Author:** ![Trevor\_Blackford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trevor_blackford/32/120087_2.png) [@Trevor\_Blackford](https://discuss.elastic.co/u/Trevor_Blackford)\ **Post date:** [August 21, 2024, 1:12pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/2 "2024-08-21T13:12:04Z")
This might be because the setup phase for the template is prior to events being processed. Since the filestream id appears to be equivalent to fields.custom-index you might try referring to that instead. ---
**Author:** ![oseker](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oseker](https://discuss.elastic.co/u/oseker)\ **Post date:** [August 21, 2024, 1:32pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/3 "2024-08-21T13:32:15Z")
Can you please show an example how to refer filestream id? ---
**Author:** ![Trevor\_Blackford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trevor_blackford/32/120087_2.png) [@Trevor\_Blackford](https://discuss.elastic.co/u/Trevor_Blackford)\ **Post date:** [August 21, 2024, 2:55pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/4 "2024-08-21T14:55:18Z")
There's nothing wrong with the fields.custom\_index reference. I think the problem is in the template setup command, and the availability of the key. I believe the setup template command must run prior to the evaluation of any data. Does your data differ significantly in which you want to map the available fields differently? Do you really want to apply different templates? If so, you may be able to refer to those templates statically and sequentially. If not, you could create a pattern that applies to all indices, such as filestream-\* ---
**Author:** ![oseker](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oseker](https://discuss.elastic.co/u/oseker)\ **Post date:** [August 22, 2024, 1:46pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/5 "2024-08-22T13:46:17Z")
Thanks for the answer. I need to apply different templates. I couldnt find any example about referring templates statically. Can you please point me to an example? Single index pattern works but, yeah, for every filestream input type an index should be created. ---
**Author:** ![Trevor\_Blackford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trevor_blackford/32/120087_2.png) [@Trevor\_Blackford](https://discuss.elastic.co/u/Trevor_Blackford)\ **Post date:** [August 22, 2024, 7:23pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/6 "2024-08-22T19:23:52Z")
A different index can still be created with the output.elasticsearch index parameter. That should stay dynamic. I'm wondering if you need different templates for each index. The main reason to have different templates if you need to map the same field names to different types. But the managed templates Elastic provides do a lot of the heavy lifting for you, and your have setup.template.enabled: true ---
**Author:** ![oseker](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oseker](https://discuss.elastic.co/u/oseker)\ **Post date:** [August 23, 2024, 11:47am UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/7 "2024-08-23T11:47:18Z")
Managed templates work well for us as well. I modified the lines, but no luck. ```auto output.elasticsearch: hosts: ["https://myelastic"] username: "user" password: 'pass' index: "%{[inputs.id]}-%{yyyy.MM.dd}" setup.template.enabled: false setup.ilm.enabled: true setup.ilm.policy_name: "7-days@lifecycle" ``` ---
**Author:** ![Trevor\_Blackford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trevor_blackford/32/120087_2.png) [@Trevor\_Blackford](https://discuss.elastic.co/u/Trevor_Blackford)\ **Post date:** [August 23, 2024, 3:53pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/8 "2024-08-23T15:53:58Z")
Could you do? ```auto index: "logs-%{[fields.custom_index]}-%{yyyy.MM.dd}" ``` Starting with logs\* will use the existing stack managed index template > **[Index templates | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html)** In the output the fields.custom\_index value will be available. ---
**Author:** ![oseker](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oseker](https://discuss.elastic.co/u/oseker)\ **Post date:** [August 23, 2024, 5:08pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/9 "2024-08-23T17:08:38Z")
Unfortunately, I encountered another error. I apologize for misdirecting you. Custom fields are not necessary for input types. To utilize as dynamic index naming, I add them. In my situation, fields: custom-index: is entirely unnecessary. There were no examples of using the input type id value that I could locate. I simply need different index names for different log files with lifecycle policy ofcourse. ```auto Exiting: error initializing publisher: unsupported format expression "yyyy.MM.dd" in index ``` ---
**Author:** ![oseker](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oseker](https://discuss.elastic.co/u/oseker)\ **Post date:** [August 27, 2024, 3:42pm UTC](https://discuss.elastic.co/t/multiple-log-paths-with-multiple-index/365253/10 "2024-08-27T15:42:48Z")
> [@Trevor\_Blackford](#): > > `index: "logs-%{[fields.custom_index]}-%{yyyy.MM.dd}"` There must be a plus sign before date format. ```auto index: "logs-%{[fields.custom_index]}-%{+yyyy.MM.dd}" ``` But after this change error changed to: ```auto "failed to publish events: temporary bulk send failure","service.name":"filebeat" ``` Pffff too many frustrations for a configuration.