# Multiple indices from one log file

**URL:** <https://discuss.elastic.co/t/multiple-indices-from-one-log-file/315290>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 27, 2022, 4:27pm UTC](https://discuss.elastic.co/t/multiple-indices-from-one-log-file/315290 "2022-09-27T16:27:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kkovacs](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kkovacs](https://discuss.elastic.co/u/kkovacs)\
**Post date:** [September 27, 2022, 4:27pm UTC](https://discuss.elastic.co/t/multiple-indices-from-one-log-file/315290/1 "2022-09-27T16:27:09Z")

</div>

Dear Community!

I have a question regarding filebeat configuration. **Is there any method to make two separate index from the same log file?** My purpose is to make two separate index for Kibana from the same log file - to use one index to search for full logs (for debugging) and use the other index for reporting tasks (in this case I would use processors to dissect and remove the unnecessary fields from the log)

Something like this:

```auto
filebeat.inputs:
  - type: log
    enabled: true
    fields:
      log_type: log-1
    paths:
      - path/to/logfile.log
  - type: log
    enabled: true
    fields:
      log_type: log-2
    paths:
      - path/to/logfile.log

```

I know that the log file "paths" can't be the same (as seen above), this is why I interested in what can be an alternative solution.

I am using filebeat version 7.12, and Kibana 7.6.2.

---

<div class="post-metadata">

**Author:** ![grfneto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grfneto/32/125776_2.png) [@grfneto](https://discuss.elastic.co/u/grfneto)\
**Post date:** [September 29, 2022, 2:28am UTC](https://discuss.elastic.co/t/multiple-indices-from-one-log-file/315290/2 "2022-09-29T02:28:16Z")

</div>

Hi @kkovacs

Maybe in logstash you have more flexibility to create one or multiple pipelines, where you can manipulate the log the way you want and also have the log in its raw form.

[Multiple Pipelines | Logstash Reference [8.4] | Elastic](https://www.elastic.co/guide/en/logstash/8.4/multiple-pipelines.html#multiple-pipelines)

best regards

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 29, 2022, 3:04am UTC](https://discuss.elastic.co/t/multiple-indices-from-one-log-file/315290/3 "2022-09-29T03:04:09Z")

</div>

What is your output?

If your output is Elasticsearch you may be able to use the `indices` configuration.

The [documentation](https://www.elastic.co/guide/en/beats/filebeat/7.12/elasticsearch-output.html#indices-option-es) has a couple of examples that may fit your use case.

If your output is Logstash, then you should do that in Logstash, which is way easier.

---

<div class="post-metadata">

**Author:** ![kkovacs](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@kkovacs](https://discuss.elastic.co/u/kkovacs)\
**Post date:** [September 29, 2022, 6:38am UTC](https://discuss.elastic.co/t/multiple-indices-from-one-log-file/315290/4 "2022-09-29T06:38:25Z")

</div>

First of all: Thank you guys for the answers.

I do not use Logstash right now. I am trying to solve the problem without using it. My output is Elasticsearch so I am going to check the linked documentation for possible solution.

This is my output right now:

```auto
output:
  elasticsearch:
    hosts: [...ip address...]
    index: "%{[fields.log_type]}-%{+yyyy.MM.dd}"

```

Thanks again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 27, 2022, 8:38am UTC](https://discuss.elastic.co/t/multiple-indices-from-one-log-file/315290/5 "2022-10-27T08:38:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.