# Output to different index based on field
**URL:** https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804
**Category:** Logstash
**Created:** [January 8, 2022, 2:52pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804 "2022-01-08T14:52:16Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@krsecurity](https://discuss.elastic.co/u/krsecurity)
#### Post date: [January 8, 2022, 2:52pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/1 "2022-01-08T14:52:16Z")
Hi there,
I have a couple of use-cases where I want to ingest filebeat module data, send it through Logstash, and then put it into different indexes based on what filebeat module that it used.
So for example, the two I have currently are NetFlow and ThreatIntel -
I want to be able to send NetFlow to index-NetFlow and ThreatIntel to index-ThreatIntel in Elastic/Kibana.
Now i've tried a number of different configs, below is one such example:
```auto
input {
beats {
port => 5044
}
}
filter {
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["localhost:9200"]
index => "index-%{[fields][events.module]}"
```
This results in just naming the index exactly what is there, it doesn't pull the fields from the message at all. I've tried without the [fields] part as well, you can see my attempts in Kibana here:
Can anyone give any pointers on what I'm doing wrong? I managed to pull fields from Metadata but the Metadata fields are the same for Type which is \_doc, so close to useless. I then tried to convert a standard field into a metadata field and then pull it, but that didn't appear to work either.
Thanks
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [January 8, 2022, 4:51pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/2 "2022-01-08T16:51:34Z")
I don't know the way to access filebeat module name from logstash.
How about configuring the logstash output of filebeat as a workaround? According to the reference, you can set `index` parameter and it can be accessed in Logstash's output section as `%{[@metadata][beat]}` .
> **[Configure the Logstash output | Filebeat Reference \[7.16\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#logstash-index)**
---
### Author:  [@krsecurity](https://discuss.elastic.co/u/krsecurity)
#### Post date: [January 8, 2022, 4:54pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/3 "2022-01-08T16:54:28Z")
I'm not as familiar with Filebeat configuration, how do I set the index parameter? I guess I set it as if I was ingesting it directly to Elasticsearch but actually not?
---
### Author:  [@krsecurity](https://discuss.elastic.co/u/krsecurity)
#### Post date: [January 8, 2022, 5:02pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/4 "2022-01-08T17:02:57Z")
Tried this, didn't work:
```auto
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
index: "filebeat-%{event.module}"
```
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [January 8, 2022, 5:05pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/5 "2022-01-08T17:05:53Z")
> I'm not as familiar with Filebeat configuration, how do I set the index parameter?
I recommend to read the reference linked in the previous post carefully.
> I guess I set it as if I was ingesting it directly to Elasticsearch but actually not?
I'm not sure about your filebeat setting (you did not share) but it seems you configured the filebeat to output to logstash because indices were created and documents were indexed. Of course you can configure filebeat to output directly to elasticsearch if you want. Please see the `elasticsearch` part of configurein the output of Filebeat.
> **[Configure the output | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html)**
---
### Author:  [@krsecurity](https://discuss.elastic.co/u/krsecurity)
#### Post date: [January 8, 2022, 5:10pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/6 "2022-01-08T17:10:30Z")
Hi Tomo,
I'm trying to use Logstash where I can for now, simply because I'm more familiar with it.
I've added some additional replies above, it appears all my dynamic references to fields within my filebeat.yml just don't resolve, they all get passed through as literals and my indexes get named %{[fields.event]} etc.
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [January 8, 2022, 5:24pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/7 "2022-01-08T17:24:39Z")
Isn't it just that unresolved dynamic references were treated as literals?
I suppose there are no such fields in Netflow module output.
> **[NetFlow fields | Filebeat Reference \[7.16\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-netflow.html)**
And also you have to check the format of field reference in logstash configuration. `%{[fields][event]}` is the way to access the field `fields.event`.
> **[Accessing event data and fields in the configuration | Logstash Reference...](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf)**
I really recommend you to read these references carefully.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [January 8, 2022, 5:26pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/8 "2022-01-08T17:26:21Z")
If you are using filebeat modules, the parsing and processing that will create the `event` object and other fields from the module will be done using a ingest pipeline in Elasticseach, those fields do not exist in Logstash, that is the reason you are getting the literal `%{field.name}`.
I think that you will need to edit the module configuration and add the fields there, for each module, but I'm not sure as do not use the filebeat modules much.
---
### Author:  [@krsecurity](https://discuss.elastic.co/u/krsecurity)
#### Post date: [January 8, 2022, 5:48pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/9 "2022-01-08T17:48:44Z")
It's certainly a field within the output shown in Kibana:
And I tried [fields][event], same scenario.
---
### Author:  [@krsecurity](https://discuss.elastic.co/u/krsecurity)
#### Post date: [January 8, 2022, 5:51pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/10 "2022-01-08T17:51:25Z")
Also, I don't need a field that exists within NetFlow, I need a field that exists within EVERY Filebeat output...not sure if this is even possible.
Maybe I need to go back to the drawing board but essentially how do I take two filebeat modules and parse them into two separate indexes?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [January 8, 2022, 6:52pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/11 "2022-01-08T18:52:04Z")
I think you are right, your document probably has the `event.module` field already, the issue could be that you are referring to it in a wrong way.
Try the following in your Logstash output:
```auto
index => "index-%{[event][module]}"
```
This is the correct way to make reference to nested fields in logstash, using `event.module` makes a reference to a field with a literal dot in the name.
I'm not sure that this will work because you are not parsing the message in Logstash.
Filebeat modules work better when you send the data directly to Elasticsearch, putting Logstash in between them can be confusing some times as you need to adjuste some things and since you are not using any filter in your Logstash pipeline, why not send it directly to Elasticsearch from filebeat?
You would be able to use something like this to save in different indices:
```auto
output.elasticsearch:
hosts: ["http://localhost:9200"]
index: "index-%{[event.module]}-%{+yyyy.MM.dd}"
```
---
### Author:  [@krsecurity](https://discuss.elastic.co/u/krsecurity)
#### Post date: [January 8, 2022, 11:01pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/12 "2022-01-08T23:01:30Z")
In the end, I gave up trying to use logstash and used this method of achieving what I desired:
```auto
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
indices:
- index: "filebeat-netflow"
when.equals:
event.module: "netflow"
- index: "filebeat-threatintel"
when.equals:
event.module: "threatintel"
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 5, 2022, 11:02pm UTC](https://discuss.elastic.co/t/output-to-different-index-based-on-field/293804/13 "2022-02-05T23:02:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.