# Create multiple indexes while pushing data from filebeat to elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [September 14, 2018, 4:09pm UTC](https://discuss.elastic.co/t/create-multiple-indexes-while-pushing-data-from-filebeat-to-elasticsearch/148649 "2018-09-14T16:09:38Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![souravtyson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/souravtyson/32/45427_2.png) [@souravtyson](https://discuss.elastic.co/u/souravtyson)\ **Post date:** [September 14, 2018, 4:09pm UTC](https://discuss.elastic.co/t/create-multiple-indexes-while-pushing-data-from-filebeat-to-elasticsearch/148649/1 "2018-09-14T16:09:39Z")
Hi team, I am trying to push the nginx logs using filebeat to elasticsearch. I wanted to created multiple indexes based on data logged in my file. For example : i have a field of response time that gets logged into the log file. But for some cases the value comes as "-". So what i want to do is push the data to two different indexes of elasticsearch based on the field value. I have gone through the documentation and question blog and had found something useful and tried to do the same. Here is a link [https://discuss.elastic.co/t/set-indices-based-on-a-field-value/69579](https://discuss.elastic.co/t/set-indices-based-on-a-field-value/69579) and [https://www.elastic.co/guide/en/beats/filebeat/5.3/elasticsearch-output.html#\_indices](https://www.elastic.co/guide/en/beats/filebeat/5.3/elasticsearch-output.html#_indices). My configuration is shown here : ``` output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] index: "nginx-access-%{[beat.version]}-%{+yyyy.MM.dd}" indices: - index: "abc-%{[beat.version]}-%{+yyyy.MM.dd}" when.equals: nginx.access.upstr_resp_time: "-" setup.template.settings: index.number_of_shards: 3 #index.codec: best_compression #_source.enabled: false setup.template.name: "nginx-access" setup.template.pattern: "nginx-access-*" setup.template.name: "abc" setup.template.pattern: "abc-*" setup.template.enable: true setup.template.overwrite: false ``` As you can see i wanted to push the data to index nginx-access-\* by default and if the response time have "-" will send to abc-\*. In my log file their are 14 data. Now i want to see all those data in my kibana. So when i click on discover i can see only one index being displayed. Is their something wrong i am doing. Please help me with the issue Note ☹ The index shown on kibana show 13 data and the another 1 data that is left over have response time value as "\_" which is not shown on kibana. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2018, 5:21am UTC](https://discuss.elastic.co/t/create-multiple-indexes-while-pushing-data-from-filebeat-to-elasticsearch/148649/2 "2018-09-16T05:21:44Z")
[Filebeat modules use ingest pipelines](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html) to parse the data, and these are run in Elasticsearch. The fields you are looking to filter on in the Elasticsearch output does therefore not exist at that point, which is why it fails to do what you are anticipating. To do what you want to do I suspect you may need to modify the ingest pipeline used to change the index name based the contents of the fields you mentioned. ---
**Author:** ![souravtyson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/souravtyson/32/45427_2.png) [@souravtyson](https://discuss.elastic.co/u/souravtyson)\ **Post date:** [September 16, 2018, 8:41am UTC](https://discuss.elastic.co/t/create-multiple-indexes-while-pushing-data-from-filebeat-to-elasticsearch/148649/3 "2018-09-16T08:41:16Z")
can you give one example of how i can do it ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 16, 2018, 3:53pm UTC](https://discuss.elastic.co/t/create-multiple-indexes-while-pushing-data-from-filebeat-to-elasticsearch/148649/4 "2018-09-16T15:53:04Z")
I do unfortunately not have any suitable example handy. Why not simply delete the response time field if it is set to `-` and then send it all to the same index? ---
**Author:** ![souravtyson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/souravtyson/32/45427_2.png) [@souravtyson](https://discuss.elastic.co/u/souravtyson)\ **Post date:** [September 17, 2018, 11:02am UTC](https://discuss.elastic.co/t/create-multiple-indexes-while-pushing-data-from-filebeat-to-elasticsearch/148649/5 "2018-09-17T11:02:04Z")
actually i need to push the data so that i can see what had happened to the those kind of request. So i tried one more option ``` index: "nginx-access-%{[beat.version]}-%{+yyyy.MM.dd}" indices: - index: "abc-%{[beat.version]}-%{+yyyy.MM.dd}" when.contains: message: "list" ``` this created me two indexes. so why does message works? Is it that the message exist at that point in time ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 15, 2018, 11:02am UTC](https://discuss.elastic.co/t/create-multiple-indexes-while-pushing-data-from-filebeat-to-elasticsearch/148649/6 "2018-10-15T11:02:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.