# Read one file, make two filters and send for two different indexes **URL:** \ **Category:** Logstash\ **Created:** [June 2, 2018, 4:56pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291 "2018-06-02T16:56:22Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![paulobezerr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paulobezerr/32/24080_2.png) [@paulobezerr](https://discuss.elastic.co/u/paulobezerr)\ **Post date:** [June 2, 2018, 4:56pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/1 "2018-06-02T16:56:22Z")
I have a big log CSV file, receiving logs from different equipments. My logstash read the lines and send to a daily index. I need to keep the last equipment report in a different index. Too many slow visualizations aggregate top hit to get this value, but I think that I don't need to do this. My plan is: import logs normally as today with logstash-\* create a second index 'last\_report', where id is equipment\_id. So, every new line actually will be a update on the last\_report index. Is this possible with logstash, or did I need to think in a different way? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 2, 2018, 5:19pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/2 "2018-06-02T17:19:08Z")
That is a quite common method to make sure the latest state can be retrieved efficiently. It should be fine doing that with Logstash, at least as long as you do not have documents being updated very frequently. ---
**Author:** ![paulobezerr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paulobezerr/32/24080_2.png) [@paulobezerr](https://discuss.elastic.co/u/paulobezerr)\ **Post date:** [June 2, 2018, 5:21pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/3 "2018-06-02T17:21:10Z")
Hi, thank you for your answer! So, can I just create another pipeline to look at the same file at the same time? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 2, 2018, 5:21pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/4 "2018-06-02T17:21:48Z")
You can have two elasticsearch outputs in the same pipeline. ---
**Author:** ![paulobezerr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paulobezerr/32/24080_2.png) [@paulobezerr](https://discuss.elastic.co/u/paulobezerr)\ **Post date:** [June 2, 2018, 5:23pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/5 "2018-06-02T17:23:21Z")
But my problem is, one of them, I have a generated id (default), the other one, must be equipment\_id, so next lines will just update old documents or insert a new one. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 2, 2018, 5:26pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/6 "2018-06-02T17:26:25Z")
How frequently do you expect a single document in the new index to be updated? ---
**Author:** ![paulobezerr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paulobezerr/32/24080_2.png) [@paulobezerr](https://discuss.elastic.co/u/paulobezerr)\ **Post date:** [June 2, 2018, 5:28pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/7 "2018-06-02T17:28:54Z")
At least 1 update per hour. Some times, we force a request in equipment and can happen before, but is not usual. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 2, 2018, 5:37pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/8 "2018-06-02T17:37:31Z")
That is not very frequent so should not be a problem. I would recommend creating two elasticsearch output plugins, one to write the documents with auto-generated id into the existing index and one to write it using equipment\_id to the new index. This will result in an insert the first time and an update every time after that. ---
**Author:** ![paulobezerr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paulobezerr/32/24080_2.png) [@paulobezerr](https://discuss.elastic.co/u/paulobezerr)\ **Post date:** [June 2, 2018, 6:30pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/9 "2018-06-02T18:30:21Z")
It works. Thank you very much! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 30, 2018, 6:30pm UTC](https://discuss.elastic.co/t/read-one-file-make-two-filters-and-send-for-two-different-indexes/134291/10 "2018-06-30T18:30:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.