# Logstash pipeline filter by date

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-filter-by-date/86969>\
**Category:** Logstash\
**Created:** [May 24, 2017, 12:16pm UTC](https://discuss.elastic.co/t/logstash-pipeline-filter-by-date/86969 "2017-05-24T12:16:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ezequielo](https://avatars.discourse-cdn.com/v4/letter/e/b2d939/32.png) [@ezequielo](https://discuss.elastic.co/u/ezequielo)\
**Post date:** [May 24, 2017, 12:16pm UTC](https://discuss.elastic.co/t/logstash-pipeline-filter-by-date/86969/1 "2017-05-24T12:16:07Z")

</div>

Hi all,

I have created a logstash pipeline that fetches data from a database (MySQL) and load it into ES.  
So it basically consists of a select over a couple of tables and output it to ES. I have noticed that lately this is taking longer and longer so I guess it's fetching all rows and trying to insert new and old entries into ES.

If that's true I need to find a mechanism to avoid fetching entries that are already in ES. What's the most usual pattern for this? I have been thinking about two alternatives:

- New column in database table to indicate whether it has been loaded in ES or not

- Keep a datetime in ES or my database in order to fetch this value and use it in the input query

Im not sure about any of those because they make assumptions or modify the stored data. What do you think? Do you have any other alternative?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 28, 2017, 6:53pm UTC](https://discuss.elastic.co/t/logstash-pipeline-filter-by-date/86969/2 "2017-05-28T18:53:50Z")

</div>

The second option is a good one. If you maintain a column with the last modified time of the row Logstash can efficiently ask queries that return only rows modified since the last time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 25, 2017, 7:01pm UTC](https://discuss.elastic.co/t/logstash-pipeline-filter-by-date/86969/3 "2017-06-25T19:01:05Z")

</div>

Closed on June 25, 2017, 7:01pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.