# Is it possible? **URL:** \ **Category:** Elasticsearch\ **Tags:** painless, ingest-pipeline\ **Created:** [August 12, 2021, 11:50am UTC](https://discuss.elastic.co/t/is-it-possible/281197 "2021-08-12T11:50:14Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ailengcon](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@ailengcon](https://discuss.elastic.co/u/ailengcon)\ **Post date:** [August 12, 2021, 11:50am UTC](https://discuss.elastic.co/t/is-it-possible/281197/1 "2021-08-12T11:50:14Z")
Can someone answer me this. I have a database with several configurations, and each configuration can be linked to the same serial\_number. To be able to analyze the data correctly, I need to filter så that each serial\_number is grouped as one configuration. So can I use an ingest pipeline to set up condition to say " if there is some document here with the same serial\_number, only ingest the newest one of each"? Is anything possible with these processors and the painless language? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 16, 2021, 4:57am UTC](https://discuss.elastic.co/t/is-it-possible/281197/2 "2021-08-16T04:57:30Z")
Can you elaborate more on why you only want the latest one stored? ---
**Author:** ![ailengcon](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@ailengcon](https://discuss.elastic.co/u/ailengcon)\ **Post date:** [August 16, 2021, 5:42am UTC](https://discuss.elastic.co/t/is-it-possible/281197/3 "2021-08-16T05:42:11Z")
the configurations get stored in the database when a user press save in the program, and if the user presses save multiple times the same configuration get's stored again. This can happen if a user is inexperienced or forgets a setting before saving. So it's a incorrect design of the system I guess, they admit this, but to get accurate statistics from the database they want to filter the redundant data out. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 16, 2021, 5:45am UTC](https://discuss.elastic.co/t/is-it-possible/281197/4 "2021-08-16T05:45:04Z")
You have two options; 1. Keep all state changes and only provide the latest, using a top hits aggregation. This lets you track changes over time and do analysis on them 2. Use the serial as a document ID and then it'll overwrite with the latest each time ---
**Author:** ![ailengcon](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@ailengcon](https://discuss.elastic.co/u/ailengcon)\ **Post date:** [August 16, 2021, 5:58am UTC](https://discuss.elastic.co/t/is-it-possible/281197/5 "2021-08-16T05:58:04Z")
Thank you very much, I don't understand what state changes are, but at least I know it is possible. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 16, 2021, 5:58am UTC](https://discuss.elastic.co/t/is-it-possible/281197/6 "2021-08-16T05:58:37Z")
State change means every time the config is changed, ie it goes from one state to another. ---
**Author:** ![ailengcon](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@ailengcon](https://discuss.elastic.co/u/ailengcon)\ **Post date:** [September 6, 2021, 7:17am UTC](https://discuss.elastic.co/t/is-it-possible/281197/7 "2021-09-06T07:17:18Z")
Hi, can I ask you to elaborate on the first option more? I got hired to implement elasticsearch so I need more information on how to move forward. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 6, 2021, 8:27am UTC](https://discuss.elastic.co/t/is-it-possible/281197/8 "2021-09-06T08:27:48Z")
Elaborate in what sense? ---
**Author:** ![ailengcon](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@ailengcon](https://discuss.elastic.co/u/ailengcon)\ **Post date:** [September 6, 2021, 9:00am UTC](https://discuss.elastic.co/t/is-it-possible/281197/9 "2021-09-06T09:00:04Z")
I'm not sure if I understand the concept of state changes yet. There are several configurations with the same serial\_number, how would their state change? And would you do still do this with ingest pipelines? What processor? I guess it is not possible to both keep all documents and also use just one of each serial number to analyze statistics? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 6, 2021, 11:07pm UTC](https://discuss.elastic.co/t/is-it-possible/281197/10 "2021-09-06T23:07:33Z")
Basically you want to use Elasticsearch as a time series datastore. Where each event comes in with a timestamp and serial and whatever else is logged. Then you can graph changes over time, either at an individual serial level, or on an aggregated level. If you want to retrieve only the latest event, which contains whatever state is logged, then you can do that easily. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2021, 11:08pm UTC](https://discuss.elastic.co/t/is-it-possible/281197/11 "2021-10-04T23:08:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.