# Ingest Lag or Pipeline not working? **URL:** \ **Category:** Logstash\ **Created:** [September 11, 2022, 7:36pm UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148 "2022-09-11T19:36:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ely\_96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ely_96/32/103816_2.png) [@Ely\_96](https://discuss.elastic.co/u/Ely_96)\ **Post date:** [September 11, 2022, 7:36pm UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148/1 "2022-09-11T19:36:29Z")
Hi guys, I have a doubt my pipelines. I have 2 pipelines to see (in Kibana) the most recent requests within a platform. Each request is a json that has a couple of fields and: - req-id - req-timestamp My pipelines: 1. The first pipeline (pipeline\_temp) populates the "temp" index which receives all the documents in realtime; 2. The second pipeline (pipeline\_main) populates the "main" index; it is scheduled to start every 10 minutes, it has input the temp index and for each document it checks that there is not a document with the same req-id with a greater req-timestamp and empties the "temp" index. In my "main" index I currently have about 12 million documents and I see that in the main index there are also req-ids with different req-timestamps (not just the most recent). The loading of these documents seems to be random, the pipeline seems to work correctly 80% of the time but about 20% fails. Could it be a data ingestion delay problem? Maybe the main pipeline checks if there are req-id with more recent req-timestamps, but if the document has not already been ingested the check fails Thanks in advance Ely ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 11, 2022, 8:01pm UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148/2 "2022-09-11T20:01:28Z")
Why fails? Timeout? Is _req-id_ unique value? 12 mil records in total in main index? How many docs usually has temp index? Are you using ILM for temp? ---
**Author:** ![Ely\_96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ely_96/32/103816_2.png) [@Ely\_96](https://discuss.elastic.co/u/Ely_96)\ **Post date:** [September 11, 2022, 8:23pm UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148/3 "2022-09-11T20:23:23Z")
Hi Rios, Thanks a lot for your answer. I dont know why fails... this is the purpose of my topic 🙂 Yes, req-id is unique value; main index increase every 10 minutes, when the pipeline main runs.. but now I have 12 mil docs. And no, I'm not using ILM (elasticsearch use the default value). Thanks!! ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 11, 2022, 8:49pm UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148/4 "2022-09-11T20:49:05Z")
Is there any error in /var/log/logstash/logstash-plain.log? With temp index, you try to avoid duplicated records based on unique req-ids? If req-ids=12345 and req-timestamps='10092022' in index, and temp index get req-ids=12345 and newer req-timestamps='11092022' , will be update of full record for req-ids=12345 or just req-timestamps in main index? ---
**Author:** ![Ely\_96](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ely_96/32/103816_2.png) [@Ely\_96](https://discuss.elastic.co/u/Ely_96)\ **Post date:** [September 12, 2022, 6:46am UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148/5 "2022-09-12T06:46:44Z")
Hi Rios, I just asked to have access to that lo ... let's see as soon as I obtain it. What could it contain? In the temp index I load (and gradually empty) everything that arrives. I clean by timestamps only occurs in the main index ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 12, 2022, 1:02pm UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148/6 "2022-09-12T13:02:31Z")
Search for error or timeout. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 10, 2022, 1:03pm UTC](https://discuss.elastic.co/t/ingest-lag-or-pipeline-not-working/314148/7 "2022-10-10T13:03:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.