# Tracking down which pipelines fails to parse **URL:** \ **Category:** Kibana\ **Created:** [May 20, 2019, 8:11am UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763 "2019-05-20T08:11:42Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [May 20, 2019, 8:11am UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/1 "2019-05-20T08:11:42Z")
I underwent a pretty bumpy upgrade process behind me and now that the stack stabilized, I noticed that for some pipelines there were no data parsed. As a lot of information is parsed and I've got a dozen pipelines, it is not easy for me to find out which pipelines failed to parse the data or simply didn't receive any. Is there any solution in ES Stack to monitor grok parsing failure? Like creating a Dashboard or a feature I'm not thinking about? ---
**Author:** ![Joe\_Fleming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_fleming/32/3561_2.png) [@Joe\_Fleming](https://discuss.elastic.co/u/Joe_Fleming)\ **Post date:** [May 20, 2019, 7:07pm UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/2 "2019-05-20T19:07:42Z")
The [node stats API](https://www.elastic.co/guide/en/elasticsearch/reference/master/cluster-nodes-stats.html#ingest-stats) can be used to fetch ingest usage statistics, globally and on a per pipeline basis. That will at least help you figure out which pipelines are failing. There are also several ways to [handle failures in pipelines](https://www.elastic.co/guide/en/elasticsearch/reference/master/handling-failure-in-pipelines.html), are you doing anything with the failures or just discarding the documents/outcome? ---
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [May 21, 2019, 7:25am UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/3 "2019-05-21T07:25:36Z")
> [@Joe\_Fleming](#): > > are you doing anything with the failures or just discarding the documents/outcome? Currently I'm doing nothing with the failures. ---
**Author:** ![Joe\_Fleming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_fleming/32/3561_2.png) [@Joe\_Fleming](https://discuss.elastic.co/u/Joe_Fleming)\ **Post date:** [May 21, 2019, 5:22pm UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/4 "2019-05-21T17:22:20Z")
It's a really good idea to do _something_ with failures, otherwise the failed documents just don't get ingested. Maybe you don't care much if you lose data, but at the very least recording some failure metrics would be helpful (though, as noted, you can get basic metrics from the node stats API). ---
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [May 23, 2019, 7:24am UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/5 "2019-05-23T07:24:08Z")
Let's say I would like to store the failure in its own indexes using a failure pipeline storing message and original pipeline. Is it possible? Is it a good idea? ---
**Author:** ![Joe\_Fleming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_fleming/32/3561_2.png) [@Joe\_Fleming](https://discuss.elastic.co/u/Joe_Fleming)\ **Post date:** [May 29, 2019, 5:33pm UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/6 "2019-05-29T17:33:08Z")
> Is it a good idea? I certainly think so. Otherwise the documents don't get indexed, and if you're using something like filebeat, the indexing will fail and filebeat will stop sending logs in while it waits for the index to succeed (and it never will). > Let's say I would like to store the failure in its own indexes using a failure pipeline storing message and original pipeline. Is it possible? Certainly, and personally that's the route I'd go. If I were in your situation, I would basically just keep track of the input and the resulting error in a new index, so that I could go back and retry it later. And once you know the failing input, you can use the [Simulate API](https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html) to tweak things until it doesn't generate an error anymore. The docs have a whole section on [handling errors](https://www.elastic.co/guide/en/elasticsearch/reference/master/handling-failure-in-pipelines.html) using the `on_failure` parameter. I'm no expert in ingest pipelines, but my understanding is that you can handle errors in 2 ways, per-process and as a kind of global catch-all. I believe that works pretty much like any other processor, meaning you can use painless to modify or append things to the original document and do whatever you need. It's probably also worth noting that Elasticsearch's object type fields support an [enabled setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html), and when set to false, the input won't be mapped to any fields, it just stores it in the `_source`. With this in mind, my naive approach would be to have 2 disabled fields, one for the original input and one for the resulting error object, and `on_failure` I'd just shove both pieces into a new document in some kind of failure index and come back to it later. There's probably better options, and I'm sure if I read the "Handling failures in Pipelines" docs, I might come up with a better plan 😉 . But the takeaway here is that by handling errors you stop the pipeline from stoping ingestion, and by keeping track of the failures you can investigate the issue and fix the problem (either in the pipeline itself or at the source of the data). ---
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [June 7, 2019, 8:14am UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/7 "2019-06-07T08:14:31Z")
Well for the time being, I can't either create or change pipeline because [I'm getting timeout](https://discuss.elastic.co/t/es-6-x-timeout-by-index-lifetime-policies-creation/180773). ---
**Author:** ![EldrosKandar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eldroskandar/32/33188_2.png) [@EldrosKandar](https://discuss.elastic.co/u/EldrosKandar)\ **Post date:** [July 4, 2019, 1:59pm UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/8 "2019-07-04T13:59:01Z")
A maintenance is planned soon, and I will had the on\_failure option to the grok processor directly through a script. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 1, 2019, 1:59pm UTC](https://discuss.elastic.co/t/tracking-down-which-pipelines-fails-to-parse/181763/9 "2019-08-01T13:59:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.