# Pipelines - Losing values with Foreach and Grok **URL:** \ **Category:** Elasticsearch\ **Created:** [November 2, 2018, 7:55pm UTC](https://discuss.elastic.co/t/pipelines-losing-values-with-foreach-and-grok/155234 "2018-11-02T19:55:48Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Josiah\_Raiche](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@Josiah\_Raiche](https://discuss.elastic.co/u/Josiah_Raiche)\ **Post date:** [November 2, 2018, 7:55pm UTC](https://discuss.elastic.co/t/pipelines-losing-values-with-foreach-and-grok/155234/1 "2018-11-02T19:55:48Z")
Similar to this unanswered question: [Ingest: foreach + grok processor - how to write result into an array?](https://discuss.elastic.co/t/ingest-foreach-grok-processor-how-to-write-result-into-an-array/90281) I'm trying to parse a multiline document. Each line is a step in trace route and I'd like to process each line into an array. Here's the relevant field, execbeat's `exec.stdout` for a windows `tracert` command: ```auto Tracing route to example.com [123.456.789.10] over a maximum of 30 hops: 1 24 ms 1 ms 1 ms 123.456.789.14 2 2 ms 1 ms 2 ms example.com [123.456.789.12] 3 5 ms 1 ms 1 ms example.com [123.456.789.10] Trace complete. ``` Desired output, though I'd also be happy with separate top level arrays for each key, or even joined strings. ```auto { ... "hops": [ {"hop_step": 1, "hop_ip": "123.456.789.14", "rtt_1": 24, "rtt_2": 1}, {"hop_step": 2, "hop_ip": "123.456.789.12", "rtt_1": 2, "rtt_2": 1} [... up to 30] ] } ``` Here's my pipeline: ```auto PUT _ingest/pipeline/tracert { "description": "A pipeline for parsing tracert results", "processors": [ { "set": { "field": "hops", "value": "" } }, { "split": { "field": "exec.stdout", "separator": "[\r\n]+" } }, { "foreach": { "field": "exec.stdout", "processor": { "grok": { "field": "_ingest._value", "patterns": ["%{POSINT:hop_step}\\s+%{POSINT:rtt_1} ms\\s+ %{POSINT:rtt_2} ms\\s+ %{POSINT:rtt_3} ms\\s+%{USERNAME:hop_server_name}? \\[?%{IP:hop_ip}"], "ignore_failure" : true } } } } ] } ``` Grok works great and discards the non-matching lines, retaining the ones I like. But since foreach can only accept 1 processor, there's no way to capture those values, and they are overwritten by subsequent lines. This seems like a common case; how can I do this? ---
**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\ **Post date:** [November 5, 2018, 5:36pm UTC](https://discuss.elastic.co/t/pipelines-losing-values-with-foreach-and-grok/155234/2 "2018-11-05T17:36:39Z")
Unfortunately it is not possible with using just Foreach and Grok for the reasons you mention. However, It is possible to do this is in Painless with the script processor. You will have to enable regular expressions in you elasticsearch config : `script.painless.regex.enabled: true` and manually map between Grok and the underlying regex as defined [here](https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/grok-patterns). I took a quick swing, and hopefully this is a starting point to implement something like this (the example as-is is fragile and meant only as getting started example) ```auto POST _ingest/pipeline/_simulate?verbose { "pipeline": { "description": "test", "processors": [ { "script": { "lang": "painless", "source": """ String[] lines = /[\r\n]+/.split(ctx.exec?.stdout); List hops = new ArrayList(); for( String line : lines){ Map hop = new HashMap(); Matcher matcher = /^(\b(?:[1-9][0-9]*)\b)\s+(\b(?:[1-9][0-9]*)\b) ms\s+ (\b(?:[1-9][0-9]*)\b) ms\s+ (\b(?:[1-9][0-9]*)\b) ms\s+([a-zA-Z0-9._-]+)\s+\[(.*)\].*$/.matcher(line); if(matcher.matches()){ if(matcher.groupCount() >= 1){ hop.put("hop", matcher.group(1)); } if(matcher.groupCount() >= 2){ hop.put("rtt_1", matcher.group(2)); } if(matcher.groupCount() >= 3){ hop.put("rtt_2", matcher.group(3)); } if(matcher.groupCount() >= 4){ hop.put("rtt_3", matcher.group(4)); } if(matcher.groupCount() >= 5){ hop.put("user_name", matcher.group(5)); } if(matcher.groupCount() >= 6){ hop.put("ip", matcher.group(6)); } hops.add(hop); } } ctx.hops = hops; """ } } ] }, "docs": [ { "_index": "test", "_type": "_doc", "_id": "99", "_source": { "exec": { "stdout": "Tracing route to example.com [123.456.789.10]\nover a maximum of 30 hops:\n1 24 ms 1 ms 1 ms 123.456.789.14\n2 2 ms 1 ms 2 ms example.com [123.456.789.12]\n3 5 ms 1 ms 1 ms example.com [123.456.789.10]" } } } ] } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 3, 2018, 5:36pm UTC](https://discuss.elastic.co/t/pipelines-losing-values-with-foreach-and-grok/155234/3 "2018-12-03T17:36:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.