# Pipeline Grok Expressions

**URL:** <https://discuss.elastic.co/t/pipeline-grok-expressions/169174>\
**Category:** Elasticsearch\
**Created:** [February 20, 2019, 9:55am UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174 "2019-02-20T09:55:26Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sharad\_Ananth](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)\
**Post date:** [February 20, 2019, 9:55am UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/1 "2019-02-20T09:55:26Z")

</div>

I have a logline :  
`172.17.60.26 - - [14/Feb/2019:01:50:47 -0000] "GET http://live-paytv.mobitv.com/mm/dash/live/19033/LIVESERVICE_2403/V5000_W/258351504.m4s http/1.1" 200 3696282 200 3696282 0 0 591 579 674 566 1.695 1.598 DIRECT FIN FIN TCP_MISS "MOBI_EXO2Player;Dalvik/2.1.0 (Linux; U; Android 5.1.1; AFTT Build/LVY48F)" 133ba155-2f5a-4c64-8426-465286558c46`

I am using the following grok expression :

```
{
      "grok": {
        "field": "message",
        "patterns": ["""%{IP:source_ip} %{GREEDYDATA} \[%{HTTPDATE:request_date}\] \"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} \"%{GREEDYDATA:user_agent}\" %{GREEDYDATA:x_play_back_session_id}""",
        """%{IP:source_ip} %{GREEDYDATA} \[%{HTTPDATE:request_date}\] \"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent:int} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} %{GREEDYDATA:user_agent}
        """]
      }
    } 

```

When I use this grok processor using PUT PIPELINE API in Kibana Dev Console, this works fine. But when I put the grok processor in a json file and trigger it using the command line :"curl -H 'Content-Type: application/json' -X PUT '[http://localhost:9200/\_ingest/pipeline/trial-pipeline](http://localhost:9200/_ingest/pipeline/trial-pipeline)' -d@pipeline.json", it gives an error

{"error":{"root\_cause":[{"type":"parse\_exception","reason":"Failed to parse content to map"}],"type":"parse\_exception","reason":"Failed to parse content to map","caused\_by":{"type":"json\_parse\_exception","reason":"Unrecognized character escape '[' (code 91)\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@3ae9f3ad; line: 1, column: 170]"}},"status":400}.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 20, 2019, 11:05am UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/2 "2019-02-20T11:05:22Z")

</div>

Could you try to replace `"""` by `"` and every internal `"` by `\"`?

---

<div class="post-metadata">

**Author:** ![Sharad\_Ananth](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)\
**Post date:** [February 20, 2019, 11:39am UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/3 "2019-02-20T11:39:23Z")

</div>

Hey David, sorry about the topic. I forgot to put the json in code format. I have edited it. You can see the actual code in the topic now.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 20, 2019, 2:37pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/5 "2019-02-20T14:37:52Z")

</div>

Could you share your full `pipeline.json` file that reproduces this problem so we can start from it to reproduce?

---

<div class="post-metadata">

**Author:** ![Sharad\_Ananth](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)\
**Post date:** [February 20, 2019, 3:09pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/7 "2019-02-20T15:09:34Z")

</div>

This is the pipeline.json file.

```
{
  "description" : "Pipeline for ingest node",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:source_ip} %{GREEDYDATA} \[%{HTTPDATE:request_date}\] \"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} \"%{GREEDYDATA:user_agent}\" %{GREEDYDATA:x_play_back_session_id}",
        "%{IP:source_ip} %{GREEDYDATA} \[%{HTTPDATE:request_date}\] \"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent:int} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} %{GREEDYDATA:user_agent}"]
      }
    },
    
    {
      "convert" : {
        "field" : "bytes_sent",
        "type": "integer"
      }
    },
    {
          "dissect": {
            "field": "uri_path",
           "if":"(ctx.uri_path.contains('hls5') && ctx.uri_path.contains('live') && (ctx.uri_path.contains('m3u8') || ctx.uri_path.contains('ts'))) || (ctx.uri_path.contains('dash') && ctx.uri_path.contains('live') && ctx.uri_path.contains('m4s'))",
            "pattern": "/%{a}/%{protocol}/%{stream_type}/%{backend_channel_id}/%{e}/%{variant}/%{g}.%{h}"
          }
        },
        
        
        
        {
  "remove": {
    "field": ["a","e","g","h"]
  }
}      
 ]
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 20, 2019, 3:33pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/8 "2019-02-20T15:33:06Z")

</div>

Try with:

```auto
{
  "description": "Pipeline for ingest node",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{IP:source_ip} %{GREEDYDATA} \\[%{HTTPDATE:request_date}\\] \\\"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\\\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} \\\"%{GREEDYDATA:user_agent}\\\" %{GREEDYDATA:x_play_back_session_id}",
          "%{IP:source_ip} %{GREEDYDATA} \\[%{HTTPDATE:request_date}\\] \\\"%{WORD:http_method} %{URIPROTO:http_proto}://%{URIHOST:uri_host}%{URIPATH:uri_path}%{GREEDYDATA:uri_query} http/%{NUMBER:http_version}\\\" %{NUMBER:response_code} %{NUMBER:bytes_sent:int} %{NUMBER:origin_response_code} %{NUMBER:origin_bytes_sent:int} %{NUMBER:client_req_content_length} %{NUMBER:proxy_req_length} %{NUMBER:client_req_header_length} %{NUMBER:proxy_resp_header_length} %{NUMBER:proxy_req_header_length} %{NUMBER:origin_header_resp_length} %{NUMBER:time_to_serve:} %{NUMBER:origin_time_to_serve:} %{WORD:proxy_hierarchy_route} %{WORD:finish_status_client} %{WORD:finish_status_origin} %{WORD:cache_result_code} %{GREEDYDATA:user_agent}"
        ]
      }
    },
    {
      "convert": {
        "field": "bytes_sent",
        "type": "integer"
      }
    },
    {
      "dissect": {
        "field": "uri_path",
        "if": "(ctx.uri_path.contains(\"hls5\") && ctx.uri_path.contains(\"live\") && (ctx.uri_path.contains(\"m3u8\") || ctx.uri_path.contains(\"ts\"))) || (ctx.uri_path.contains(\"dash\") && ctx.uri_path.contains(\"live\") && ctx.uri_path.contains(\"m4s\"))",
        "pattern": "/%{a}/%{protocol}/%{stream_type}/%{backend_channel_id}/%{e}/%{variant}/%{g}.%{h}"
      }
    },
    {
      "remove": {
        "field": [
          "a",
          "e",
          "g",
          "h"
        ]
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Sharad\_Ananth](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)\
**Post date:** [February 20, 2019, 4:51pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/9 "2019-02-20T16:51:54Z")

</div>

The grok passed, but I am seeing this error:

`org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [a] not present as part of path [a]`

---

<div class="post-metadata">

**Author:** ![Sharad\_Ananth](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)\
**Post date:** [February 20, 2019, 5:26pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/10 "2019-02-20T17:26:02Z")

</div>

Also, seeing this error:

`ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure`

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 20, 2019, 6:34pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/11 "2019-02-20T18:34:05Z")

</div>

That's other questions. I' d open another question as the original one is now solved.

---

<div class="post-metadata">

**Author:** ![Sharad\_Ananth](https://avatars.discourse-cdn.com/v4/letter/s/e5b9ba/32.png) [@Sharad\_Ananth](https://discuss.elastic.co/u/Sharad_Ananth)\
**Post date:** [February 20, 2019, 6:45pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/12 "2019-02-20T18:45:36Z")

</div>

Okay, thanks, David!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 20, 2019, 6:45pm UTC](https://discuss.elastic.co/t/pipeline-grok-expressions/169174/13 "2019-03-20T18:45:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.