# Parsing a varying length line with custom grok filter, only returns the first field

**URL:** <https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533>\
**Category:** Logstash\
**Created:** [November 26, 2019, 2:33pm UTC](https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533 "2019-11-26T14:33:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![anar](https://avatars.discourse-cdn.com/v4/letter/a/a6a055/32.png) [@anar](https://discuss.elastic.co/u/anar)\
**Post date:** [November 26, 2019, 2:33pm UTC](https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533/1 "2019-11-26T14:33:33Z")

</div>

Hello,

I'm trying to set up a custom grok filter for my data input, but when I test it in Kibana's Grok Debugger, I only get the value for the first field (field1). I'm using grok instead of the csv parser because after field7 the last data field is varying length, and it should just be a single entry in Logstash (I will do some post processing on it afterwards).

My data looks like this:

`91877900$|$11613428$|$DEVICE$|$CUSTOM-DEVICE1$|$UTC+02:00$|$["13","19","24","53","60","61","65","66","67","8","1"]$|$title=News$|$genre=News Broadcast$|$startTime=1574190000000$|$programId=659107083$|`

My grok pattern looks like this:

`%{INT:field1}\$|$ %{INT:field2}\$|$ %{WORD:field3}\$|$ %{DATA:field4}\$|$ %{DATA:field5}\$|$ %{TZ:field6}\$|$ %{GREEDYDATA:field7}\$|$ %{GREEDYDATA:theRestOfIt}`

Can someone help with this? I'm getting stuck on this part, and I don't understand why my output looks like this:

```
{
  "field1": "91877900"
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 26, 2019, 2:45pm UTC](https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533/2 "2019-11-26T14:45:18Z")

</div>

You need to escape all of the $ and | with \

| is used for alternation -- foo|bar matches either foo or bar, so your pattern match any one of

```
%{INT:field1}\$
$ %{INT:field2}\$
$ %{WORD:field3}\$$
etc.

```

So once if matches the first INT it does not check the rest of the patterns.

---

<div class="post-metadata">

**Author:** ![anar](https://avatars.discourse-cdn.com/v4/letter/a/a6a055/32.png) [@anar](https://discuss.elastic.co/u/anar)\
**Post date:** [November 26, 2019, 2:55pm UTC](https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533/3 "2019-11-26T14:55:15Z")

</div>

Thank you for the quick reply, Badger.

I tried escaping the $ and | with `\`, but if I run  
`%{INT:field1}\$ \|\$%{INT:field2}\$`  
or  
`%{INT:field1}\$ \$%{INT:field2}\$`  
on my input, I get a "Provided Grok patterns do not match data in the input" error.

I also get the same error if I try:  
`%{INT:field1}\$\|\$ %{INT:field2}\$\|\$ %{WORD:field3}\$\|\$ %{DATA:field4}\$\|\$ %{DATA:field5}\$\|\$ %{TZ:field6}\$\|\$ %{GREEDYDATA:field7}\$\|\$ %{GREEDYDATA:theRestOfIt}`

Am I missing something?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 26, 2019, 4:32pm UTC](https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533/4 "2019-11-26T16:32:31Z")

</div>

Remove all the spaces and replace TZ with DATA.

```
input { generator { count => 1 lines => ['91877900$|$11613428$|$DEVICE$|$CUSTOM-DEVICE1$|$UTC+02:00$|$["13","19","24","53","60","61","65","66","67","8","1"]$|$title=News$|$genre=News Broadcast$|$startTime=1574190000000$|$programId=659107083$|' ] } }
filter {
    grok { match => { "message" => "%{INT:field1}\$\|\$%{INT:field2}\$\|\$%{WORD:field3}\$\|\$%{DATA:field4}\$\|\$%{DATA:field5}\$\|\$%{DATA:field6}\$\|\$%{GREEDYDATA:field7}\$\|\$%{GREEDYDATA:theRestOfIt}" } }
}
output { stdout { codec => rubydebug { metadata => false } } }

```

produces

```
     "field6" => "[\"13\",\"19\",\"24\",\"53\",\"60\",\"61\",\"65\",\"66\",\"67\",\"8\",\"1\"]",
     "field1" => "91877900",
"theRestOfIt" => "programId=659107083$|",
     "field7" => "title=News$|$genre=News Broadcast$|$startTime=1574190000000",

```

etc.

---

<div class="post-metadata">

**Author:** ![anar](https://avatars.discourse-cdn.com/v4/letter/a/a6a055/32.png) [@anar](https://discuss.elastic.co/u/anar)\
**Post date:** [November 27, 2019, 11:52am UTC](https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533/5 "2019-11-27T11:52:43Z")

</div>

Thanks a lot for the help, that solved it!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2019, 11:53am UTC](https://discuss.elastic.co/t/parsing-a-varying-length-line-with-custom-grok-filter-only-returns-the-first-field/209533/6 "2019-12-25T11:53:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.