# Problem processing squid logs with logstash

**URL:** <https://discuss.elastic.co/t/problem-processing-squid-logs-with-logstash/61137>\
**Category:** Logstash\
**Created:** [September 21, 2016, 1:22pm UTC](https://discuss.elastic.co/t/problem-processing-squid-logs-with-logstash/61137 "2016-09-21T13:22:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ageldenberg](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@ageldenberg](https://discuss.elastic.co/u/ageldenberg)\
**Post date:** [September 21, 2016, 1:22pm UTC](https://discuss.elastic.co/t/problem-processing-squid-logs-with-logstash/61137/1 "2016-09-21T13:22:46Z")

</div>

Hello, everyone.

I am breaking my head over this issue.

Logstash configuration has the following grok statement:

%{MONTH}\s+%{MONTHDAY}\s+%{TIME}\s+%{IPORHOST:src\_proxy}?\s+\S+\s+%{BASE16FLOAT:timestamp}\s+%{NUMBER:request\_msec:float}\s+%{IPV4:src\_ip}\s+%{WORD:cache\_result}/%{NUMBER:response\_status:int}\s+%{NUMBER:response\_size:int}\s+%{WORD:http\_method}\s+(%{URIPROTO:http\_proto}://)?%{IPORHOSTWITHUNDERSCORE:dst\_host}(?::%{POSINT:port:int})?(?:%{URIPATHPARAM:uri\_param})?_\s+%{DATA:cache\_user}\s+%{DATA:request\_route}/(?:%{IPORHOST:forwarded\_to}|-)\s_(?:%{GREEDYDATA:content\_type}|-)?

uilizing the following custom patterns:

HOSTWITHUNDERSCORE \b(?:[0-9A-Za-z][0-9A-Za-z\_-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z\_-]{0,62}))\*(.?|\b)  
IPORHOSTWITHUNDERSCORE (?:%{IP}|%{HOSTWITHUNDERSCORE})

The following squid statements follow with \_grokparsefailure while being processed by logstash on the server. However, they cleanly match the test on the grok test web site ( [http://grokconstructor.appspot.com/do/match#result](http://grokconstructor.appspot.com/do/match#result) ):

Sep 21 08:40:24 proxy squid[3635]: 1474461624.308 87 1.1.1.13 TCP\_MISS/200 313 GET [http://aax.amazon-adsystem.com/x/px/IDfEVLsFwE5MoEa-0noSYhoAAAFXTMLUDQEAAAzmEXiz-w/{"adCsm":%20[{"vfrd":1,"dbg":"366x47"},{"lteu":"0.08","ltut":"0.05","ltpq":"0.24","ltvd":"0.22","lths":"0.16","ltpm":"0.28","ltfm":"0.65","csmTot":"5.06"}],%20"pixelId":%20"gc1383zqwx4aq0k9",%20"ts":%201474461624324}&cb=8238899](http://aax.amazon-adsystem.com/x/px/IDfEVLsFwE5MoEa-0noSYhoAAAFXTMLUDQEAAAzmEXiz-w/%7B%22adCsm%22:%20%5B%7B%22vfrd%22:1,%22dbg%22:%22366x47%22%7D,%7B%22lteu%22:%220.08%22,%22ltut%22:%220.05%22,%22ltpq%22:%220.24%22,%22ltvd%22:%220.22%22,%22lths%22:%220.16%22,%22ltpm%22:%220.28%22,%22ltfm%22:%220.65%22,%22csmTot%22:%225.06%22%7D%5D,%20%22pixelId%22:%20%22gc1383zqwx4aq0k9%22,%20%22ts%22:%201474461624324%7D&cb=8238899) user1 USERHASH\_PARENT/path1.ext image/gif

Sep 21 09:11:10 proxy squid[3635]: 1474463470.424 46 1.1.1.14 TCP\_MISS/302 610 GET [http://dpm.demdex.net/ibs:dpid=30862&puuid=2392662699457081&redir=https%3A%2F%2Ft.mookie1.com%2Ft%2Fv1%2Fevent%3FmigClientId%3L7413%26migAction%3Dsync%26migSource%3Dmig%26migParam1%3D${DD\_UUID}](http://dpm.demdex.net/ibs:dpid=30862&puuid=2392662699457081&redir=https%3A%2F%2Ft.mookie1.com%2Ft%2Fv1%2Fevent%3FmigClientId%253L7413%26migAction%3Dsync%26migSource%3Dmig%26migParam1%3D%24%7BDD_UUID%7D) user2 USERHASH\_PARENT/path3.ext -

I cannot understand why they are not being processed. The configuration seems correct. How can I troubleshoot further?

I will greatly appreciate your insights.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 25, 2016, 7:54pm UTC](https://discuss.elastic.co/t/problem-processing-squid-logs-with-logstash/61137/2 "2016-09-25T19:54:38Z")

</div>

Have you investigated which part of the expression is causing the mismatch? Start with the simplest possible expression, e.g. `%{MONTH}` and add more pieces until you yet again get `_grokparsefailure`.

> ```
> HOSTWITHUNDERSCORE \b(?:[0-9A-Za-z][0-9A-Za-z_\-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z_\-]{0,62}))*(.?|\b)
> 
> ```

I think this kind of grok pattern is a mistake. I don't think Logstash's purpose is to validate hostnames, so I'd use the simplest possible expression. The hostname ends when we encounter either a colon or a slash, yes? Hence:

```
HOSTWITHUNDERSCORE [^:/]+

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:37am UTC](https://discuss.elastic.co/t/problem-processing-squid-logs-with-logstash/61137/3 "2017-07-06T04:37:01Z")

</div>