# Squid parsing stopped to parse all fields

**URL:** <https://discuss.elastic.co/t/squid-parsing-stopped-to-parse-all-fields/180477>\
**Category:** Logstash\
**Created:** [May 10, 2019, 4:13am UTC](https://discuss.elastic.co/t/squid-parsing-stopped-to-parse-all-fields/180477 "2019-05-10T04:13:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mauriciothomsen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mauriciothomsen/32/36507_2.png) [@mauriciothomsen](https://discuss.elastic.co/u/mauriciothomsen)\
**Post date:** [May 10, 2019, 4:13am UTC](https://discuss.elastic.co/t/squid-parsing-stopped-to-parse-all-fields/180477/1 "2019-05-10T04:13:39Z")

</div>

**Hello everybody.**

\*\*I have a Logstash job loading data from a kafka topic, running for many months without any issue. The only change done on the last 3 weeks was the upgrade to ELK 7.0. Still, after the upgrade the loading was working fine however, after a certain day, it stopped to parse most of the fields for no reason. \*\*

**Here's the .conf file:**

input {  
kafka {  
bootstrap\_servers =\> ":9094"  
topics =\> ["proxy"]  
codec =\> "json"  
}  
}

filter {  
grok {  
patterns\_dir =\> ["./patterns"]  
break\_on\_match =\> false  
match =\> { "payload" =\> "%{SQUID\_TIMESTAMP}" }  
match =\> { "payload" =\> "%{TCP\_STATUS}" }  
match =\> { "payload" =\> "%{TARGET\_WEBSITE}" }  
match =\> { "payload" =\> "%{TARGET\_WEBSITE\_GET}" }  
match =\> { "payload" =\> "%{TARGET\_WEBSITE\_GET2}" }  
match =\> { "payload" =\> "%{SQUID\_BYTES}" }  
match =\> { "payload" =\> "%{SQUID\_IP}" }  
match =\> { "payload" =\> "%{SQUID\_SOURCE\_IP}" }  
}

date {  
match =\> ["proxy\_timestamp","UNIX"]  
}

mutate {  
convert =\> { "bytes" =\> "integer" }  
}

geoip {  
source =\> "ip\_address"  
}

translate {  
field =\> "source\_ip"  
destination =\> "source"  
dictionary =\> {  
"192.168.1.XXX" =\> "windows-hp"  
"192.168.1.XXX" =\> "linux-1"  
"192.168.1.XXX" =\> "linux-2"  
"192.168.1.XXX" =\> "linux-3"  
"192.168.1.XXX" =\> "laptop"  
"192.168.1.XXX" =\> "windows-tablet"  
"192.168.1.XXX" =\> "linux-master"  
}  
}

}

output {  
stdout {  
codec =\> rubydebug  
}

elasticsearch {  
hosts =\> ["192.168.1.XXX:9200","192.168.1.XXX:9200","192.168.1.XXX:9200"]  
index =\> "proxy-%{+YYYY.MM}"  
}  
}

**Here's the pattern file:**

SQUID\_TIMESTAMP ^(?\<proxy\_timestamp\>\d\*)  
TARGET\_WEBSITE CONNECT\s(?[^:]_)  
TARGET\_WEBSITE\_GET GET htt\w_://(?\w\*.\w\*.\w\*)  
TARGET\_WEBSITE\_GET2 GET htt\w\*://(?\w\*.\w\*)  
TCP\_STATUS TCP\_(?\<tcp\_status\>[^/]_)  
SQUID\_BYTES TCP\_\w_/\d\*\s(?\d\*)  
SQUID\_SOURCE\_IP \d\*.\d\*\s\*\d\*\s(?\<source\_ip\>\d\*.\d\*.\d\*.\d\*)  
SQUID\_IP HIER\_DIRECT/(?\<ip\_address\>\d\*.\d\*.\d\*.\d\*)

**And below a sample of the output:**

{  
"source\_ip" =\> "192.168.1.XXX",  
"@version" =\> "1",  
"@timestamp" =\> 2019-05-10T04:09:09.360Z,  
"payload" =\> "1557461305.894 686929 192.168.1.XXX TCP\_TUNNEL/200 194202 CONNECT [www.elastic.co:443](http://www.elastic.co:443) - HIER\_DIRECT/151.101.130.217 -",  
"schema" =\> {  
"type" =\> "string",  
"optional" =\> false  
},  
"tags" =\> [  
[0] "\_geoip\_lookup\_failure"  
],  
"source" =\> "windows-hp"  
}

\*\*Note that the parsing of "source\_ip" worked, since it showed the value and properly mapped the device name, but all other regex in the pattern file did not work. There was absolutely no change o any configuration file so I have no idea what can be causing this issue. \*\*

**Appreciate any tip or feedback.**

**Thanks.**

---

<div class="post-metadata">

**Author:** ![mauriciothomsen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mauriciothomsen/32/36507_2.png) [@mauriciothomsen](https://discuss.elastic.co/u/mauriciothomsen)\
**Post date:** [May 10, 2019, 11:35pm UTC](https://discuss.elastic.co/t/squid-parsing-stopped-to-parse-all-fields/180477/2 "2019-05-10T23:35:00Z")

</div>

Just made it worked. I basically changed the grok filter to the following:

filter {  
grok {  
patterns\_dir =\> ["./patterns"]  
break\_on\_match =\> false  
match =\> { "payload" =\> [  
"%{SQUID\_SOURCE\_IP}",  
"%{SQUID\_IP}",  
"%{TCP\_STATUS}",  
"%{TARGET\_WEBSITE}",  
"%{TARGET\_WEBSITE\_GET}",  
"%{TARGET\_WEBSITE\_GET2}",  
"%{SQUID\_BYTES}"  
] }  
}

It's cleaner and make more sense, although I still couldn't figure out why the previous one stopped to work - maybe due to version.

Anyway, I'm keeping this post in case somebody bumps on the same issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 7, 2019, 11:35pm UTC](https://discuss.elastic.co/t/squid-parsing-stopped-to-parse-all-fields/180477/3 "2019-06-07T23:35:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.