HAPROXY (HAPROXYHTTP) truncates last field if over 1024, breaking grok pattern

Elastic Stack Logstash
1

According to haproxy, the standard httplog truncates the final field if it's over 1024 characters long (from the manual):

"http_request" is the complete HTTP request line, including the method,
request and HTTP version string. Non-printable characters are encoded (see
below the section "Non-printable characters"). This is always the last
field, and it is always delimited by quotes and is the only one which can
contain quotes. If new fields are added to the log format, they will be
added before this field. This field might be truncated if the request is
huge and does not fit in the standard syslog buffer (1024 characters). This
is the reason why this field must always remain the last one.

My default filter:
filter { if [type] == "haproxy-access" { grok { match => ["message", "%{HAPROXYHTTP}"] } } }
is failing on any truncated line with a _grokparsefailure. It works fine on non-truncated input. Any suggestions on how to customized the HAPROXYHTTP pattern to fix this?

2

Can you provide examples of both log types?

3

Here's a non-truncated request:

Aug 8 07:08:40 localhost haproxy[28997]: 10.0.2.138:48690 [08/Aug/2016:07:08:40.434] solr_read read_solr_backends/ss1 0/0/0/20/21 200 58492 - - ---- 0/0/0/0/0 0/0 "GET /solr/core/select/?q=cat%3A%2825749%29+&fq=NOT+pt:(ba+OR+bb)+&start=0&rows=25&indent=on&fq=pp:15&fq=photograph:true+&wt=json&json.nl=arrarr&_source=10.0.8.111 HTTP/1.1"

Here's a truncated example, with the length of the final field extending past 1024 characters:

Aug 11 14:56:08 localhost haproxy[1685]: 10.0.2.138:38369 [11/Aug/2016:14:56:08.675] solr_read read_solr_backends/ss5 0/0/0/55/55 200 3804 - - ---- 0/0/0/0/0 0/0 "GET /solr/core/select/?q=title_nostem:(harlem)%20&fq=NOT+product_type:(bb+OR+bc+OR+bd)+&sort=price_ship_us+asc,+score+desc&start=0&rows=20&facet.sort=true&facet.mincount=1&facet=true&facet.limit=20&f.format.facet.method=enum&f.CN.facet.method=enum&f.first.facet.method=enum&f.signed.facet.method=enum&f.dj.facet.method=enum&f.country_id.facet.method=enum&f.photo.facet.method=enum&facet.field=format&facet.field=CN&facet.field=first&facet.field=signed&facet.field=dj&facet.field=country_id&facet.field=photo&facet.field=product_type&facet.query=date:[2000+TO+*]&facet.query=date:[1950+TO+1999]&facet.query=date:[1900+TO+1949]&facet.query=date:[1850+TO+1899]&facet.query=date:[1800+TO+1849]&facet.query=date:[1700+TO+1799]&facet.query=date:[1600+TO+1699]&facet.query=date:[*+TO+1599]&facet.query=fastest_shipping_max_us_ti:[*+TO+3]&fl=id&indent=on&fq=partner_program

You will notice, just as stated in the Haproxy documentation I quoted above, that if the last field is over 1024 characters, haproxy truncates it (and doesn't even put the finishing quote around the field). I'm looking into custom logging on the haproxy side, but this will affect everyone who has request strings > 1024.

4

A short request that the default grok filter works with:

Aug 8 07:08:43 localhost haproxy[28997]: ip_address_stripped:48864 [08/Aug/2016:07:08:43.408] solr_read read_solr_backends/ss5 0/0/0/18/18 200 545 - - ---- 0/0/0/0/0 0/0 "GET /solr/core/select/?q=categories%3A%2827953%29+&fq=NOT+product_type:(bp+OR+bd)+&start=0&rows=25&indent=on&fq=partner_programs:1005&fq=photo:true+&wt=json&json.nl=arrarr&_source=10.0.2.132 HTTP/1.1"

A long request that haproxy truncates:

Aug 11 14:56:08 localhost haproxy[1685]: ip_address_stripped:38369 [11/Aug/2016:14:56:08.675] solr_read read_solr_backends/ss5 0/0/0/55/55 200 3804 - - ---- 0/0/0/0/0 0/0 "GET /solr/core/select/?q=title_nostem:(harlem)%20&fq=NOT+product_type:(bp+OR+bd+OR+ns)+&sort=price_ship_us+asc,+score+desc&start=0&rows=20&facet.sort=true&facet.mincount=1&facet=true&facet.limit=20&f.format.facet.method=enum&f.CN.facet.method=enum&f.first.facet.method=enum&f.signed.facet.method=enum&f.dj.facet.method=enum&f.country_id.facet.method=enum&f.photo.facet.method=enum&facet.field=format&facet.field=CN&facet.field=first&facet.field=signed&facet.field=dj&facet.field=country_id&facet.field=photo&facet.field=product_type&facet.query=date:[2000+TO+]&facet.query=date:[1950+TO+1999]&facet.query=date:[1900+TO+1949]&facet.query=date:[1850+TO+1899]&facet.query=date:[1800+TO+1849]&facet.query=date:[1700+TO+1799]&facet.query=date:[1600+TO+1699]&facet.query=date:[+TO+1599]&facet.query=fastest_shipping_max_us_ti:[*+TO+3]&fl=id&indent=on&fq=partner_program

Even IE8 supports up to 2000 character URI strings, but haproxy only logs the first 1024 characters (as per their documentation quoted above) You'll notice that haproxy doesn't even put the final quote in to complete the filed, it just stops printing.

5

Bump. I'm going to make a custom filter that does a greedy search on the end, but this issue is bigger than just my use-case.