# AWS ELB ingest pipeline bug **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 18, 2021, 9:09am UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356 "2021-06-18T09:09:43Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![stephank](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephank/32/90510_2.png) [@stephank](https://discuss.elastic.co/u/stephank)\ **Post date:** [June 18, 2021, 9:09am UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/1 "2021-06-18T09:09:43Z")
I believe there is a bug in the Filebeat AWS ELB ingest pipeline. In Kibana Logs, they show up as: ```auto [aws][access] 1.2.3.4 "GET HTTP/2.0" 200 152966 ``` Clearly missing the request path. I was able to fix this with the following change: ```auto --- a/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml +++ b/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml @@ -77,7 +77,7 @@ processors: (?:-|%{NUMBER:aws.elb.backend.http.response.status_code:long}) %{NUMBER:http.request.body.bytes:long} %{NUMBER:http.response.body.bytes:long} - \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:http.request.referrer}) (?:-|HTTP/%{NOTSPACE:http.version})\" + \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:_tmp.url_orig}) (?:-|HTTP/%{NOTSPACE:http.version})\" \"%{DATA:user_agent.original}\" %{ELBSSL} ELBTCPLOG: >- @@ -110,6 +110,11 @@ processors: field: event.category value: web + - uri_parts: + if: 'ctx.http != null' + field: _tmp.url_orig + ignore_failure: true + - set: if: 'ctx.http == null' field: 'aws.elb.protocol' ``` And now it does appear correctly: ```auto [aws][access] 1.2.3.4 "GET /blogs? HTTP/2.0" 200 6815 ``` Apparently, Kibana Logs is using te `generic_webserver` rules to display this record in both cases? And using `uri_parts` correctly sets the `url.*` fields that were expected, instead of `http.request.referrer`. I'm not sure why `generic_webserver` is setup to always show a `?` even if no query string is present. I guess that's benign, but does look a little weird. ---
**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\ **Post date:** [June 18, 2021, 10:43am UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/2 "2021-06-18T10:43:12Z")
Hi @stephank can you share what version of Filebeat are you running? @Kaiyan_Sheng do you think we have an issue here? ---
**Author:** ![stephank](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephank/32/90510_2.png) [@stephank](https://discuss.elastic.co/u/stephank)\ **Post date:** [June 18, 2021, 10:57am UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/3 "2021-06-18T10:57:24Z")
Sorry, should’ve mentioned. Filebeat is 7.13.2, as well as Elasticsearch and Kibana. I’m using Elastic Cloud on AWS. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [June 20, 2021, 2:01pm UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/4 "2021-06-20T14:01:32Z")
What do you mean by `generic_webserver`? Also changing the grok patterns doesn't affect the original log message so I don't understand how you're saying it did. ---
**Author:** ![stephank](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephank/32/90510_2.png) [@stephank](https://discuss.elastic.co/u/stephank)\ **Post date:** [June 20, 2021, 2:37pm UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/5 "2021-06-20T14:37:54Z")
On mobile, so I don’t have the full path, but I was talking about `generic_webserver.ts` in the Kibana source tree. That appears to provide rules for how Kibana Logs displays records. (It’s doing more than just display `message`, apparently.) Changing the grok patterns in the Filebeat ingest changes the fields to what those Kibana rules expect. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [June 20, 2021, 6:58pm UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/6 "2021-06-20T18:58:00Z")
Ahh, your talking about the actual Logs section in Kibana. And looking at the docs, it does look like it should be the request url, not the referer. Can you submit an issue on GitHub for this? I can then implement the changes ---
**Author:** ![stephank](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephank/32/90510_2.png) [@stephank](https://discuss.elastic.co/u/stephank)\ **Post date:** [June 23, 2021, 9:33am UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/7 "2021-06-23T09:33:29Z")
Sorry for the delay. I created the issue just now: [Filebeat AWS ELB ingest fields differ from Kibana Observability · Issue #26435 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/26435) ---
**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\ **Post date:** [June 23, 2021, 9:53am UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/8 "2021-06-23T09:53:23Z")
Feel free to pick it up @legoguy1000! Thank you so much!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 21, 2021, 11:53am UTC](https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356/9 "2021-07-21T11:53:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.