AWS ELB ingest pipeline bug

[stephank (Stéphan)]

I believe there is a bug in the Filebeat AWS ELB ingest pipeline. In Kibana Logs, they show up as:

[aws][access] 1.2.3.4  "GET  HTTP/2.0" 200 152966

Clearly missing the request path. I was able to fix this with the following change:

--- a/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
+++ b/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
@@ -77,7 +77,7 @@ processors:
           (?:-|%{NUMBER:aws.elb.backend.http.response.status_code:long})
           %{NUMBER:http.request.body.bytes:long}
           %{NUMBER:http.response.body.bytes:long}
-          \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:http.request.referrer}) (?:-|HTTP/%{NOTSPACE:http.version})\"
+          \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:_tmp.url_orig}) (?:-|HTTP/%{NOTSPACE:http.version})\"
           \"%{DATA:user_agent.original}\"
           %{ELBSSL}
         ELBTCPLOG: >-
@@ -110,6 +110,11 @@ processors:
       field: event.category
       value: web

+  - uri_parts:
+      if: 'ctx.http != null'
+      field: _tmp.url_orig
+      ignore_failure: true
+
   - set:
       if: 'ctx.http == null'
       field: 'aws.elb.protocol'

And now it does appear correctly:

[aws][access] 1.2.3.4  "GET /blogs? HTTP/2.0" 200 6815

Apparently, Kibana Logs is using te generic_webserver rules to display this record in both cases? And using uri_parts correctly sets the url.* fields that were expected, instead of http.request.referrer.

I'm not sure why generic_webserver is setup to always show a ? even if no query string is present. I guess that's benign, but does look a little weird.

[ChrsMark (Chris Mark)]

Hi @stephank can you share what version of Filebeat are you running?

@Kaiyan_Sheng do you think we have an issue here?

[stephank (Stéphan)]

Sorry, should’ve mentioned. Filebeat is 7.13.2, as well as Elasticsearch and Kibana. I’m using Elastic Cloud on AWS.

[legoguy1000 (Alex)]

What do you mean by generic_webserver? Also changing the grok patterns doesn't affect the original log message so I don't understand how you're saying it did.

[stephank (Stéphan)]

On mobile, so I don’t have the full path, but I was talking about generic_webserver.ts in the Kibana source tree. That appears to provide rules for how Kibana Logs displays records. (It’s doing more than just display message, apparently.)

Changing the grok patterns in the Filebeat ingest changes the fields to what those Kibana rules expect.

[legoguy1000 (Alex)]

Ahh, your talking about the actual Logs section in Kibana. And looking at the docs, it does look like it should be the request url, not the referer. Can you submit an issue on GitHub for this? I can then implement the changes

[stephank (Stéphan)]

Sorry for the delay. I created the issue just now: Filebeat AWS ELB ingest fields differ from Kibana Observability · Issue #26435 · elastic/beats · GitHub

[Kaiyan_Sheng]

Feel free to pick it up @legoguy1000! Thank you so much!!