Filebeat, s3access & ECS

cosmo · February 5, 2020, 2:53pm

Hello

Looking to start migrating to ECS, for all the benefits listed on the blog

Got nginx running through the filebeat modules, happy days (although the blog post suggests I should have event.original and message fields, which I don't appear to have...)

I want to convert some s3 access logs, that are currently getting pulled in by logstash. I figured I could be lazy and steal the filebeat pipeline

github.com

elastic/beats/blob/master/x-pack/filebeat/module/aws/s3access/ingest/pipeline.yml

description: "Pipeline for s3 server access logs"

processors:
  - grok:
      field: message
      patterns:
        - >-
          %{BASE16NUM:aws.s3access.bucket_owner} %{HOSTNAME:aws.s3access.bucket} \[%{HTTPDATE:_temp_.s3access_time}\]
          %{IP:aws.s3access.remote_ip} (?:-|%{S3REQUESTER:aws.s3access.requester}) %{S3REQUESTID:aws.s3access.request_id}
          %{S3OPERATION:aws.s3access.operation} (?:-|%{S3KEY:aws.s3access.key}) (?:-|\"%{DATA:aws.s3access.request_uri}\")
          %{NUMBER:aws.s3access.http_status:long} (?:-|%{WORD:aws.s3access.error_code}) (?:-|%{NUMBER:aws.s3access.bytes_sent:long})
          (?:-|%{NUMBER:aws.s3access.object_size:long}) (?:-|%{NUMBER:aws.s3access.total_time:long}) (?:-|%{NUMBER:aws.s3access.turn_around_time:long})
          (?:-|\"%{DATA:aws.s3access.referrer}\") (?:-|\"(-|%{DATA:aws.s3access.user_agent})\") (?:-|%{S3KEY:aws.s3access.version_id})
          (?:-|%{S3ID:aws.s3access.host_id}) (?:-|%{S3VERSION:aws.s3access.signature_version}) (?:-|%{S3KEY:aws.s3access.cipher_suite})
          (?:-|%{WORD:aws.s3access.authentication_type}) (?:-|%{S3ID:aws.s3access.host_header}) (?:-|%{S3VERSION:aws.s3access.tls_version})
      pattern_definitions:
        S3REQUESTER: "[a-zA-Z0-9\\/_\\.\\-%:@]+"
        S3REQUESTID: "[a-zA-Z0-9]+"
        S3OPERATION: "%{WORD}.%{WORD}.%{WORD}"
        S3KEY: "[a-zA-Z0-9\\/_\\.\\-%+]+"

This file has been truncated. show original

However as I understand things, that doesn't seem to be using ECS yet?
(EG it uses aws.s3access.remote_ip rather than source.ip)

Am I right in this belief, and I need to go off an manually map all the fields to ECS?
Or am I looking at the wrong pipeline for filebeat?

Thanks

Kaiyan_Sheng · February 11, 2020, 2:22pm

Hi @cosmo, yeah you are right, aws.s3access.remote_ip is not using ECS right now. The fields in s3access fileset that's using ECS are: tls.cipher, tls.version and tls.version_protocol. Maybe https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/aws/vpcflow/ingest/pipeline.yml for vpcflow fileset will be a better ECS example for you. I will create a github issue to track converting fields in s3access to ECS as well. Thanks!

system · March 10, 2020, 4:22pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.