Elastic Serverless Forwarder Field Extraction Issue

Elastic Observability Logs
1

Hello,

I'm currently ingesting CloudWatch logs via Kinesis > Elastic Serverless Forwarder, but I'm having issues with injecting existing "root" fields after expanding events from JSON object lists. I've attached my config.yaml and sample logs.

Elastic Serverless Forwarder:

inputs:
  - type: "kinesis-data-stream"
    id: "arn:aws:kinesis:us-east-1::stream/ElasticRecipientStream"
    expand_event_list_from_field: "logEvents"
    root_fields_to_add_to_expanded_event: "all"
    tags:
      - "aws-serverless-kinesis"
    outputs:
      - type: "elasticsearch"
        args:
          # either elasticsearch_url or cloud_id, elasticsearch_url takes precedence
          # elasticsearch_url: "http(s)://domain.tld:port"
          cloud_id: "<redacted>"
          # either api_key or username/password, api_key takes precedence
          api_key: "<redacted>"
          es_datastream_name: "logs-aws-serverless"
          batch_max_actions: 500
          batch_max_bytes: 10485760
          ssl_assert_fingerprint: ""

Essentially my events are formatted as seen in the Elastic Serverless Forwarder documentation below. The example shows the logs without "root_fields_to_add_to_expanded_event" setting, but my logs are still missing the root fields even with this setting enabled.

image
image1321×969 70.1 KB

2

The expanded logs you've shared are not valid json -- can you confirm the contents? Based on the docs, i would expect the value for the message key to be stringified json.

Can you share what version of the serverless forwarder you're running? Adding root fields to expanded messages was added in v1.8.0

3

Ahh ha. We're currently on version 1.6.0 which explains why I'm seeing the expanded logEvents object field working as expected and not the the root field injection. Are there any changelogs where I can see updates? The Elastic Serverless Forwarder (ESF) documentation didn't really specify anything regarding settings and compatible ESF versions.

Thanks! :raised_hands:

1 Like
4

I wish I had a better answer for you but the changelog for ESF can be viewed here elastic-serverless-forwarder/CHANGELOG.md at main · elastic/elastic-serverless-forwarder · GitHub

Please do let me know if you run into any remaining issues after rolling out the latest version of the lambda.

1 Like