Elastic Serverless Forwarder not reading multiline messages from s3-sqs

Elastic Observability
1

I am having an issue ingesting logs from Elastic Serverless Forwarder (ESF) using s3-sqs. The specific account is still using ESF v1.9.0, but we're in the process of updating that, in case this is part of the issue.

We are using the following config to read from an s3-sqs input:

---
inputs:
  - type: "s3-sqs"
    id: "arn:aws:sqs:us-east-1:000000000000:my-sqs-queue"
    multiline:
      match: "after"
      negate: true
      pattern: '^\['
      type: "pattern"
    outputs:
      - type: "elasticsearch"
        tags: ["myapp"]
        args:
          elasticsearch_url: "https://my-elastic-cloud.us-east-1.aws.elastic-cloud.com:443"
          api_key: "changeme"
          es_datastream_name: "logs-generic.log-default"
          batch_max_actions: 500
          batch_max_bytes: 10485760

The log is from a vendor application. It is very freeform, but the log lines we care about are in the following format:

[2024-09-24T03:00:00,375Z] Log Message 1
[2024-09-24T03:00:03,554Z] Log Message 2

This is part of log message 2, including the blank line above this line.
Most multiline messages start with a blank line.

Still another line in log message 2, including the blank line above.
[2024-09-24T03:00:05,449Z] Log Message 3
This is also part of log message 3. There is no blank line above this line.
[2024-09-24T03:00:07,375Z] ...

As you can see, there are blank lines in the log message. However, only the first line of the log message (starting with ^\[) is getting picked up. Everything else is being dropped.

I assume this is something with the log pattern, as I haven't been able to find any open or closed issues regarding multiline in the ESF repo.

Help?

Updates

Update #1

The multiline message documentation states that the Python regex library is used, and the example pattern shows the following:

pattern: '^\\['

However, when we use this, we get an error stating:

[ERROR] ConfigFileException: unterminated character set

The only way to get rid of this error was to use a single backslash to escape the pattern. This may indicate that the documentation needs to be updated.

Update #2

Testing the logs both in the multiline message Go playground and in Python both yield the expected results for processing log messages.

2

Removed #elastic-cloud

3

Removed #elastic-cloud

4

Removed #elastic-cloud