Amazon Data Firehose integration stopped working

Elastic Stack Elastic Agent
1

The Amazon Data Firehose integration recently stopped working in all our Elastic Cloud environments.
No configuration changes were made in AWS and Elastic.
Firehose shows no errors and all deliveries are successful.
Re-installing the integration and restarting Elastic did not resolve the issue.

How do we debug this issue?

2

@ppborfig Welcome to the community.

Couple questions.

First did you open a support ticket?

What version of the AWS Firehose Integration are you on?

Did you upgrade it recently?

Screenshot 2024-10-31 at 8.51.56 PM
Screenshot 2024-10-31 at 8.51.56 PM2318×1294 228 KB

Can you check to see if your logs got routed to a different data stream?

Starting around the same time..We have seen a reported issue the same behavior of the firehouse.

The data stream where data is saved has the default name "logs-awsfirehose-default".

And what is your setting for

aws.firehose.parameters.es_datastream_name: in the the Firehose Configuration.

3

Hi, @stephenb
I did open a support ticket.

I don't remember what version we were on during the incident. Updating AWS and Amazon Data Firehose integrations to the latest have not resolved the issue.

We have multiple Firehose instances, each with its own es_datastream_name. All Firehose instances are happy, but the data streams in Elastic do not receive new data.

4

Looks There is an issue...

There is a workaround but it is not pretty.

New validation was added that validates that the value of

aws.firehose.parameters.es_datastream_name

Is a valid datastream naming convention if it is not (which yours most likely is not) it will fall back to logs-awsfirehose-default

Did you check to see if your data is in a datastream named

Data streams must follow datastream naming convention see here

You can add the following pipeline as a temporary workaround.

You have hardcode the value of the destination to the value you want
For some reason destination requires a static value so it can not use mustache syntax, which would make this much cleaner.

"aws.firehose.parameters.es_datastream_name": "my-non-compliant-ds-name"

so set destination to the hard coded value

"destination": "my-non-compliant-ds-name",

Do not set

"destination": "{{aws.firehose.parameters.es_datastream_name}}",

Example and I tested this and it works.

PUT _ingest/pipeline/logs-awsfirehose@custom
{
  "processors": [
    {
      "set": {
        "field": "pipeline_custom_name",
        "value": "logs-awsfirehose@custom",
        "override": false
      }
    },
    {
      "reroute": {
        "destination": "my-non-compliant-ds-name",  <<< Where this is the hardcoded value in aws.firehose.parameters.es_datastream_name
        "ignore_failure": true
      }
    }
  ]
}

I have tested this and it works and should allow customers to continue to recieve logs in the destination the need to while they work on a migration plan

5

Note if you have many streams...

If you have multiple streams you will need to add if logic like below.
The reason I have to make a temporary field name is because aws.firehose.parameters.es_datastream_name is a flattened field which it seems you can not use in the if logic perhaps someone knows better.

PUT _ingest/pipeline/logs-awsfirehose@custom
{
  "processors": [
    {
      "set": {
        "field": "pipeline_custom_name",
        "value": "logs-awsfirehose@custom",
        "override": false
      }
    },
    {
      "set": {
        "field": "es_datastream_name",
        "value": "{{aws.firehose.parameters.es_datastream_name}}"
      }
    },
    {
      "reroute": {
        "if" : "ctx.es_datastream_name == 'my-custom-ds'",
        "destination": "my-custom-ds'",
        "ignore_failure": true
      }
    },
    {
      "reroute": {
        "if" : "ctx.es_datastream_name == 'my-other-custom-ds'",
        "destination": "my-other-custom-ds",
        "ignore_failure": true
      }
    }
  ]
}
6

I see that the data is not routed to logs-awsfirehose-default: there is no datastream with that name (I do see logs-awsfirehose index template though).
Each Firehose instance uses a separate API key with a role that allows writing to only the specified index/datastream. I would expect permission errors, but there were none.
Adding logs-awsfirehose* to the permissions did not help either.

It is important to note that no configuration changes or updates (Elastic and Integrations) were made before the incident.

I will try to setup a new Firehose->Elastic connection and see if it works.

7

Understood.

^^^ This is why the logs were not written to the default

Do you have the integration AWS Firehose assets installed on the Elasticsearch Side as I showed above.

What is your value of aws.firehose.parameters.es_datastream_name:

Did you try my fix above? That should work.

If you set up a new stream please use a compliant data stream name or the fix I provided.