# Amazon Data Firehose integration stopped working
**URL:** https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883
**Category:** Elastic Agent
**Tags:** integrations
**Created:** [October 31, 2024, 2:42pm UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883 "2024-10-31T14:42:14Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@ppborfig](https://discuss.elastic.co/u/ppborfig)
#### Post date: [October 31, 2024, 2:42pm UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883/1 "2024-10-31T14:42:14Z")
The Amazon Data Firehose integration recently stopped working in all our Elastic Cloud environments.
No configuration changes were made in AWS and Elastic.
Firehose shows no errors and all deliveries are successful.
Re-installing the integration and restarting Elastic did not resolve the issue.
How do we debug this issue?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 1, 2024, 3:44am UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883/2 "2024-11-01T03:44:59Z")
@ppborfig Welcome to the community.
Couple questions.
First did you open a support ticket?
What version of the AWS Firehose Integration are you on?
Did you upgrade it recently?
Can you check to see if your logs got routed to a different data stream?
Starting around the same time..We have seen a reported issue the same behavior of the firehouse.
The data stream where data is saved has the default name "logs-awsfirehose-default".
And what is your setting for
`aws.firehose.parameters.es_datastream_name:` in the the Firehose Configuration.
---
### Author:  [@ppborfig](https://discuss.elastic.co/u/ppborfig)
#### Post date: [November 1, 2024, 6:46am UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883/3 "2024-11-01T06:46:57Z")
Hi, @stephenb
I did open a support ticket.
I don't remember what version we were on during the incident. Updating AWS and Amazon Data Firehose integrations to the latest have not resolved the issue.
We have multiple Firehose instances, each with its own `es_datastream_name`. All Firehose instances are happy, but the data streams in Elastic do not receive new data.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 1, 2024, 2:07pm UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883/4 "2024-11-01T14:07:22Z")
Looks There is an issue...
There is a workaround but it is not pretty.
New validation was added that validates that the value of
`aws.firehose.parameters.es_datastream_name`
Is a valid datastream naming convention if it is not (which yours most likely is not) it will fall back to `logs-awsfirehose-default`
Did you check to see if your data is in a datastream named
Data streams must follow datastream naming convention see [here](https://www.elastic.co/guide/en/fleet/8.15/data-streams.html#data-streams-naming-scheme)
You can add the following pipeline as a temporary workaround.
You have hardcode the value of the `destination` to the value you want
For some reason `destination` requires a static value so it can not use mustache syntax, which would make this much cleaner.
`"aws.firehose.parameters.es_datastream_name": "my-non-compliant-ds-name"`
so set `destination` to the hard coded value
`"destination": "my-non-compliant-ds-name", `
**Do not set**
`"destination": "{{aws.firehose.parameters.es_datastream_name}}",`
Example and I tested this and it works.
```auto
PUT _ingest/pipeline/logs-awsfirehose@custom
{
"processors": [
{
"set": {
"field": "pipeline_custom_name",
"value": "logs-awsfirehose@custom",
"override": false
}
},
{
"reroute": {
"destination": "my-non-compliant-ds-name", <<< Where this is the hardcoded value in aws.firehose.parameters.es_datastream_name
"ignore_failure": true
}
}
]
}
```
I have tested this and it works and should allow customers to continue to recieve logs in the destination the need to while they work on a migration plan
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 1, 2024, 3:23pm UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883/5 "2024-11-01T15:23:43Z")
Note if you have many streams...
If you have multiple streams you will need to add `if` logic like below.
The reason I have to make a temporary field name is because `aws.firehose.parameters.es_datastream_name` is a `flattened` field which it seems you can not use in the `if` logic perhaps someone knows better.
```auto
PUT _ingest/pipeline/logs-awsfirehose@custom
{
"processors": [
{
"set": {
"field": "pipeline_custom_name",
"value": "logs-awsfirehose@custom",
"override": false
}
},
{
"set": {
"field": "es_datastream_name",
"value": "{{aws.firehose.parameters.es_datastream_name}}"
}
},
{
"reroute": {
"if" : "ctx.es_datastream_name == 'my-custom-ds'",
"destination": "my-custom-ds'",
"ignore_failure": true
}
},
{
"reroute": {
"if" : "ctx.es_datastream_name == 'my-other-custom-ds'",
"destination": "my-other-custom-ds",
"ignore_failure": true
}
}
]
}
```
---
### Author:  [@ppborfig](https://discuss.elastic.co/u/ppborfig)
#### Post date: [November 1, 2024, 3:54pm UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883/6 "2024-11-01T15:54:19Z")
I see that the data is _not_ routed to `logs-awsfirehose-default`: there is no datastream with that name (I do see `logs-awsfirehose` index template though).
Each Firehose instance uses a separate API key with a role that allows writing to _only_ the specified index/datastream. I would expect permission errors, but there were none.
Adding `logs-awsfirehose*` to the permissions did not help either.
It is important to note that no configuration changes or updates (Elastic and Integrations) were made before the incident.
I will try to setup a new Firehose-\>Elastic connection and see if it works.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 1, 2024, 4:06pm UTC](https://discuss.elastic.co/t/amazon-data-firehose-integration-stopped-working/369883/7 "2024-11-01T16:06:36Z")
> [@ppborfig](#):
>
> It is important to note that no configuration changes or updates (Elastic and Integrations) were made before the incident.
Understood.
> [@ppborfig](#):
>
> Each Firehose instance uses a separate API key with a role that allows writing to _only_ the specified index/datastream. I would expect permission errors, but there were none.
> Adding `logs-awsfirehose*`
**^^^ This is why the logs were not written to the default**
Do you have the integration AWS Firehose assets installed on the Elasticsearch Side as I showed above.
What is your value of `aws.firehose.parameters.es_datastream_name:`
Did you try my fix above? That should work.
If you set up a new stream please use a compliant data stream name or the fix I provided.