# AWS CloudFront Ingest Pipeline Failing **URL:** \ **Category:** Elasticsearch\ **Tags:** ingest-pipeline\ **Created:** [February 6, 2024, 7:59pm UTC](https://discuss.elastic.co/t/aws-cloudfront-ingest-pipeline-failing/352678 "2024-02-06T19:59:44Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [February 6, 2024, 7:59pm UTC](https://discuss.elastic.co/t/aws-cloudfront-ingest-pipeline-failing/352678/1 "2024-02-06T19:59:44Z")
I am having an issue with logs ingesting into `logs-aws.cloudfront_logs-*` from a specific account, using Elastic Serverless Forwarder. In one specific account and this account only, I am receiving the following error in Elasticsearch for events: `Provided Grok expressions do not match field value`. When I take a raw log entry from the CloudWatch log group and run it through the `POST _ingest/pipeline/logs-aws.cloudfront_logs-2.11.3/_simulate` API, it parses as expected. When I copy the log entry from the `error.message` field in Kibana and run it through the same API, it also parses as expected. This only happens for events being sent from Elastic Serverless Forwarder for this log group in this account. I have verified that the log group is being ingested by Elastic Serverless Forwarder and that the logs are being sent to the correct index, and I have also verified that this is NOT occurring for any other correctly-configured log groups in this account OR for cloudFront logs in other accounts. This is my configuration for the log group in question: ```yaml - type: 'cloudwatch-logs' id: 'arn:aws:logs:::log-group:cloudfront/:*' outputs: - type: 'elasticsearch' args: elasticsearch_url: 'https://myinstance.es..aws.elastic-cloud.com:443' api_key: '' es_datastream_name: 'logs-aws.cloudfront_logs-default' batch_max_actions: 500 batch_max_bytes: 10485760 ``` I am using Elastic Cloud 8.12.1, and AWS integration v2.11.3. # Edit I'm also getting this error periodically: ```auto grok pattern matching was interrupted after [1000] ms ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 6, 2024, 10:09pm UTC](https://discuss.elastic.co/t/aws-cloudfront-ingest-pipeline-failing/352678/2 "2024-02-06T22:09:03Z")
Hi @DougR We see this sometimes when a source, for some reason, does not match... The only way anyone can help if you provide a sample raw message so Most Likely, you are going to need to add a new pattern to the pipeline... > [@DougR](#): > > When I take a raw log entry from the CloudWatch log group and run it through the `POST _ingest/pipeline/logs-aws.cloudfront_logs-2.11.3/_simulate` API, it parses as expected. When I copy the log entry from the `error.message` field in Kibana and run it through the same API, it also parses as expected. Confused a bit 🙂 Is this on a message that has the error message `"Grok expressions do not match"`? I am confused ... are you saying that on a message that shows Grok Error then you run it again it does not? (that _can_ happen for some very specific reasons .. long detail there, but unlikely) Can you share this?? Are you setting preserve event original? You should do that and then use that where the Grok fails... can help with debugging.. Show us some examples perhaps we can help.... ![Screenshot 2024-02-06 at 2.08.22 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/d/dd5bc564e398473a2cc917f57a0e1701d17081dc.png) Oh this is from serverless forwarder... hmm not sure if there is a preserve event or not... we can edit the pipeline and take out the remove if needed... Show us some examples ... we can take a look ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [February 7, 2024, 5:56pm UTC](https://discuss.elastic.co/t/aws-cloudfront-ingest-pipeline-failing/352678/3 "2024-02-07T17:56:47Z")
> [@stephenb](#): > > Confused a bit 🙂 > > Is this on a message that has the error message `"Grok expressions do not match"`? > > I am confused ... are you saying that on a message that shows Grok Error then you run it again it does not? (that _can_ happen for some very specific reasons .. long detail there, but unlikely) I'm getting both error messages, in fact. But only when I send CloudFront logs from THIS ACCOUNT ONLY to Elastic Cloud. No other account shows this issue. Additionally: - If I copy the raw event from CloudWatch and do a simulate to run it through that ingest pipeline, it processes the event as expected with no errors. - If I pull the original event out of the `error.message` field and do the same thing, I also get no errors. I haven't attempted to preserve the original event yet to test that. > [@](#): > > Oh this is from serverless forwarder... hmm not sure if there is a preserve event or not... we can edit the pipeline and take out the remove if needed... As far as the preserve event goes, unless this integration is different than every other one that I've looked at, all that switch does is add a `preserve-original-event` tag to the event, which prevents the `remove` in the pipeline from being triggered. I use this same method in a number of custom pipelines I've written. Additionally, it's a `grok` failure, not a failure to delete a field, so it doesn't seem likely that this is the issue. I'll see what I can do about pulling some raw events. There's a possibility that this MAY be related to another issue we're seeing in that account. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 7, 2024, 6:50pm UTC](https://discuss.elastic.co/t/aws-cloudfront-ingest-pipeline-failing/352678/4 "2024-02-07T18:50:13Z")
Hi @DougR The point on preserving was to make sure the `event.original` is preserved If it is, fine, that is solved..... > [@DougR](#): > > THIS ACCOUNT ONLY to Elastic Cloud. I hear ya... so you're gonna have to figure out what is going on... 🙂 Something is different. Does EVERY message from that account fail? So what I would do is ... something along the following Simulate does not ALWAYS expose every issue... I would take the raw cloudfront message that seems to fail and actually post it as a documents Something like these to see what you see... . The first one should create a new data stream... see what you get The second will not use the template but will call the data stream. ```auto POST logs-aws.cloudfront-mytestnamepace/_doc { "message" : "cloudfront raw message" } POST my-generic-index/_doc?pipeline=logs-aws.cloudfront_logs-2.11.3 { "message" : "cloudfront raw message" } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 6, 2024, 6:50pm UTC](https://discuss.elastic.co/t/aws-cloudfront-ingest-pipeline-failing/352678/5 "2024-03-06T18:50:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.