OK, for any other poor sods who came here trying to do the same thing, here's how I got it working:
First, you need to find the
filebeat-[version]-nginx-access-pipeline and Clone it.
Name it something like:
When editing the clone, change the failure processors to remove the
Set processor and add a
Pipeline processor targeting the
filebeat-[version]-nginx-error-pipeline. I didn't put any conditions in, but you could, if you're confident with how that works.
Next, update your filebeat config to include:
Now, if you're like me, and you're working with an existing filebeat index template that you now can't edit, or your pushing stuff in from filebeats across your arch and you don't want to risk any nasty side-effects, you can configure a new index template like so:
Now, when cloudwatch logs get indexed into that index by filebeat's awscloudwatch module, they'll go through the
filebeat-[version]-awscloudwatch-nginx-access-pipeline first, when the
grok fails, it'll send it onto the
filebeat-[version]-awscloudwatch-nginx-error-pipeline and you'll get all the lovely enrichments that you always wanted.
YOU ARE WELCOME