# Is it possible to process AWS Cloudwatch logs using the nginx module?

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 9, 2021, 12:47pm UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438 "2021-06-09T12:47:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [June 9, 2021, 12:47pm UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438/1 "2021-06-09T12:47:04Z")

</div>

I have a service deployed to ECS which is basically an nginx instance.

I want to ingest the logs using filebeat - I can do this using the aws cloudwatch input type, but it doesn't grok the message field like the nginx module does.

Is it somehow possible to push aws cloudwatch logs through the nginx pipeline?

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [June 10, 2021, 8:09am UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438/2 "2021-06-10T08:09:53Z")

</div>

I've tried munging the pipeline from the nginx module into the aws cloudwatch one and then forcing the pipelines to be updated, like so:

```auto
filebeat setup --pipelines --modules aws/cloudwatch

```

The command ran successfully, but ultimately it didn't work.

Actually, most of the data I want out of these logs I can get from a `dissect` processor, but I'd be loath to miss out on the extra data that `geoip` offers...

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [June 10, 2021, 3:04pm UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438/3 "2021-06-10T15:04:29Z")

</div>

It looks like I should be able to specify the nginx pipeline in the `output.elasticsearch` part of my config...but when I try this it doesn't seem to work.

For example:

```auto
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["${MONITORING_SERVER}"]
  pipelines:
    - pipeline: "filebeat-7.12.0-awscloudwatch-nginx-access-pipeline"
      # Ignore filebeat log stream
      when.not:
        contains:
          message: "[monitoring]"

```

"It doesn't seem to work" as in the new fields that should be generated by the pipeline aren't being created.

There are no error fields either to indicate that something went wrong.  
There are no errors in the filebeat logs (although I wasn't expecting any?)

Any idea how I can debug this?

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [June 10, 2021, 3:23pm UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438/4 "2021-06-10T15:23:16Z")

</div>

When I test the pipeline using a document indexed from awscloudwatch using filebeat, the result looks good...so why isn't the `output.elasticsearch` pipelines setting working?

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [June 10, 2021, 3:36pm UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438/5 "2021-06-10T15:36:14Z")

</div>

Looking at this: [Ingest pipeline not working for filebeat](https://discuss.elastic.co/t/ingest-pipeline-not-working-for-filebeat/228015)  
it seems that because I'm using the aws-cloudwatch module I can't specify another pipeline in `output.elasticsearch`.

In order to apply their solution, though, I'll need to setup my own index with index settings so that it doesn't affect other filebeat-ingested stuff.

---

<div class="post-metadata">

**Author:** ![ndtreviv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndtreviv/32/22494_2.png) [@ndtreviv](https://discuss.elastic.co/u/ndtreviv)\
**Post date:** [June 10, 2021, 8:34pm UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438/6 "2021-06-10T20:34:48Z")

</div>

OK, for any other poor sods who came here trying to do the same thing, here's how I got it working:

First, you need to find the `filebeat-[version]-nginx-access-pipeline` and Clone it.  
Name it something like: `filebeat-[version]-awscloudwatch-nginx-access-pipeline`.

When editing the clone, change the failure processors to remove the `Set` processor and add a `Pipeline` processor targeting the `filebeat-[version]-nginx-error-pipeline`. I didn't put any conditions in, but you could, if you're confident with how that works.

Next, update your filebeat config to include:

```auto
setup.template.settings:
   ...
  index.final_pipeline: "filebeat-[version]-awscloudwatch-nginx-access-pipeline"

```

Now, if you're like me, and you're working with an existing filebeat index template that you now can't edit, or your pushing stuff in from filebeats across your arch and you don't want to risk any nasty side-effects, you can configure a new index template like so:

```auto
setup.template.name: filebeat-awscloudwatch-nginx
setup.template.pattern: filebeat-awscloudwatch-nginx-*
setup.ilm.rollover_alias: filebeat-awscloudwatch-nginx

...

output.elasticsearch:
   ...
  index: "filebeat-awscloudwatch-nginx-%{+yyyy.MM.dd}"

```

Now, when cloudwatch logs get indexed into that index by filebeat's awscloudwatch module, they'll go through the `filebeat-[version]-awscloudwatch-nginx-access-pipeline` first, when the `grok` fails, it'll send it onto the `filebeat-[version]-awscloudwatch-nginx-error-pipeline` and you'll get all the lovely enrichments that you always wanted.

YOU ARE WELCOME  
🖐  
🎤

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2021, 10:34pm UTC](https://discuss.elastic.co/t/is-it-possible-to-process-aws-cloudwatch-logs-using-the-nginx-module/275438/7 "2021-07-08T22:34:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.