ELK stack and additional CloudWatch output

Elastic Stack Logstash
1

I would like to output all my logs to Elastic Search and when certain conditions are met, send them as metrics to Cloud Watch, which should take care of alerting me.

They way I understand it is, that I have to add a CW_metrics field in my event that contains the metric name for cloudwatch. However, I don't want this metric to appear in Elastic Search, only for the Cloudwatch output. What would be a good way to achieve this?

Example: php error log contains a line with an Exception, that I can match using Logstash filters. In this case I want to send Cloudwatch a Count of 1 for the metric "php_exception_count".

But I just can't wrap my head around how to do this... Any examples or ideas?

2

If I understand correctly, you want to use conditionals based on that tag - https://www.elastic.co/guide/en/logstash/current/configuration.html#conditionals

3

Yes, I got it working!

What I didn't know is, that the Cloudwatch plugin can also reference a metadata field, so that the metric name i.e. is written to @metadata[metric_name] and thus not sent to Elasticsearch :smile:

Also, when this field is not present, the cloudwatch plugin will not send a metric.

4

Can you share your config, it may help others :slight_smile:

5

This is what I've done:

  # general metric for indexed logs
  cloudwatch {
    access_key_id => "access_key"
    secret_access_key => "secret_key"
    region => "eu-central-1"
    timeframe => "10s"
    #field_metricname => "[@metadata][metricname]"
    metricname => "IndexedLogCount"
    namespace => "ELK"
  }

  # log count metric
  if "log_count_metric" in [tags] {
    cloudwatch {
      access_key_id => "access_key"
      secret_access_key => "secret_key"
      region => "eu-central-1"
      timeframe => "10s"
      metricname => "LogCount"
      field_dimensions => "[@metadata][metric_log_count_dimensions]"
      namespace => "ELK"
    }
  }