[SOLVED] Many metrics to graphite via include_metrics but from @metadata

[simmel (Simon Lundström)]

I don't want to send our metrics fields which are many (ES complains that we have more than 1000 fields) and unwanted to elasticsearch.

Our previous approach was to use:

graphite {
  host => "graphite.domain.tld"
  fields_are_metrics => true
  timestamp_field => "@timestamp"
  include_metrics => [
    "^server\.",
    "^service\."
  ]
}

That doesn't avoid getting them into ES. So I thought about putting them all in the @metadata field but then I can't use any wildcard to get them all in.

graphite {
  host => "127.0.0.1"
  fields_are_metrics => true
  timestamp_field => "@timestamp"
  include_metrics => [
    "\[@metadata\]server\.",
    "\[@metadata\]service\."
  ]
}

doesn't work, obviously.

Any ideas?

[simmel (Simon Lundström)]

Hope this helps someone.

What we ended up doing is instead of:

if [@metadata][rsyslogd-pstats][name] =~ "^im(tcp|udp)" and [@metadata][rsyslogd-pstats][submitted] and [@metadata][rsyslogd-pstats][submitted] > 0 {
  mutate {
    add_field => {
      "server.linux_%{[@metadata][logsource_graph]}.rsyslog.%{[@metadata][rsyslogd-pstats][name]}.submitted" => "%{[@metadata][rsyslogd-pstats][submitted]}"
    }
    add_tag => ["graph"]
  }
}

we did:

if [@metadata][rsyslogd-pstats][name] =~ "^im(tcp|udp)" and [@metadata][rsyslogd-pstats][submitted] and [@metadata][rsyslogd-pstats][submitted] > 0 {
  clone {
    clones => ["graph"]
    add_field => {
      "server.linux_%{[@metadata][logsource_graph]}.rsyslog.%{[@metadata][rsyslogd-pstats][name]}.submitted" => "%{[@metadata][rsyslogd-pstats][submitted]}"
    }
    add_tag => ["graph"]
  }
}

which made it possible to send all events with the graph tag to the metrics system, in our case graphite, and else send it to elasticsearch, e.g:

if "graph" in [tags] {
  graphite {
    fields_are_metrics => true
    include_metrics => [
      "^server\.",
      "^service\."
    ]
  }
}

else {
  elasticsearch {
  }
} 

that way we don't get metrics extracted from the logs indexed in elasticsearch.