I'm currently working on a beat. To make it work well in an environment with quite a few logs, I would like to gather as much metrics as possible and expose them to "Prometheus". I saw people working on sending metrics to an Elasticsearch cluster, but in my environment I use "Prometheus" to monitor services. I would like to use those metrics to "improve" my setup: I see elasticsearch consuming a lot of CPU and memory.
Logs are sent via TCP linewise (EOL:
\r\n) to a TCP-server, which parses the lines it receives using "grok" and publishes them to Elasticsearch via
[Appliance] -> TCP -> [TCP-Server] | [libbeat] -> Elasticsearch
- Is there a public API I can use to get access to captured metrics (including those generated by libbeat)?
- Are there any guidelines to access metrics?
- Is there any documenation about how to use those metrics?
- Count of lines received via TCP
- Total bytes received via TCP
- Duration till published
- Count of lines parsed
- Count of events in queue before published
- Count of events published by libbeat