I am in the process of understanding the Elastic product suite, and I don't fully understand 'Metrics'. I know I can send structured log events to Elasticsearch, and I can search and filter on specific fields that I added to the log events using Kibana. I can also build a visualization using those fields and their values.
But we don't retain the logs, and there is no such concept of sampling I could find. So log events are not the same as metrics, although log events can contain metric-like data (e.g. duration of a request: "GET /endpoint completed with status 200 in 345 ms").
When not using a regular log event, how do I use Elastic 'Metrics' to instrument my code and send "RequestDuration": 345, or "RequestSuccessful": +1?
I understand I can use the Elasticsearch client to send anything to Elasticsearch. I am using Serilog, a logging library for .NET which has a plugin to send structured logs to Elasticsearch.
According to your own learning materials on elastic.co, there is a difference between logs and metrics (and traces). They are different 'pillars of observability'. And Elastic has a product called 'Metrics'.
My questions are:
What makes something a metric?
How does an index that contains log events differ from an index that contains metrics?
How do I make sure that when I use an Elasticsearch client (indirectly via my logging library) to send a metric, it is persisted/handled as a metric event, and not as a log event?
Elasticsearch is a distributed search engine and data store. It is also capable of generating calculations or extract statistics from the data we store into it.
For Elasticsearch metrics and logs are just documents with different structure and contents.
Metric documents usually contain synthetic stats about what happened in a transaction, or a short period of time.
Usually numeric values and statuses, counters, values.
They are usually periodic. They have a timestamp and metadata to know from which they originated.
Logs documents usually have a structured or unstructured message you've collected from an application or a service, plus a timestamp and metadata to know from which they originated.
Usually there's a "message" field which contains the key information about the log line.
The difference between them are the mappings of the indices and the contents.
The default metric dashboards point to indices named metricbeat because Metricbeat is the tool we usually suggest to collect and send metrics.
The default log dashboard and UI point to indices named filebeat or logstash, but this is configurable.
The advantage is you can use Elasticsearch and the other Elastic products or clients (or even non-elastic tools or libraries) to build your solution.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.