Applying Elastic Common Scheman(ECS) in multi language environments

Elastic Observability Logs
1

Hello,

We are operating Filebeat on about 2,000 Machines and 2~30 K8S Clusters.
Also we accept ECS as the structured logging format standard for our company.
Currently, we applied ECS layout to services that are written in Java, Node.JS.

We want to apply ECS Loging layout to services that are wirtten in .Net and C++.
Each case have issue for applying ECS.

  1. C++ : No library on this language on Github
    Question : Do you have any plan to support C++ log layout?

  2. .Net : Value of log.level filed is different from each language.
    Question : Do you have any recommendation on this environment

    • Problem Details
      • Value of log.level
        • Java : INFO, WARN, ERROR
        • .Net : Info, Warn, Error
      • The default index template created by Filebeat defines 'log.value' as keyword type without normalizer
        • This means we have to change our previous query string as follow
          • before : labels.serverGroup : dummyService AND log.level : (WARN OR ERROR)
          • after : labels.serverGroup: dummyService AND log.level : (WARN OR Warn OR ERROR OR Error)
      • So, we modifed the index template.(Add lowercase normalizer to log.level field)
        • Then we can search logs with the previous query.
        • But, when we want to get like log count of each log level, so use aggregation request, the result is as follow
// Request
POST filebeat-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "query_string": {
            "analyze_wildcard": true,
            "query": "*"
          }
        }
      ]
    }
  },
  "aggs": {
    "3": {
      "terms": {
        "field": "log.level",
        "size": 10,
        "order": {
          "_key": "asc"
        },
        "min_doc_count": 1
      }
    }
  }
}

// Response
{
  "took" : 31,
  "timed_out" : false,
  "_shards" : {
    "total" : 60,
    "successful" : 60,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 77,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "3" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "INFO",
          "doc_count" : 72
        },
        {
          "key" : "WARN",
          "doc_count" : 2
        },
        {
          "key" : "warn",
          "doc_count" : 3
        }
      ]
    }
  }
}
2

Hi and thanks for your question!

We currently don't have plans for that.
Which C++ logging frameworks are you using?

The lowercase normalizer sounds like the way to go for your use case.

Aggregations should return the normalized value, see normalizer | Elasticsearch Guide [8.11] | Elastic.

Maybe you have aggregated over the wrong field or the mapping has not been applied for the index you're aggregating over. Remember: you can't change the mapping of existing indices, only new ones.

It can get a bit more complex if not only the casing differs, such as WARN vs WARNING. In that case, you may want to use an ingest node pipeline to normalize the values.

2 Likes
3

Hi,

Thank you for the quick answer.

  1. We are using Log4Cpp(http://log4cpp.sourceforge.net/)

  2. Yor are right about the aggregation with normalizated value.
    The problem is that i was using wildcard index whan search.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.