Applying Elastic Common Scheman(ECS) in multi language environments

[Bingu_Shim]

Hello,

We are operating Filebeat on about 2,000 Machines and 2~30 K8S Clusters.
Also we accept ECS as the structured logging format standard for our company.
Currently, we applied ECS layout to services that are written in Java, Node.JS.

We want to apply ECS Loging layout to services that are wirtten in .Net and C++.
Each case have issue for applying ECS.

1. C++ : No library on this language on Github
   Question : Do you have any plan to support C++ log layout?

2. .Net : Value of log.level filed is different from each language.
   Question : Do you have any recommendation on this environment

   • Problem Details
     • Value of log.level
       • Java : INFO, WARN, ERROR
       • .Net : Info, Warn, Error
     • The default index template created by Filebeat defines 'log.value' as keyword type without normalizer
       • This means we have to change our previous query string as follow
         • before : labels.serverGroup : dummyService AND log.level : (WARN OR ERROR)
         • after : labels.serverGroup: dummyService AND log.level : (WARN OR Warn OR ERROR OR Error)
     • So, we modifed the index template.(Add lowercase normalizer to log.level field)
       • Then we can search logs with the previous query.
       • But, when we want to get like log count of each log level, so use aggregation request, the result is as follow

// Request
POST filebeat-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "query_string": {
            "analyze_wildcard": true,
            "query": "*"
          }
        }
      ]
    }
  },
  "aggs": {
    "3": {
      "terms": {
        "field": "log.level",
        "size": 10,
        "order": {
          "_key": "asc"
        },
        "min_doc_count": 1
      }
    }
  }
}

// Response
{
  "took" : 31,
  "timed_out" : false,
  "_shards" : {
    "total" : 60,
    "successful" : 60,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 77,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "3" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "INFO",
          "doc_count" : 72
        },
        {
          "key" : "WARN",
          "doc_count" : 2
        },
        {
          "key" : "warn",
          "doc_count" : 3
        }
      ]
    }
  }
}

[felixbarny (Felix Barnsteiner)]

Hi and thanks for your question!

@Bingu_Shim

Do you have any plan to support C++ log layout?

We currently don't have plans for that.
Which C++ logging frameworks are you using?

@Bingu_Shim

Value of log.level filed is different from each language.
Question : Do you have any recommendation on this environment

The lowercase normalizer sounds like the way to go for your use case.

Aggregations should return the normalized value, see normalizer | Elasticsearch Guide [8.11] | Elastic.

Maybe you have aggregated over the wrong field or the mapping has not been applied for the index you're aggregating over. Remember: you can't change the mapping of existing indices, only new ones.

It can get a bit more complex if not only the casing differs, such as WARN vs WARNING. In that case, you may want to use an ingest node pipeline to normalize the values.

[Bingu_Shim]

Hi,

Thank you for the quick answer.

1. We are using Log4Cpp(http://log4cpp.sourceforge.net/)

2. Yor are right about the aggregation with normalizated value.
   The problem is that i was using wildcard index whan search.