Confusion, dot-notation vs. sub-objects

anton-johansson · October 8, 2021, 8:07am

We're a bit confused about the specification regarding dot-notation vs. sub-objects. If you look in the documentation, it is specified as dot-notation, see here:

This is implemented as dot-notation in the Java/Log4j2-implementation, see here:

github.com

elastic/ecs-logging-java/blob/fb287cc003ca48cbb9440243b50d79031e2dfa4f/ecs-logging-core/src/main/java/co/elastic/logging/EcsJsonSerializer.java#L52

    
      
          }
          
          
public static void serializeObjectStart(StringBuilder builder, long timeMillis) {
              builder.append('{');
              builder.append("\"@timestamp\":\"");
              TIMESTAMP_SERIALIZER.serializeEpochTimestampAsIsoDateTime(builder, timeMillis);
              builder.append("\", ");
          }
          
          
public static void serializeEcsVersion(StringBuilder builder) {
              builder.append("\"ecs.version\": \"1.2.0\",");
          }
          
          
public static void serializeObjectEnd(StringBuilder builder) {
              removeIfEndsWith(builder, ",");
              builder.append('}');
              builder.append('\n');
          }
          
          
public static void serializeLoggerName(StringBuilder builder, String loggerName) {
              if (loggerName != null) {

However, in the JavaScript/Pino implementation, it's implemented as sub-objects:

github.com

elastic/ecs-logging-nodejs/blob/6e0566dbd937b137250c0305be00787b8649dd80/loggers/pino/index.js#L158

    
      
            res,
            err,
            ...ecsObj
          } = obj
          
          
// https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html
          // For "ecs.version" we take a heavier-handed approach, because it is
          // a require ecs-logging field: overwrite any possible "ecs" value from
          // the log statement. This means we don't need to spend the time
          // guarding against "ecs" being null, Array, Buffer, Date, etc.
          ecsObj.ecs = { version }
          
          
if (apm) {
            // A mis-configured APM Agent can be "started" but not have a
            // "serviceName".
            if (apmServiceName) {
              // Per https://github.com/elastic/ecs-logging/blob/master/spec/spec.json
              // "service.name" and "event.dataset" should be automatically set
              // if not already by the user.
              if (!isServiceNameInBindings) {
                const service = ecsObj.service

... and can also be seen in examples here:

Which format is really the one to use? What happens if we use both layouts and put logs in the same Elasticsearch index. The mappings won't really look the same, will they?

Thanks in advance!

felixbarny · October 11, 2021, 6:52am

Hi and thanks for your question.

The logs itself use dotted and nested notation interchangeably. Before ingesting it to Elasticsearch, it's highly recommended to normalize to the nested notation. Either with Filebeat's expand_keys option (see Get started | ECS Logging Java Reference [1.x] | Elastic) or with an Elasticsearch ingest pipeline, using the dot_expander (see [Meta] Enable seamless ECS log onboarding for all log inputs · Issue #1454 · elastic/integrations · GitHub).

anton-johansson · October 11, 2021, 3:15pm

Allright, thanks! We'll try the expand_keys option.

system · November 8, 2021, 3:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Best Practices / Recommendations for Custom Objects Logs ecs-elastic-common-schema , dotnet	3	1452	June 29, 2021
Elastic Common Schema Java Implementation Elasticsearch	4	974	July 30, 2019
Handling dot notated fields Elasticsearch	5	3756	June 7, 2018
When to use "Object" field datatype vs flat fieldnames Elasticsearch ecs-elastic-common-schema	7	3263	March 4, 2020
Serilog EcsTextFormatter: Not what I expected Logs dotnet	2	947	March 15, 2021

Confusion, dot-notation vs. sub-objects

Related topics