How does elasticsearch index json

[Zeeshan_Alam]

We are trying to push log into elastic which adhere to Elastic Common Schema (ECS) .

Does elastic index data differently for the logs shown below. Both are having same data just the JSON format is different.

Format 1

{
  "log.level": "INFO",
  "log.logger": "org.elasticsearch.bootstrap.Bootstrap",
  "log.file.path": "/var/log/fun-times.log"
}

Format 2

{
  "log": {
    "level": "INFO",
    "logger": "org.elasticsearch.bootstrap.Bootstrap",
    "file": {
      "path": "/var/log/fun-times.log"
    }
  }
}

Ref: Log Fields | Elastic Common Schema (ECS) Reference [1.11] | Elastic

[kgeller (Kylie Geller)]

There is no difference to Elasticsearch between those two formats.

See Questions and Answers | Elastic Common Schema (ECS) Reference [1.12] | Elastic

[stephenb (Stephen Brown)]

@kgeller Welcome to the Community!

@Zeeshan_Alam Just to add on a bit to what @kgeller said.

As was said, The 2 Formats are Indexed exactly the same, meaning how the fields are stored and then queried are exactly this same but there are some subtle difference

1. And it may be obvious but the _source documents will remain different.... with the "." notation the source will not be automatically converted from Format 1 to Format 2 (see below)

2. IF you wanted to ingest those documents and use an ingest pipeline to operate on them (change, set, mutate etc) ingest pipelines will not work on Format 1 See Here unless you use the dot_expander processor. I bring this up because it can cause confusion sometime when people try to use ingest pipeline with documents with the "." notations, ingest pipeline are executed pre-index time so they work on the source document... or in the case with format 1... will not work

GET discuss/_search
{
  "fields": [
    "*"
  ]
}

Result: Note the _source is unchanged for each but the indexed fields are equivalent.

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "discuss",
        "_type" : "_doc",
        "_id" : "eIAcMnwBSSVLweAdY8S_",
        "_score" : 1.0,
        "_source" : {
          "log.level" : "INFO",
          "log.logger" : "org.elasticsearch.bootstrap.Bootstrap",
          "log.file.path" : "/var/log/fun-times.log"
        },
        "fields" : {
          "log.level.keyword" : [
            "INFO"
          ],
          "log.file.path" : [
            "/var/log/fun-times.log"
          ],
          "log.level" : [
            "INFO"
          ],
          "log.logger.keyword" : [
            "org.elasticsearch.bootstrap.Bootstrap"
          ],
          "log.logger" : [
            "org.elasticsearch.bootstrap.Bootstrap"
          ],
          "log.file.path.keyword" : [
            "/var/log/fun-times.log"
          ]
        }
      },
      {
        "_index" : "discuss",
        "_type" : "_doc",
        "_id" : "eYAcMnwBSSVLweAd2cTe",
        "_score" : 1.0,
        "_source" : {
          "log" : {
            "level" : "INFO",
            "logger" : "org.elasticsearch.bootstrap.Bootstrap",
            "file" : {
              "path" : "/var/log/fun-times.log"
            }
          }
        },
        "fields" : {
          "log.level.keyword" : [
            "INFO"
          ],
          "log.file.path" : [
            "/var/log/fun-times.log"
          ],
          "log.level" : [
            "INFO"
          ],
          "log.logger.keyword" : [
            "org.elasticsearch.bootstrap.Bootstrap"
          ],
          "log.logger" : [
            "org.elasticsearch.bootstrap.Bootstrap"
          ],
          "log.file.path.keyword" : [
            "/var/log/fun-times.log"
          ]
        }
      }
    ]
  }
}