Beat templates do not stick to ECS?

Elastic Stack Beats
, ,
1

Hello,
am I seeing this correctly that the Beat-templates sometimes do not stick to the ECS? For example I wanted to prepare a mapping template for our ELK-Stack and so I merged auditbeat, filebeat and winlogbeat mappings together via a short python script and wanted to combine it with the ECS (because here everything goes through some logstash-buffer/-filter combination) so all possible fields are mapped in proper way. My plan is to create one index for beats and one for other stuff that is mostly syslog-filtered stuff.
Unfortunately the mapping gets a mess by just merging the "mappings", because some fields from the Beats use alias-fields and some not. Also there are fields that have the same name but in one beat-template there are also some nested fields below.

For example the "source"-field. As far as I can see in Filebeat it's a single field, in Auditbeat and ECS-template it's a nested field.

    # filebeat              
              "source": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              
    # ecs
      "source": {
        "properties": {
          "address": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "as": {
            "properties": {
              "number": {
                "type": "long"
              },
              "organization": {
                "properties": {
                  "name": {
                    "fields": {
                      "text": {
                        "norms": false,
                        "type": "text"
                      }
                    },
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              }
            }
          },

    # auditbeat
      "source": {
        "properties": {
          "address": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "as": {
            "properties": {
              "number": {
                "type": "long"
              },
              "organization": {
                "properties": {
                  "name": {
                    "fields": {
                      "text": {
                        "norms": false,
                        "type": "text"
                      }
                    },
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              }
            }
          },
          "bytes": {
            "type": "long"
          },
          "domain": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          [...shortened...]

Also the fields that use aliases give me a headache

    # filebeat & winlogbeat            
        "container": {
          "properties": {
            "id": {
              "path": "docker.container.id",
              "type": "alias"
            },
            "image": {
              "properties": {
                "name": {
                  "path": "docker.container.image",
                  "type": "alias"
                }
              }
            },
            "name": {
              "path": "docker.container.name",
              "type": "alias"
            }
          }
        },
              
    # ecs
        "container": {
          "properties": {
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "image": {
              "properties": {
                "name": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "tag": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "labels": {
              "type": "object"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "runtime": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },

It would be awesome if that gets fixed some day so everything really keeps to ECS. Also ECS uses:

      "mappings": {
         "_doc": {

and beat-templates

      "mappings": {
         "doc": {

which one should you use?

2

You should really keep these in their own indices. Simply dumping them all into the single one will likely lead to a mapping explosion, which is not a great thing to deal with.

3

ok. thx for clarification. :slight_smile:

but imho if elastic is proposing to use ECS they really should fix this so the mappings are really everywhere the same - justmy2cents

4

Oh for sure, there's probably a few discrepancies that we are yet to sort out.
If you find some, please create a GitHub issue so we can get them resolved.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.