Hello,
am I seeing this correctly that the Beat-templates sometimes do not stick to the ECS? For example I wanted to prepare a mapping template for our ELK-Stack and so I merged auditbeat, filebeat and winlogbeat mappings together via a short python script and wanted to combine it with the ECS (because here everything goes through some logstash-buffer/-filter combination) so all possible fields are mapped in proper way. My plan is to create one index for beats and one for other stuff that is mostly syslog-filtered stuff.
Unfortunately the mapping gets a mess by just merging the "mappings", because some fields from the Beats use alias-fields and some not. Also there are fields that have the same name but in one beat-template there are also some nested fields below.
For example the "source"-field. As far as I can see in Filebeat it's a single field, in Auditbeat and ECS-template it's a nested field.
# filebeat
"source": {
"ignore_above": 1024,
"type": "keyword"
},
# ecs
"source": {
"properties": {
"address": {
"ignore_above": 1024,
"type": "keyword"
},
"as": {
"properties": {
"number": {
"type": "long"
},
"organization": {
"properties": {
"name": {
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
# auditbeat
"source": {
"properties": {
"address": {
"ignore_above": 1024,
"type": "keyword"
},
"as": {
"properties": {
"number": {
"type": "long"
},
"organization": {
"properties": {
"name": {
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
"bytes": {
"type": "long"
},
"domain": {
"ignore_above": 1024,
"type": "keyword"
},
[...shortened...]
Also the fields that use aliases give me a headache
# filebeat & winlogbeat
"container": {
"properties": {
"id": {
"path": "docker.container.id",
"type": "alias"
},
"image": {
"properties": {
"name": {
"path": "docker.container.image",
"type": "alias"
}
}
},
"name": {
"path": "docker.container.name",
"type": "alias"
}
}
},
# ecs
"container": {
"properties": {
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"image": {
"properties": {
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"tag": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"labels": {
"type": "object"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"runtime": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
It would be awesome if that gets fixed some day so everything really keeps to ECS. Also ECS uses:
"mappings": {
"_doc": {
and beat-templates
"mappings": {
"doc": {
which one should you use?
