# Help with reducing mapping **URL:** \ **Category:** Kibana\ **Created:** [January 27, 2021, 4:15pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394 "2021-01-27T16:15:10Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![DavidoffX](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@DavidoffX](https://discuss.elastic.co/u/DavidoffX)\ **Post date:** [January 27, 2021, 4:15pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/1 "2021-01-27T16:15:10Z")
Hi All. I'm new to the ELK stack, and am hoping I'm not trying to re-invent the wheel. Steep learning curve at the moment and am just after some help and guidance really. I'm using FileBeat to send logs out from our Kubernetes cluster. I was originally outputting the logs directly into elasticsearch, but found that having everything outputting into one single index wasnt really working how we wanted. After some research and the following post which I found really usfull I now send the logs through filebeat rather than elasticsearch, this is working nicely. > [@Separate indexes for each kubernetes namespace](https://discuss.elastic.co/t/separate-indexes-for-each-kubernetes-namespace/169131/4): > > Hi @Badger, Thanks for the hint and recommendations. I was able to get the namespace(s) as index name(s). In case someone is looking for config Here's how the fields looks like when queried from kibana. { "\_index": "test-1", "\_type": "doc", "\_id": "upAUFGkB4pTTTbuh8MFh", "\_version": 1, "\_score": 0, "\_source": { "host": { "name": "filebeat-kztmv" }, "source": "/var/lib/docker/containers/41ce86fd65c4de705ef65331f79cc946033140a2379ead7ab3443e587f73bb7a/41ce86fd65c4d… But.... The mapping for the indexes in elasticsearch are way more heavyweight than what we need. Ive followed this guide around changing a mapping, but when I repoint the alias using the "atomic" step, although the mapping that Ive created is correct, none of the documents from the original index are present in the new index with simpler map. > **[Changing Mapping with Zero Downtime](https://www.elastic.co/blog/changing-mapping-with-zero-downtime)** When I use the re-index api to move documents in, the mapping in the new index is ignored and all the original indexes are copying over. > **[Reindex API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)** Basically all I really need to achieve is that when we use the "Discover link in Kibana, we dont need to see all the kubernetes fieds and other metadata, we are just wanting to see info about the source of the request, and the "message" fields from the mapping. ---
**Author:** ![DavidoffX](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@DavidoffX](https://discuss.elastic.co/u/DavidoffX)\ **Post date:** [January 27, 2021, 8:41pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/2 "2021-01-27T20:41:13Z")
I belive that what I'm trying do may be achieveable quite simply by using a logstash filter. My logstash.conf looks like this ``` input { beats { port => 5044 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] } } output { elasticsearch { hosts => "elasticsearch:9200" manage_template => false index => "%{[kubernetes][namespace]}" document_type => "%{[@metadata][type]}" } } ``` But the mappings of the resulting indexes in eleasticsearch are still enormous like: ``` { "mappings": { "properties": { "@timestamp": { "type": "date" }, "@version": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" }, "agent": { "properties": { "ephemeral_id": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" }, "hostname": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" }, "id": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" }, "name": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" }, "type": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" }, "version": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" } } }, "cloud": { "properties": { "account": { "properties": { "id": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" } } }, "availability_zone": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" }, "image": { "properties": { "id": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" } } }, "instance": { "properties": { "id": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" } } }, "machine": { "properties": { "type": { "fields": { "keyword": { "ignore_above": 256, "type": "keyword" } }, "type": "text" } } }, ``` etc. etc. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 27, 2021, 9:38pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/3 "2021-01-27T21:38:05Z")
If filebeat is adding fields you do not want then you may be able to disable the [processors](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html) that are adding them. If there are specific top-level fields you want to remove then a [prune](https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html) filter may help. If you blacklist a top-level object ([agent] for example) then all of the fields within it will also be removed from the event. If there is a specific set of fields you want to keep then you also have the option of creating a mapping that defines them and turning off [dynamic mappin](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html)g. ---
**Author:** ![DavidoffX](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@DavidoffX](https://discuss.elastic.co/u/DavidoffX)\ **Post date:** [January 27, 2021, 10:06pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/4 "2021-01-27T22:06:16Z")
Hi @Badger, Thanks for the reply. You're right there were processors defined in filebeat.yml that I did not need. I'm also looking at the prune filter, thanks for the tip. As I mentioned I have already decided what the mapping should look like, so I think removing the unwanted processors to reduce the overall size of the payload from filebeat combined with turning off dynamic mapping, and defining my own mappings as I have already tried to do should get me to where I want to be. I've had a quick look around to try to find out how to disable dynamic mapping, but am still unsure how to achieve this exactly? Thanks, David ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 27, 2021, 10:19pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/5 "2021-01-27T22:19:25Z")
Read through [this](https://stackoverflow.com/questions/33263673/disable-dynamic-mapping-creation-for-only-specific-indexes-on-elasticsearch) thread. ---
**Author:** ![DavidoffX](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@DavidoffX](https://discuss.elastic.co/u/DavidoffX)\ **Post date:** [January 28, 2021, 1:20pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/6 "2021-01-28T13:20:13Z")
Hi @Badger, Again, thankyou for the reply. I've hit a bit of a brick wall with this. So I'm unable to blacklist the "kubernetes" name as this breaks the indexing as im using `index => "%{[kubernetes][namespace]}"` in my output, and looking at the plugin dosc it only supports excluding top level fields. Fine, but then I cant seem to exclude any additional fields within the "kubernetes" property other than namespace. Ive read the stack overflow article you linked to numerous times. Setting dynamic to "strict" means no documents are indexed for obvious reasons (they contain undeclared fields from the mapping), and setting it to false doesnt seem to work - new fields are still created dynamically within the mapping??? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 28, 2021, 5:30pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/7 "2021-01-28T17:30:16Z")
> [@DavidoffX](#): > > setting it to false doesnt seem to work - new fields are still created dynamically within the mapping You should ask a question about this in the elasticsearch forum. ---
**Author:** ![DavidoffX](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@DavidoffX](https://discuss.elastic.co/u/DavidoffX)\ **Post date:** [January 29, 2021, 12:35pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/8 "2021-01-29T12:35:17Z")
@Badger I've done some more testing and I was incorrect about the mapping changing. Having false as the value for dynamic **does** actually prevent the mapping changing, but the data seen in kibana appears to be the full \_source of the data from logstash into elasticsearch. The values are persisted in elastic even though they arent indexed, or changes made to the mapping. So am back to needing to find a way to reduce the payload from logstash to elasticsearch. I need to remove subkeys from the kubernetes top level field. As discussed previously this isnt possible with the prune filter, but I'm thinking it **should** be possible, maybe using ruby, like you recommended here: > [@How to dynamically move nested key value to root level](https://discuss.elastic.co/t/how-to-dynamically-move-nested-key-value-to-root-level/180006): > > This is my logstash config file: input { http { id =\> bulkHttpInput port =\> 8088 additional\_codecs =\> {"application/json" =\> "es\_bulk"} codec =\> es\_bulk } } filter { mutate { remove\_field =\> ["headers"] } } output { elasticsearch { id =\> elasticOutputOfHttp index =\> "%{[@metadata][\_index]}" document\_type =\> "%{[@metadata][\_type]}" document\_id =\> "%{[doc][docID]}" doc\_as\_upsert =\> "true" } } and the output is like : { "host" =\> "127.0.0.1", "@… ---
**Author:** ![DavidoffX](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@DavidoffX](https://discuss.elastic.co/u/DavidoffX)\ **Post date:** [January 31, 2021, 7:33pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/9 "2021-01-31T19:33:46Z")
Turned out this was me not understanding how elasticsearch actually works. Link here for reference -\> [Dynamic mapping setting not honoured](https://discuss.elastic.co/t/dynamic-mapping-setting-not-honoured/262551) ---
**Author:** ![DavidoffX](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@DavidoffX](https://discuss.elastic.co/u/DavidoffX)\ **Post date:** [January 31, 2021, 7:36pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/10 "2021-01-31T19:36:35Z")
The solution I settled on was to extract the required properties from the subkeys, set them as top level props in the event (using ruby filter), and then after drop the bulky data I didnt need from the event using its top level key in the prune filter, hence it never makes it to eleasticsearch ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 28, 2021, 7:36pm UTC](https://discuss.elastic.co/t/help-with-reducing-mapping/262394/11 "2021-02-28T19:36:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.