Large number of Empty Fields in Discover Tab

Elastic Stack Beats
1

Hello all,

I am trying reduce the noise when using Kibana and I have noticed that within the discover tab and when trying to create a lens there are 5700+ fields which are always displayed as empty. I have tried to go through the process to build a custom fields.yml file, change the metricbeat.yml to allow this to be loaded in and then delete the old index however, I still see all these fields despite removing them from what I believe should have been what creates the template for the datastream/index.

In my metricbeat.yml I have added the following:
setup.template.enabled: false

My fields.yml file is down from 32k lines to about 16k lines I have primarily kept the following keys:

key: ecs
key: beat
key: docker
key: host
key: process
key: common
key: system

Is this a known bug I did find this: [UnifiedFieldList] Fields are not marked as Empty in Discover sidebar · Issue #147124 · elastic/kibana · GitHub

But it doesn't appear to be quite my issue. But I am getting a little close to the end of my rope for fixing this. How do I limit or stop these from being imported into the datastream/index template?

2

Hi

I'm new to using Elasticsearch and Kibana. But I have the same/similar issue. There are fields in my index that I know I will never need.

aws.billing-AmortizedCost.unit
aws.billing-BlendedCost.amount
aws.billing.BlendedCost.unit
aws.billingCurrency
aws.billing.end_date
aws.billing.EstimatedCharges
aws.billing-group_definition.key
aws.billing.group_definition.type
aws.billing.NormalizedUsageAmount-amount
aws.billing.NormalizedUsageAmount-unit
aws.billing.ServiceName
aws.billing-start_date
aws.billing.UnblendedCost.amount
aws.billing-lJnbIendedCost-unit
aws.billing-lJsageQuantity.amount
aws.billing.UsageQuantity.unit
aws.cloudwatch.namespace
aws.dynamodb.metrics.AccountMaxReads. max
aws.dynamodb.metrics.AccountMaxTabIeLevelReads. max
aws.dynamodb.metrics.AccountMaxTabIeLevelWrites.max
aws.dynamodb.metrics.AccountMaxWrites.max
aws.dynamodb.metrics.AccountProvisionedReadCapacitylJtiIization.avg
aws.dynamodb.metrics.AccountProvisionedWriteCapacityUtilization.av
aws.dynamodb.metrics.ConditionalCheckFailedRequests.sum
aws.dynamodb.metrics.ConsumedReadCapacityUnits.avg
aws.dynamodb.metrics.ConsumedReadCapacityUnits.sum

and a lot more (7000+). How can I get rid of those fields?

Thanks for your help

3

I did find a solution to this issue but it's a little convoluted. If you go to Kibana then Index Management, find the DataStream/Index you want to change. From there you find the template that is used to create that DataStream/Index. Copy that Template and name it something else. Edit the template by clicking the little edit button.

image
image1628×63 7.49 KB

From there go to Mappings shown here:

image
image1331×718 44.3 KB

Remove any mappings you don't want by deleting them. Save that template but also update the priority setting to something higher than 150.

Now... you need to delete your old Index or DataStream for this to work. YOU WILL LOSE THE DATA. If you want to migrate it - I don't know how. But I deleted my old Datastream/Index and then reinitialized it on my remote node, since the new template will have a higher priority it will take precedence over the default template and will be used. Now I have a much more manageable number of fields.

image

It's still really dumb that even if all the fields are empty across the whole Index you can't hide them and imo a huge miss on the UI/UX.

1 Like
4

GREAT, that helped me a lot. Now I need to find out where the template files are saved or how I can export them.

5

I believe the template files should be generated from the fields.yml file and updating this and then doing the metricbeat setup should generate the initial template however, I found it much easier to simply create another template via the gui in Kibana and then just attach that new template with a higher priority to the newly created Index/Datastream.