ElasticAPM Unknown Field Mapping

[happyinteger]

Kibana version: 7.11.1

Elasticsearch version: 7.11.1

APM Server version: 7.9.1

APM Agent language and version: rum-js 5.6.3

Browser version: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 OPR/75.0.3969.171

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Download page

Fresh install or upgraded from other version? Fresh Install

I'm using ElasticAPM server with rum-js agent. My application can successfully send data to the APM server. Although some fields have already assigned a type, there are some other fields which are "unknown field" and be able to filter them, I have to do a mapping manually. Using this path in Kibana UI: Stack Management --> Index Management --> Index Templates --> Legacy Index Templates --> Edit Template (apm-7.9.1)

The data that I received from my application like below

Unknown fields:

After mapping process the unknown fields are fixed according to types that I'm selecting. But I wonder why I have to do this, there are another fields in span object and despite I didn't any mapping, they are not "unknown field" type. I want to find out what's the logic behind this mapping process. Thanks in advance.

[axw (Andrew Wilkins)]

Hi @happyinteger, welcome to the forum!

But I wonder why I have to do this, there are another fields in span object and despite I didn't any mapping, they are not "unknown field" type. I want to find out what's the logic behind this mapping process. Thanks in advance.

The reason we don't index every field is because this comes with a cost both in terms of storage size and ingestion speed in Elasticsearch.

Now that Elasticsearch supports runtime fields, we're considering making these less commonly searched fields runtime fields. This would make them searchable, but more slowly than if they were indexed at ingestion time.

[happyinteger]

Hello @axw ,
Thank you for your explanation.