java.lang.IllegalArgumentException: Fielddata when upgrading to 7.0.0

Elastic Observability APM
1

Hi! I run 6.6.0 in several envs ok and today was trying to test a 7.0.0 setup of elastic APM before upgrading those envs but I'm facing some problems, hope anyone can help.

My setup is as follows (done with 6.6.0 successfully):

apm-servers -> logstash -> elasticsearch

As stated in https://www.elastic.co/guide/en/apm/server/master/logstash-output.html:

For this configuration, you must load the index template into Elasticsearch manually because the options for auto loading the template are only available for the Elasticsearch output.

So what I do is:
1- export the template definition in the apm-server:

apm-server export template > apm-template-6.6.0.json

2- in logstash, pipeline for apm looks as follow:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts           => [ 'my-elasticsearch:9200' ]
    index           => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+YYYY.MM.dd}"
    manage_template => true
    template_name   => "apm"
    template        => "/etc/logstash/templates/apm-template-6.6.0.json"
  }
}

While doing same steps for 7.0.0 I cannot see the services in APM in kibana and elasticsearch is logging following errors:

"Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [service.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.",
"at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:767) ~[elasticsearch-7.0.0.jar:7.0.0]",
"at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116) ~[elasticsearch-7.0.0.jar:7.0.0]"

Any idea?

2

Your process and configuration look great except you'll need to update your template to use the 7.0.0 template export - there were significant field changes introduced in 7.0.0 that account for the issue you have encountered.

3

Hi @gil thanks for your response.
Yes that's what I mean, I repeated the steps including the export of template after installing apm-server 7.0.0 but now getting above errors.
note I'm not reusing the elasticsearch cluster but starting up a new one completely

4

OK I saw something in elasticsearch that doesn't seem correct:

{"type": "server", "timestamp": "2019-04-22T15:51:02,122+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "docker-cluster", "node.name": "bc34ff2855d8", "cluster.uuid": "v6EAYWCoSBe04WwYtt7skg", "node.id": "aDQ5gEwRQXq3lWZX5f9w-Q", "message": "adding template [apm] for index patterns [logstash-*]" }

so seems is not pushing the apm template correctly for index patterns apm-* but applying default logstash-*

however my pipeline definition is clear:

output {
elasticsearch {
hosts => [ <%= @elasticsearch_hosts.map{ |item| "'#{item}'" }.join(', ') -%> ]
index => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+YYYY.MM.dd}"
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates/<%= @template_file %>"
template_name => "apm"
}
}

and the template exported from apm-server starts like:

{
  "index_patterns": [
    "apm-7.0.0-*"
  ],
  "mappings": {
    "_meta": {
      "beat": "apm",
      "version": "7.0.0"
    },
    "date_detection": false,

What am I doing wrong?

Thanks

5

ok seems I'm facing https://github.com/elastic/logstash/issues/10687

6

Good find. I've confirmed locally that ilm_enabled => false can work around this - is that an option for you?

7

yes just confirmed myself as well that disabling ilm works.
Yep I'll use that option while is fixed properly.
Thanks!

1 Like
8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.