Cannot read property 'map' of undefined, bad request while create APM dashboards in Kibana

[gasparuben]

Kibana version: 6.2.4

Elasticsearch version: 6.2.4

APM Server version: 6.8

APM Agent language and version: Still not there

Browser version: Chrome 80.0.3987.106

Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum. Installed apm-server on a centos7 VM.

Fresh install or upgraded from other version?
Fresh install
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
I did change some configuration at apm-server.yml

apm-server]# grep collaborati apm-server.yml
setup.template.name: "collaborativeapps6-apm-%{[beat.version]}"
setup.template.pattern: "collaborativeapps6-apm-%{[beat.version]}-*"
setup.dashboards.kibana_index: .kibana_collaborativeapps6
  host: "es-collaborativeapps6.domain.com:443/kibana_basic"
  username: "collaborativeapps"
  hosts: ["es-collaborativeapps6.domain.com:9203"]
  username: "collaborativeapps"
  index: "collaborativeapps6-apm-%{[beat.version]}-%{+yyyy.MM.dd}"
    - index: "collaborativeapps6-apm-%{[beat.version]}-sourcemap"
    - index: "collaborativeapps6-apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
    - index: "collaborativeapps6-apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
    - index: "collaborativeapps6-apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
    - index: "collaborativeapps6-apm-%{[beat.version]}-metric-%{+yyyy.MM.dd}"
    - index: "collaborativeapps6-apm-%{[beat.version]}-onboarding-%{+yyyy.MM.dd}"

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

While running apm-server setup I get an error. This is consistent also if trying to create the kibana dashboards using curl

# /usr/share/apm-server/bin/apm-server -c /etc/apm-server/apm-server.yml setup --dashboards -v
Loading dashboards (Kibana must be running and reachable)
Exiting: Failed to import index-pattern: Failed to load directory /usr/share/apm-server/kibana/6/index-pattern:
  error loading /usr/share/apm-server/kibana/6/index-pattern/apmserver.json: returned 400 to import file: <nil>. Response: {"statusCode":400,"error":"Bad Request","message":"Cannot read property 'map' of undefined"}

Using curl, get a bit different error:

# curl -H "Content-type: application/json"  -X POST  https://es-collaborativeapps.domain.com/kibana_basic/api/kibana/dashboards/import?force=true -u collaborativeapps:....  --data-binary "@/usr/share/apm-server/kibana/6/dashboard/apm-dashboards.json" 
{"statusCode":400,"error":"Bad Request","message":"Request must contain a kbn-xsrf header."}

APM 6.8 should be ok considering compatibility matrix for ES 6.2.4: https://www.elastic.co/es/support/matrix#matrix_compatibility

Please let me know about any hint or any further information you may require.

Thank you

[bmorelli25 (Brandon Morelli)]

Hi Gasparuben,

It looks like your apm-server.yml file isn't quite set up correctly. Using ES/KIB v6.2.4, and APM Server v6.8.6, I was able to get the following configuration to work:

setup.template.name: "collaborativeapps6-apm"
setup.template.pattern: "collaborativeapps6-apm-*"
setup.template.overwrite: true
setup.dashboards.index: collaborativeapps6-*

output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "collaborativeapps6-apm-%{[beat.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"

    - index: "collaborativeapps6-apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

    - index: "collaborativeapps6-apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"

    - index: "collaborativeapps6-apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"

    - index: "collaborativeapps6-apm-%{[beat.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"

    - index: "collaborativeapps6-apm-%{[beat.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"

You can then run ./apm-server setup and start apm-server. The APM dashboards will display data once you have it: