Not seeing data in APM Kibana App

valleram · May 17, 2021, 11:04pm

Kibana version: 7.11.1

Elasticsearch version: 7.11.1

APM Server version: 7.10.0

APM Agent language and version: Python 6.1.3

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: helm

Fresh install or upgraded from other version? Fresh intall

Is there anything special in your setup? Im using a custom index and index templated as well

Hi Team,

I'm unable to see APM Data in Kibana.
I've double check that APM server is up and running.

Also checked APM server log files and did not find any visible error, even I can see request from my service being attended by APM server.

2021-05-17T22:20:42.203Z	INFO	[kibana]	kibana/connecting_client.go:81	Successfully obtained connection to Kibana.
2021-05-17T22:20:43.188Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(https://elksaas-st-master:9200))
2021-05-17T22:20:43.188Z	WARN	[tls]	tlscommon/tls_config.go:93	SSL/TLS verifications disabled.
2021-05-17T22:20:43.188Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-05-17T22:20:43.188Z	INFO	[publisher]	pipeline/retry.go:223	  done
2021-05-17T22:20:43.199Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.11.1
2021-05-17T22:20:43.215Z	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Basic
2021-05-17T22:20:43.215Z	INFO	[index-management]	idxmgmt/manager.go:84	Overwrite ILM setup is disabled.
2021-05-17T22:20:43.216Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.11.1
2021-05-17T22:20:43.220Z	INFO	template/load.go:97	Template eso_ultron-apm-template already exists and will not be overwritten.
2021-05-17T22:20:43.220Z	INFO	[index-management]	idxmgmt/manager.go:211	Finished loading index template.
2021-05-17T22:20:43.231Z	INFO	template/load.go:97	Template apm-7.10.0-span already exists and will not be overwritten.
2021-05-17T22:20:43.231Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.0-span.
2021-05-17T22:20:43.235Z	INFO	template/load.go:97	Template apm-7.10.0-transaction already exists and will not be overwritten.
2021-05-17T22:20:43.235Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.0-transaction.
2021-05-17T22:20:43.240Z	INFO	template/load.go:97	Template apm-7.10.0-error already exists and will not be overwritten.
2021-05-17T22:20:43.240Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.0-error.
2021-05-17T22:20:43.244Z	INFO	template/load.go:97	Template apm-7.10.0-metric already exists and will not be overwritten.
2021-05-17T22:20:43.244Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.0-metric.
2021-05-17T22:20:43.253Z	INFO	template/load.go:97	Template apm-7.10.0-profile already exists and will not be overwritten.
2021-05-17T22:20:43.253Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.0-profile.
2021-05-17T22:20:43.253Z	INFO	[index-management]	idxmgmt/manager.go:126	Finished index management setup.
2021-05-17T22:20:43.257Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm
2021-05-17T22:20:43.259Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_agent
2021-05-17T22:20:43.260Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_geo
2021-05-17T22:20:43.262Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_ingest_timestamp
2021-05-17T22:20:43.263Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_remove_span_metadata
2021-05-17T22:20:43.263Z	INFO	[pipelines]	pipeline/register.go:56	Registered Ingest Pipelines successfully.
2021-05-17T22:20:43.263Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(https://elksaas-st-master:9200)) established
2021-05-17T22:25:05.442Z	INFO	[request]	middleware/log_middleware.go:101	request accepted	{"request_id": "cbfa3241-7257-49a4-be4c-905e3db2fe1f", "method": "POST", "URL": "/intake/v2/events", "content_length": 2758, "remote_address": "ip", "user-agent": "elasticapm-python/6.1.3", "event.duration": 2665484, "response_code": 202}

On the Elasticsearch output. I'm using an alias called "eso_ultron-apm" which is linked to indices with following index pattern " eso_ultron-apm-*".
Please find my config file below:

apm-server.yml: |
apm-server:
  host: "0.0.0.0:8200"
  rum:
    enabled: true
  ilm.enabled: false

  kibana:
    enabled: true
    host: "elksaas-st-kb-kibana:5601"
    protocol: "http"
    username:'${ELASTIC_USERNAME}'
    password: '${ELASTIC_PASSWORD}'
    ssl.verification_mode: none

setup.template.enabled: true
setup.template.name: "eso_ultron-apm-template"
setup.template.pattern: "*-apm*"
setup.template.overwrite: false

output.elasticsearch:
  username: '${ELASTIC_USERNAME}'
  password: '${ELASTIC_PASSWORD}'
  protocol: https
  hosts: ["https://elksaas-st-master:9200"]
  ssl.verification_mode: none
  ssl.certificate_authorities: ["/usr/share/apm-server/config/certs/elastic-certificate.pem"]
  index: "apm-%{[observer.version]}-eso-default-%{+yyyy.MM.dd}"

  indices:
    - index: "eso_ultron-apm"
      when.contains:
        service.name: "ultron"

I've confirmed that " eso_ultron-apm- *" indices are receiving data from my APM service.

On Kibana side, I'm receiving this error logs

{"type":"log","@timestamp":"2021-05-17T22:22:28+00:00","tags":["error","http"],"pid":7,"message":"StatusCodeError: [index_not_found_exception] no such index [.apm-agent-configuration], with { resource.type=\"index_or_alias\" & resource.id=\".apm-agent-configuration\" & index_uuid=\"_na_\" & index=\".apm-agent-configuration\" }\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4949:19)\n    at IncomingMessage.emit (events.js:327:22)\n    at endReadableNT (internal/streams/readable.js:1327:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21) {\n  status: 404,\n  displayName: 'NotFound',\n  path: '/.apm-agent-configuration/_search',\n  query: {},\n  body: {\n    error: {\n      root_cause: [Array],\n      type: 'index_not_found_exception',\n      reason: 'no such index [.apm-agent-configuration]',\n      'resource.type': 'index_or_alias',\n      'resource.id': '.apm-agent-configuration',\n      index_uuid: '_na_',\n      index: '.apm-agent-configuration'\n    },\n    status: 404\n  },\n  statusCode: 404,\n  response: '{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [.apm-agent-configuration]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\".apm-agent-configuration\",\"index_uuid\":\"_na_\",\"index\":\".apm-agent-configuration\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [.apm-agent-configuration]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\".apm-agent-configuration\",\"index_uuid\":\"_na_\",\"index\":\".apm-agent-configuration\"},\"status\":404}',\n  toString: [Function (anonymous)],\n  toJSON: [Function (anonymous)]\n}"}
{"type":"error","@timestamp":"2021-05-17T22:22:28+00:00","tags":[],"pid":7,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:58:19)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:45:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:312:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:221:9)"},"url":"http://elksaas-st-kb-kibana:5601/api/apm/settings/agent-configuration/search","message":"Internal Server Error"}

Can someone please help me?

Regards.
Jesus

simitt · May 18, 2021, 6:15am

Hi @valleram ,
please try changing the index settings in the APM Kibana UI so that they fit your index names.

The error you shared indicates that no agent config management is configured yet, but that should not cause any troubles with generally viewing APM data.

valleram · May 18, 2021, 12:14pm

Hi @simitt ,

Thank you.
I've already change the index settings directly in the APM Kibana UI, but I'm still unable to see any data there. I'm still receiving documents on my APM index, but in the APM UI the information is not available.

Regards

dgieselaar · May 19, 2021, 11:23am

Hey @valleram! Can you:

Provide the complete source of a document (processor.event:transaction) as it is stored in Elasticsearch?
Post a screenshot of the APM UI where you are expecting to see data?

system · June 9, 2021, 7:24am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elastic APM agent iOS APM server , ios	5	360	October 27, 2023
Not able to see APM data in Kibana APM go	9	2579	July 31, 2019
No visible data on APM in Kibana APM ui	1	285	June 29, 2023
Kibana doesn't show APM data APM java	3	673	December 25, 2019
No APM server detecting in kibana even apm server running locally APM python , server , ui	1	745	November 23, 2022

Not seeing data in APM Kibana App

Related topics