# Unable to see apm data in kibana

**URL:** <https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548>\
**Category:** APM\
**Tags:** ui\
**Created:** [November 24, 2020, 5:18pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548 "2020-11-24T17:18:23Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 24, 2020, 5:18pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/1 "2020-11-24T17:18:23Z")

</div>

Hi Team,

I am unable to see apm data in kibana dashboard. Can someone please help. Getting the error as attached. this is single standalone server (dev env) where components of elk is installed.

also i am getting error 500 when clicking APM in kibana (see attachment)

```auto
  systemctl status apm-server.service -l
● apm-server.service - Elastic APM Server
   Loaded: loaded (/usr/lib/systemd/system/apm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-11-24 21:50:26 IST; 50min ago
     Docs: https://www.elastic.co/solutions/apm
 Main PID: 3362 (apm-server)
   CGroup: /system.slice/apm-server.service
           └─3362 /usr/share/apm-server/bin/apm-server -e -c /etc/apm-server/apm-server.yml -path.home /usr/share/apm-server -path.config /etc/apm-server -path.data /var/lib/apm-server -path.logs /var/log/apm-server

Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "library_frame": true,
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "exclude_from_grouping": false,
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "line": {
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "number": 1041
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: }
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: },
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: {
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "module": "org.springframework.web.servlet",
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "function": "doDispatch",
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "library_frame": true,

```

cat /etc/apm-server/apm-server.yml

```auto
# Ansible managed

logging.level: debug
apm-server:
  host: "<IP of localhost>:7200"

path.data: /var/lib/apm-server
path.logs: /var/log/apm-server

output.kafka:
  hosts: ["<IP of localhost>:3997"]
  topic: "apm"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  max_message_bytes: 1000000

```

both ports are showing in netstat. both service `kafka` and `apm-server` are running.

inside `/var/log/apm-server`, can see old timestamp log files. so it seems logs are getting generated here.

/var/log/messages

```auto
Nov 24 02:10:00 <HOSTNAME> apm-server: 2020-11-24T02:10:00.617+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "10b7c6d4-300e-4cf8-b8cf-f2702ecf8a73", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.172", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Nov 24 02:10:00 <HOSTNAME> apm-server: 2020-11-24T02:10:00.618+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "e9a9930b-caa3-48bd-b3f6-ce433e4ab329", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.173", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}
Nov 24 02:10:04 <HOSTNAME> apm-server: 2020-11-24T02:10:04.686+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "6cb983fe-ae11-400b-9f07-0626be106fdf", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.171", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}
Nov 24 02:10:04 <HOSTNAME> apm-server: 2020-11-24T02:10:04.799+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "da02a211-53d0-4050-b3ef-b72340e67efa", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.172", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Nov 24 02:10:10 <HOSTNAME> apm-server: 2020-11-24T02:10:10.618+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "9ea602ab-28d0-43ba-ae33-ddb21cf16e9d", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.172", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Nov 24 02:10:10 <HOSTNAME> apm-server: 2020-11-24T02:10:10.633+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "ab1adb2c-8676-462a-8b52-5cbd6373083d", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.173", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}

```

able to self telnet on 3997.

Thanks,

---

<div class="post-metadata">

**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\
**Post date:** [November 24, 2020, 5:39pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/2 "2020-11-24T17:39:02Z")

</div>

I don't see an attachment for the kibana error but guess you're running into a common issue when APM Server isn't communicating directly with Elasticsearch - the index templates are missing and won't create the APM indices as kibana expects them. Please refer to [this guide](https://www.elastic.co/guide/en/kibana/7.9/troubleshooting.html#no-apm-data-found) to help resolve the issue. Briefly this entails:

1. Stop ingestion (the kafka consumer in this case)
2. Delete existing indices
3. Load index mapping templates
4. Start ingestion

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 24, 2020, 9:37pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/3 "2020-11-24T21:37:51Z")

</div>

Hi Gil,

Thanks for your prompt response.

Here is the attachment.

 ![21B7D778-A3FA-47A4-B8FE-155E40157E71_1_201_a](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9a15851758916e80ad960393ec9b2ff40951361.jpeg)

As per your reply,

1. Stop ingestion (the kafka consumer in this case) -  
you mean to say, # systemctl stop kafka.service?

2. Delete existing indices -  
I was able to list the indices in Dev tools in Kibana using `GET /_cat/indices?v`  
Is it the write command to delete APM\* indices using `DELETE /apm*`

3. Load index mapping templates -  
i) Where can i run below command to check existence of an APM index template.  
`GET /_template/apm-{version}`

can I run above command in Dev tools ?

and if it exists, can I proceed for below steps?

ii) How can I verify that we are not directly outputting data from APM server to Elasticsearch.  
if this is true, do I need to `Load the template manually?`  
If yes, is below command correct to load the index?

```auto
apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

```

in document, for above command a note is given as `The examples here assume that Logstash output is enabled.` How can I verify this?

Do we need to run `curl -XDELETE 'http://localhost:9200/apm-server-*'` to force kibana to look at newest documents?

1. Start ingestion -  
#systemctl start Kafka-service?

Thanks once again.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 25, 2020, 8:16am UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/4 "2020-11-25T08:16:28Z")

</div>

Hi,

Can someone please reply.

---

<div class="post-metadata">

**Author:** ![sqren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sqren/32/26110_2.png) [@sqren](https://discuss.elastic.co/u/sqren)\
**Post date:** [November 25, 2020, 2:38pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/5 "2020-11-25T14:38:13Z")

</div>

Hey,

I also don't see the error from Kibana attached. It looks like the logs are from APM Server.  
If you can find the Kibana logs, that would be very helpful.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 25, 2020, 3:50pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/6 "2020-11-25T15:50:51Z")

</div>

the attachment is from APM tab from kibana dashboard. You can see it says `Sorry, an error occured`

I would like to know why there is no data shown in APM tab.

(cut the below output due to limit)

journalctl -u kibana

```auto
Nov 25 20:52:06 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:22:06Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to
Nov 25 20:52:07 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:22:07Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"

```

journalctl -fu kibana

```auto
-- Logs begin at Wed 2020-11-25 11:59:40 IST. --
bulks\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"bulk [default_local] reports failures when exporting documents\\\",\\\"exceptions\\\":[{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:52:57 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:22:57Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
\"used_in_bytes\\\":16347914240},\\\"uptime_in_millis\\\":738827000,\\\"platform\\\":\\\"linux\\\",\\\"platformRelease\\\":\\\"linux-3.10.0-1062.el7.x86_64\\\",\\\"distro\\\":\\\"Red Hat Linux\\\",\\\"distroRelease\\\":\\\"Red Hat Linux-7.6\\\"},\\\"process\\\":{\\\"event_loop_delay\\\":1.2810907363891602,\\\"memory\\\":{\\\"heap\\\":{\\\"total_in_bytes\\\":260448256,\\\"used_in_bytes\\\":215544528,\\\"size_limit\\\":1526909922},\\\"resident_set_size_in_bytes\\\":297082880},\\\"uptime_in_millis\\\":181768463},\\\"requests\\\":{\\\"disconnects\\\":0,\\
\",\\\"reason\\\":\\\"bulk [default_local] reports failures when exporting documents\\\",\\\"exceptions\\\":[{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:53:47 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:23:47Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
Nov 25 20:53:48 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:23:48Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":{\\\"task\\\":{\\\"taskType\\\":\\\"maps_telemetry\\\",\\\"state\\\":\\\"{}\\\",\\\"attempts\\\":1,\\\"scheduledAt\\\":\\\"2020-11-19T12:45:52.098Z\\\",\\\"startedAt\\\":\\\"2020-11-25T15:23:48.308Z\\\",\\\"retryAt\\\":\\\"2020-11-25T15:29:48.308Z\\\",\\\"runAt\\\":\\\"2020-11-24T18:30:00.000Z\\\",\\\"status\\\":\\\"running\\\"},\\\"updated_at\\\":\\\"2020-11-25T15:23:48.308Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}

```

ps -ef |grep kibana

```auto
kibana 9604 1 0 Nov23 ? 00:30:18 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

```

systemctl status kibana.service -l

```auto
  kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-11-23 18:24:19 IST; 2 days ago
 Main PID: 9604 (node)
   CGroup: /system.slice/kibana.service
           └─9604 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Nov 25 20:56:37 ITFOOPENBANKINGAPP02UAT kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[export_exception] failed to flush export bulks :: {\"path\":\"/_monitoring/bulk\",\"query\":{\"system_id\":\"kibana\",\"system_api_version\":\"6\",\"interval
\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:56:37 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
Nov 25 20:56:39 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:39Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":{\\\"task\\\":{\\\"taskType\\\":\\\"maps_telemetry\\\",\\\"state\\\":\\\"25T15:26:39.643Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}
Nov 25 20:56:47 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:47Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[export_exception] failed to flush export bulks :: {\"path\":\"/_monitoring/bulk\",\"query\":{\"system_id\":\"kibana\",\"system_api_version\":\"6\",\"interval\":\"10000ms\"},\"body\":\"{\\\"index\\\":{\\\"_type\\\":\\"used_in_bytes\\\":217684152,\\\"size_limit\\\":1526909922},\\\"resident_set_size_in_bytes\\\":296951808},\\\"uptime_in_millis\\\":181948501},\\\"requests\\\":{\\\"disconnects\\\":0,\\\"total\\\":0},\\\"response_times\\\":{\\\"average\\\":0,\\\"max\\\":0},\\\"timestamp\\\":\\\"2020-11-25T15:26:47.602Z\\\"}\\n\",\"statusCode\":500,\"response\":\"{\\\"took\\\":9,\\\"ignored\\\":false,\\\"errors\\\":true,\\\"error\\\":{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"failed to flush export bulks\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"bulk [default_local] reports failures when exporting documents\\\",\\\"exceptions\\\":[{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:56:47 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:47Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
Nov 25 20:56:51 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:51Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":{\\\"task\\\":{\\\"taskType\\\":\\\"maps_telemetry\\\",\\\"state\\\":\\\"{\\\\\\\"runs\\\\\\\":1,\\\\\\\"stats\\\\\\\":{\\\\\\\"mapsTotalCount\\\\\\\":0,\\\\\\\"timeCaptured\\\\\\\":\\\\\\\"2020-11-23T18:30:01.884Z\\\\\\\",\\\\\\\"attributesPerMap\\\\\\\":{\\\\\"updated_at\\\":\\\"2020-11-25T15:26:51.662Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}
Nov 25 20:56:54 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:54Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":23T18:30:01.884Z\\\\\\\",\\\\\\\"attributesPerMap\\\\\\\":{\\\\\"updated_at\\\":\\\"2020-11-25T15:26:54.668Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}

```

cat /etc/kibana/kibana.yml

```auto
# Ansible managed

server.port: 8601
server.host: "0.0.0.0"
server.name: kibana_1

elasticsearch.requestTimeout: 30000

elasticsearch.hosts: ['http://<IP of localhost>:5200']
elasticsearch.username: "elastic"
elasticsearch.password: "Password"

```

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 26, 2020, 6:48am UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/8 "2020-11-26T06:48:03Z")

</div>

systemctl status kibana.service -l

kibana.service - Kibana  
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)  
Active: active (running) since Mon 2020-11-23 18:24:19 IST; 2 days ago  
Main PID: 9604 (node)  
CGroup: /system.slice/kibana.service  
└─9604 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Nov 25 20:56:37 kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[export\_exception] failed to flush export bulks :: {"path":"/\_monitoring/bulk","query":{"system\_id":"kibana","system\_api\_version":"6","interval":"10000ms"},"body":"{\"index\":{\"\_type\":\"kibana\_stats\"}}\n{\"kibana\":{\"uuid\":\"1329e789-0ad5-49e2-8b65-bb4f31a71c73\",\"name\":\"kibana\_1\",\"index\":\".kibana\",\"host\":\"0.0.0.0\",\"transport\_address\":\"0.0.0.0:8601\",\"version\":\"7.4.0\",\"snapshot\":false,\"status\":\"green\"},\"usage\":{\"kql\":{\"optInCount\":0,\"optOutCount\":0,\"defaultQueryLanguage\":\"default-kuery\"},\"localization\":{\"locale\":\"en\",\"integrities\":{},\"labelsCount\":0},\"ui\_metric\":{\"infra\_metrics\":[{\"key\":\"pageview\_\_inventory\_\_delayed\_15000ms\",\"value\":1},{\"key\":\"pageview\_\_inventory\",\"value\":2}],\"uptime\":[{\"key\":\"pageview\_\_overview\",\"value\":1},{\"key\":\"pageview\_\_overview\_\_delayed\_15000ms\",\"value\":1}]},\"index\":\".kibana\",\"dashboard\":{\"total\":95},\"visualization\":{\"total\":581},\"search\":{\"total\":57},\"index\_pattern\":{\"total\":5},\"graph\_workspace\":{\"total\":0},\"timelion\_sheet\":{\"total\":0},\"spaces\":{\"available\":true,\"enabled\":true,\"count\":1,\"usesFeatureControls\":false,\"disabledFeatures\":{\"discover\":0,\"visualize\":0,\"dashboard\":0,\"dev\_tools\":0,\"advancedSettings\":0,\"indexPatterns\":0,\"savedObjectsManagement\":0,\"graph\":0,\"monitoring\":0,\"ml\":0,\"apm\":0,\"maps\":0,\"canvas\":0,\"infrastructure\":0,\"logs\":0,\"siem\":0,\"uptime\":0}},\"ml\":{\"file\_data\_visualizer\":{\"index\_creation\_count\":0}},\"apm\":{\"has\_any\_services\":false,\"services\_per\_agent\":{}},\"maps\":{\"mapsTotalCount\":0,\"timeCaptured\":\"2020-11-23T18:30:01.884Z\",\"attributesPerMap\":{\"dataSourcesCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layersCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layerTypesCount\":{},\"emsVectorLayersCount\":{}}},\"canvas\":{},\"cloud\":{\"isCloudEnabled\":false},\"infraops\":{\"last\_24\_hours\":{\"hits\":{\"infraops\_hosts\":0,\"infraops\_docker\":0,\"infraops\_kubernetes\":0,\"logs\":9}}},\"rollups\":{\"index\_patterns\":{\"total\":0},\"saved\_searches\":{\"total\":0},\"visualizations\":{\"total\":0,\"saved\_searches\":{\"total\":0}}},\"upgrade-assistant-telemetry\":{\"ui\_open\":{\"overview\":0,\"cluster\":0,\"indices\":0},\"ui\_reindex\":{\"close\":0,\"open\":0,\"start\":0,\"stop\":0},\"features\":{\"deprecation\_logging\":{\"enabled\":true}}},\"uptime\":{\"last\_24\_hours\":{\"hits\":{\"overview\_page\":1,\"monitor\_page\":0}}},\"visualization\_types\":{\"metrics\":{\"total\":237,\"spaces\_min\":237,\"spaces\_max\":237,\"spaces\_avg\":237},\"metric\":{\"total\":39,\"spaces\_min\":39,\"spaces\_max\":39,\"spaces\_avg\":39},\"pie\":{\"total\":87,\"spaces\_min\":87,\"spaces\_max\":87,\"spaces\_avg\":87},\"histogram\":{\"total\":38,\"spaces\_min\":38,\"spaces\_max\":38,\"spaces\_avg\":38},\"table\":{\"total\":42,\"spaces\_min\":42,\"spaces\_max\":42,\"spaces\_avg\":42},\"tagcloud\":{\"total\":9,\"spaces\_min\":9,\"spaces\_max\":9,\"spaces\_avg\":9},\"input\_control\_vis\":{\"total\":14,\"spaces\_min\":14,\"spaces\_max\":14,\"spaces\_avg\":14},\"tile\_map\":{\"total\":17,\"spaces\_min\":17,\"spaces\_max\":17,\"spaces\_avg\":17},\"area\":{\"total\":13,\"spaces\_min\":13,\"spaces\_max\":13,\"spaces\_avg\":13},\"timelion\":{\"total\":39,\"spaces\_min\":39,\"spaces\_max\":39,\"spaces\_avg\":39},\"line\":{\"total\":33,\"spaces\_min\":33,\"spaces\_max\":33,\"spaces\_avg\":33},\"horizontal\_bar\":{\"total\":3,\"spaces\_min\":3,\"spaces\_max\":3,\"spaces\_avg\":3},\"markdown\":{\"total\":8,\"spaces\_min\":8,\"spaces\_max\":8,\"spaces\_avg\":8},\"gauge\":{\"total\":1,\"spaces\_min\":1,\"spaces\_max\":1,\"spaces\_avg\":1},\"heatmap\":{\"total\":1,\"spaces\_min\":1,\"spaces\_max\":1,\"spaces\_avg\":1}},\"csp\":{\"strict\":false,\"warnLegacyBrowsers\":true,\"rulesChangedFromDefault\":false},\"fileUploadTelemetry\":{\"filesUploadedTotalCount\":0},\"xpack\":{\"reporting\":{\"available\":true,\"browser\_type\":\"chromium\",\"enabled\":true,\"lastDay\":{\"\_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable\_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve\_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"last7Days\":{\"\_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable\_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve\_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"\_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable\_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve\_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}}}},\"concurrent\_connections\":0,\"os\":{\"load\":{\"1m\":0.21728515625,\"5m\":0.10888671875,\"15m\":0.12841796875},\"memory\":{\"total\_in\_bytes\":16825806848,\"free\_in\_bytes\":535048192,\"used\_in\_bytes\":16290758656},\"uptime\_in\_millis\":738997000,\"platform\":\"linux\",\"platformRelease\":\"linux-3.10.0-1062.el7.x86\_64\",\"distro\":\"Red Hat Linux\",\"distroRelease\":\"Red Hat Linux-7.6\"},\"process\":{\"event\_loop\_delay\":1.3422919511795044,\"memory\":{\"heap\":{\"total\_in\_bytes\":259399680,\"used\_in\_bytes\":215711952,\"size\_limit\":1526909922},\"resident\_set\_size\_in\_bytes\":296386560},\"uptime\_in\_millis\":181938497},\"requests\":{\"disconnects\":0,\"total\":0},\"response\_times\":{\"average\":0,\"max\":0},\"timestamp\":\"2020-11-25T15:26:37.597Z\"}\n","statusCode":500,"response":"{\"took\":8,\"ignored\":false,\"errors\":true,\"error\":{\"type\":\"export\_exception\",\"reason\":\"failed to flush export bulks\",\"caused\_by\":{\"type\":\"export\_exception\",\"reason\":\"bulk [default\_local] reports failures when exporting documents\",\"exceptions\":[{\"type\":\"export\_exception\",\"reason\":\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\",\"caused\_by\":{\"type\":\"cluster\_block\_exception\",\"reason\":\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}}]}}}"}\n at respond (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (\_stream\_readable.js:1103:12)\n at process.\_tickCallback (internal/process/next\_tick.js:63:19)"}  
Nov 25 20:56:37 kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}  
Nov 25 20:56:39 kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:39Z","tags":["error","task\_manager"],"pid":9604,"message":"Failed to poll for work: [cluster\_block\_exception] index [.kibana\_task\_manager\_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana\_task\_manager/\_update/task%3AMaps-maps\_telemetry","query":{"if\_seq\_no":23,"if\_primary\_term":16,"refresh":"wait\_for"},"body":"{\"doc\":{\"task\":{\"taskType\":\"maps\_telemetry\",\"state\":\"{\\\"runs\\\":1,\\\"stats\\\":{\\\"mapsTotalCount\\\":0,\\\"timeCaptured\\\":\\\"2020-11-23T18:30:01.884Z\\\",\\\"attributesPerMap\\\":{\\\"dataSourcesCount\\\":{\\\"min\\\":0,\\\"max\\\":0,\\\"avg\\\":0},\\\"layersCount\\\":{\\\"min\\\":0,\\\"max\\\":0,\\\"avg\\\":0},\\\"layerTypesCount\\\":{},\\\"emsVectorLayersCount\\\":{}}}}\",\"params\":\"{}\",\"attempts\":1,\"scheduledAt\":\"2020-11-19T12:45:52.098Z\",\"startedAt\":\"2020-11-25T15:26:39.643Z\",\"retryAt\":\"2020-11-25T15:32:39.643Z\",\"runAt\":\"2020-11-24T18:30:00.000Z\",\"status\":\"running\"},\"updated\_at\":\"2020-11-25T15:26:39.643Z\",\"references\":}}","statusCode":403,"response":"{\"error\":{\"root\_cause\":[{\"type\":\"cluster\_block\_exception\",\"reason\":\"index [.kibana\_task\_manager\_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster\_block\_exception\",\"reason\":\"index [.kibana\_task\_manager\_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"}

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 26, 2020, 6:58am UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/9 "2020-11-26T06:58:31Z")

</div>

journalctl -fu kibana

-- Logs begin at Thu 2020-11-26 02:31:40 IST. --  
Nov 26 12:22:33 kibana[9604]: {"type":"log","@timestamp":"2020-11-26T06:52:33Z","tags":["error","task\_manager"],"pid":9604,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}  
Nov 26 12:22:43 kibana[9604]: {"type":"log","@timestamp":"2020-11-26T06:52:43Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[cluster\_block\_exception] blocked by: [SERVICE\_UNAVAILABLE/2/no master]; :: {"path":"/\_monitoring/bulk","query":{"system\_id":"kibana","system\_api\_version":"6","interval":"10000ms"},"body":"{\"index\":{\"\_type\":\"kibana\_stats\"}}\n{\"kibana\":{\"uuid\":\"1329e789-0ad5-49e2-8b65-bb4f31a71c73\",\"name\":\"kibana\_1\",\"index\":\".kibana\",\"host\":\"0.0.0.0\",\"transport\_address\":\"0.0.0.0:8601\",\"version\":\"7.4.0\",\"snapshot\":false,\"status\":\"green\"},\"usage\":{\"kql\":{\"optInCount\":0,\"optOutCount\":0,\"defaultQueryLanguage\":\"default-kuery\"},\"localization\":{\"locale\":\"en\",\"integrities\":{},\"labelsCount\":0},\"ui\_metric\":{\"infra\_metrics\":[{\"key\":\"pageview\_\_inventory\_\_delayed\_15000ms\",\"value\":1},{\"key\":\"pageview\_\_inventory\",\"value\":2}],\"uptime\":[{\"key\":\"pageview\_\_overview\",\"value\":1},{\"key\":\"pageview\_\_overview\_\_delayed\_15000ms\",\"value\":1}]},\"index\":\".kibana\",\"dashboard\":{\"total\":95},\"visualization\":{\"total\":581},\"search\":{\"total\":57},\"index\_pattern\":{\"total\":5},\"graph\_workspace\":{\"total\":0},\"timelion\_sheet\":{\"total\":0},\"spaces\":{\"available\":true,\"enabled\":true,\"count\":1,\"usesFeatureControls\":false,\"disabledFeatures\":{\"discover\":0,\"visualize\":0,\"dashboard\":0,\"dev\_tools\":0,\"advancedSettings\":0,\"indexPatterns\":0,\"savedObjectsManagement\":0,\"graph\":0,\"monitoring\":0,\"ml\":0,\"apm\":0,\"maps\":0,\"canvas\":0,\"infrastructure\":0,\"logs\":0,\"siem\":0,\"uptime\":0}},\"ml\":{\"file\_data\_visualizer\":{\"index\_creation\_count\":0}},\"apm\":{\"has\_any\_services\":false,\"services\_per\_agent\":{}},\"maps\":{\"mapsTotalCount\":0,\"timeCaptured\":\"2020-11-23T18:30:01.884Z\",\"attributesPerMap\":{\"dataSourcesCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layersCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layerTypesCount\":{},\"emsVectorLayersCount\":{}}},\"canvas\":{},\"cloud\":{\"isCloudEnabled\":false},\"infraops\":{\"last\_24\_hours\":{\"hits\":{\"infraops\_hosts\":0,\"infraops\_docker\":0,\"infraops\_kubernetes\":0,\"logs\":9}}},\"rollups\":{\"index\_patterns\":{\"total\":0},\"saved\_searches\":{\"total\":0},\"visualizations\":{\"total\":0,\"saved\_searches\":{\"total\":0}}},\"upgrade-assistant-telemetry\":{\"ui\_open\":{\"overview\":0,\"cluster\":0,\"indices\":0},\"ui\_reindex\":{\"close\":0,\"open\":0,\"start\":0,\"stop\":0},\"features\":{\"deprecation\_logging\":{\"enabled\":false}}},\"uptime\":{\"last\_24\_hours\":{\"hits\":{\"overview\_page\":1,\"monitor\_page\":0}}},\"visualization\_types\":{\"metrics\":{\"total\":237,\"spaces\_min\":237,\"spaces\_max\":237,\"spaces\_avg\":237},\"metric\":{\"total\":39,\"spaces\_min\":39,\"spaces\_max\":39,\"spaces\_avg\":39},\"pie\":{\"total\":87,\"spaces\_min\":87,\"spaces\_max\":87,\"spaces\_avg\":87},\"histogram\":{\"total\":38,\"spaces\_min\":38,\"spaces\_max\":38,\"spaces\_avg\":38},\"table\":{\"total\":42,\"spaces\_min\":42,\"spaces\_max\":42,\"spaces\_avg\":42},\"tagcloud\":{\"total\":9,\"spaces\_min\":9,\"spaces\_max\":9,\"spaces\_avg\":9},\"input\_control\_vis\":{\"total\":14,\"spaces\_min\":14,\"spaces\_max\":14,\"spaces\_avg\":14},\"tile\_map\":{\"total\":17,\"spaces\_min\":17,\"spaces\_max\":17,\"spaces\_avg\":17},\"area\":{\"total\":13,\"spaces\_min\":13,\"spaces\_max\":13,\"spaces\_avg\":13},\"timelion\":{\"total\":39,\"spaces\_min\":39,\"spaces\_max\":39,\"spaces\_avg\":39},\"line\":{\"total\":33,\"spaces\_min\":33,\"spaces\_max\":33,\"spaces\_avg\":33},\"horizontal\_bar\":{\"total\":3,\"spaces\_min\":3,\"spaces\_max\":3,\"spaces\_avg\":3},\"markdown\":{\"total\":8,\"spaces\_min\":8,\"spaces\_max\":8,\"spaces\_avg\":8},\"gauge\":{\"total\":1,\"spaces\_min\":1,\"spaces\_max\":1,\"spaces\_avg\":1},\"heatmap\":{\"total\":1,\"spaces\_min\":1,\"spaces\_max\":1,\"spaces\_avg\":1}},\"csp\":{\"strict\":false,\"warnLegacyBrowsers\":true,\"rulesChangedFromDefault\":false},\"fileUploadTelemetry\":{\"filesUploadedTotalCount\":0},\"xpack\":{\"reporting\":{\"available\":true,\"browser\_type\":\"chromium\",\"enabled\":true,\"lastDay\":{\"\_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable\_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve\_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"last7Days\":{\"\_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable\_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve\_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"\_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable\_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve\_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}}}},\"concurrent\_connections\":0,\"os\":{\"load\":{\"1m\":0.01513671875,\"5m\":0.037109375,\"15m\":0.04541015625},\"memory\":{\"total\_in\_bytes\":16825806848,\"free\_in\_bytes\":280158208,\"used\_in\_bytes\":16545648640},\"uptime\_in\_millis\":794528000,\"platform\":\"linux\",\"platformRelease\":\"linux-3.10.0-1062.el7.x86\_64\",\"distro\":\"Red Hat Linux\",\"distroRelease\":\"Red Hat Linux-7.6\"},\"process\":{\"event\_loop\_delay\":0.752547025680542,\"memory\":{\"heap\":{\"total\_in\_bytes\":262545408,\"used\_in\_bytes\":229375040,\"size\_limit\":1526909922},\"resident\_set\_size\_in\_bytes\":297926656},\"uptime\_in\_millis\":237469036},\"requests\":{\"disconnects\":0,\"total\":0},\"response\_times\":{\"average\":0,\"max\":0},\"timestamp\":\"2020-11-26T06:52:08.135Z\"}\n","statusCode":503,"response":"{\"error\":{\"root\_cause\":[{\"type\":\"cluster\_block\_exception\",\"reason\":\"blocked by: [SERVICE\_UNAVAILABLE/2/no master];\"}],\"type\":\"cluster\_block\_exception\",\"reason\":\"blocked by: [SERVICE\_UNAVAILABLE/2/no master];\"},\"status\":503}"}\n at respond (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector. (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node\_modules/elasticsearch/node\_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (\_stream\_readable.js:1103:12)\n at process.\_tickCallback (internal/process/next\_tick.js:63:19)"}  
Nov 26 12:22:43 kibana[9604]: {"type":"log","@timestamp":"2020-11-26T06:52:43Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}

---

<div class="post-metadata">

**Author:** ![sqren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sqren/32/26110_2.png) [@sqren](https://discuss.elastic.co/u/sqren)\
**Post date:** [November 26, 2020, 1:07pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/10 "2020-11-26T13:07:59Z")

</div>

This error doesn't look APM related:

```auto
blocked by: [SERVICE_UNAVAILABLE/2/no master]

```

Please double check that you've configured elasticsearch correctly, and that the cluster is in a healthy state:

```auto
curl localhost:9200/_cat/health

```

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 26, 2020, 3:04pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/11 "2020-11-26T15:04:18Z")

</div>

Hi Sqren,

Thanks for reply. Using 5200 as Elasticsearch http port.

-rw-rw---- 1 root elasticsearch 913 Nov 19 18:12 elasticsearch.yml

curl localhost:5200/\_cat/health

`{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}`

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 26, 2020, 3:28pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/12 "2020-11-26T15:28:04Z")

</div>

Is below `cluster.initial_master_nodes`, `discovery.seed_hosts` correctly mentioned? or do we need to mentioned them inside [" "] and not [' '].

cat elasticsearch.yml

```auto
# Ansible managed

[node.name](http://node.name/): elasticsearch_1

path.data: /opt/elasticsearch

path.logs: /var/log/elasticsearch

http.port: 5200

transport.port: 5300

path.repo: /opt/elastic_snapshot

[cluster.name](http://cluster.name/): xxx-dev

cluster.initial_master_nodes: ['localhost ip']

discovery.seed_hosts: ['localhost ip']

network.host: 0.0.0.0

http.host: 0.0.0.0

node.master: True

node.data: True

node.ingest: True

discovery.zen.minimum_master_nodes: 2

xpack.security.transport.ssl.enabled: true

xpack.security.transport.ssl.verification_mode: certificate

xpack.security.transport.ssl.keystore.path: elasticsearch_1.p12

xpack.security.transport.ssl.truststore.path: elasticsearch_1.p12

# xpack.security.http.ssl.verification_mode: certificate

# xpack.security.http.ssl.keystore.path: elasticsearch_1.p12

# xpack.security.http.ssl.truststore.path: elasticsearch_1.p12

xpack.monitoring.collection.enabled: true

xpack.security.enabled: true

```

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 27, 2020, 12:58pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/14 "2020-11-27T12:58:32Z")

</div>

app related errors in /var/log/message

`Nov 27 06:32:35 <HOSTNAME> <custom-app-api>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:41 <HOSTNAME> ape-server: 2020-11-27T06:32:41.619+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "8a0b9628-1ba8-4981-ad58-389bb2d9dec1", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<localhost IP>", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 503, "error": "queue is full"} Nov 27 06:32:41 <HOSTNAME> <customer-app-admin-api>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:41 <HOSTNAME> apm-server: 2020-11-27T06:32:41.620+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "a6f911cb-e424-48bb-bc77-e4ed14fab7ce", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<localhost IP>", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 503, "error": "queue is full"} Nov 27 06:32:41 <HOSTNAME> apm-server: 2020-11-27T06:32:41.620+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "74f037f1-40ff-4c80-837f-78510d046434", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<localhost IP>", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 503, "error": "queue is full"} Nov 27 06:32:41 <HOSTNAME> <custom-admin-spa>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:41 <HOSTNAME> <custom-spa>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:50 <HOSTNAME> apm-server: 2020-11-27T06:32:50.310+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "61cbe81d-2db7-4787-9f64-281e5983dc11", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<pfpd-server-ip>", "user-agent": "elasticapm-java/1.14.0", "response_code": 503, "error": "queue is full"}`

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 30, 2020, 6:13am UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/16 "2020-11-30T06:13:23Z")

</div>

Hi All,

entered the correct password.

```auto
curl localhost:5200/_cat/health -u elastic
Enter host password for user 'elastic':
1606716560 06:09:20 xxx-dev yellow 1 1 40 40 0 0 15 0 - 72.7%

```

@gil @sqren , Can you please update.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 30, 2020, 7:43am UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/17 "2020-11-30T07:43:36Z")

</div>

In kibana, Inside HOME / Add data/ APM

for APM Server status - check APM Server status  
it shows, `You have correctly setup APM Server`

then in APM Agents  
Java  
for Agent status - check agent status  
it shows, `Data Successfully received from one or more agents`

For Load Kibana objects  
It gives below error,  
`Unable to add 1 of 1 kibana objects, Error: index [.kibana_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];`

in Index Patterns - I can see `apm-*`.

in Discover tab, I can see `apm-*` in dropdown and if choose I can see data as below.

Nov 24, 2020 @ 06:55:52.734 `parent.id`:500e0035de9b1d6b `observer.hostname`: `observer.id`:20c18025-5e1e-454e-842a-012955875a57 `observer.ephemeral_id`:6c8b2636-6cd3-455a-9977-7535f1f8c4bc `observer.type`:apm-server `observer.version`:7.4.0 `observer.version_major`:7 `trace.id`:92e98e438b09ae57307a534335f0c876 `agent.name`:java `agent.ephemeral_id`:a540c853-3c1f-42bf-8df0-dd031797713b `agent.version`:1.14.0 `@timestamp`:Nov 24, 2020 @ 06:55:52.734 `ecs.version`:1.1.0 `service.name`:pingDirectory `@version:1` `processor.name:`transaction

So why only in APM tab (left side) it shows above error (as in above attachment)

this show ok

```auto
# apm-server test config
Config OK

```

```auto
# apm-server test output
Kafka: <localhost ip>:3997...
  parse host... OK
  dns lookup... OK
  addresses: <localhost ip>
  dial up... ERROR dial tcp <localhost ip>:3997: connect: connection refused

```

```auto
telnet <localhost ip> 3997
Trying <localhost ip>...
telnet: connect to address <localhost ip>: Connection refused

```

```auto
netstat -antp |grep 3997

```

no other service is using 3997.

not sure why 3997 is not listing, apm service is up and running.

```auto
systemctl status apm-server.service -l
 apm-server.service - Elastic APM Server
   Loaded: loaded (/usr/lib/systemd/system/apm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-11-24 21:50:26 IST; 5 days ago
     Docs: https://www.elastic.co/solutions/apm
 Main PID: 3362 (apm-server)
   CGroup: /system.slice/apm-server.service
           └─3362 /usr/share/apm-server/bin/apm-server -e -c /etc/apm-server/apm-server.yml -path.home /usr/share/apm-server -path.config /etc/apm-server -path.data /var/lib/apm-server -path.logs /var/log/apm-server

Nov 30 13:06:56 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:56.783+0530 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
Nov 30 13:06:56 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:56.783+0530 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
Nov 30 13:06:56 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:56.783+0530 INFO kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530 INFO kafka/log.go:53 client/metadata fetching metadata for [apm] from broker <localhost ip>:3997
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530 INFO kafka/log.go:53 Failed to connect to broker <localhost ip>:3997: dial tcp <localhost ip>:3997: connect: connection refused
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp <localhost ip>:3997: connect: connection refused
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.634+0530 INFO [publisher] pipeline/retry.go:189 retryer: send unwait-signal to consumer
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.635+0530 INFO [publisher] pipeline/retry.go:191 done

```

---

<div class="post-metadata">

**Author:** ![sqren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sqren/32/26110_2.png) [@sqren](https://discuss.elastic.co/u/sqren)\
**Post date:** [November 30, 2020, 12:57pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/18 "2020-11-30T12:57:18Z")

</div>

Hi @prat,

From you initial screenshot I see that the request to `/api/apm/services` was failing. Were you able to find the logs for that error?

If you open your browser's Dev tools you can inspect the API requests made. Please find the one for `/api/apm/services` and replay it with `&_debug=true` appended and see the logs for that.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [November 30, 2020, 3:16pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/19 "2020-11-30T15:16:40Z")

</div>

Hi @sqren,

Thanks for reply. I can see below error in `/var/log/apm-server/apm-server.7`

`2020-11-30T13:04:58.086+0530 ERROR instance/beat.go:878 Exiting: Index management requested but the Elasticsearch output is not configured/enabled`.

Can you please tell what command need to run to inspect. Sorry I am not from developing background so not sure about this API commands.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [December 1, 2020, 3:05pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/20 "2020-12-01T15:05:13Z")

</div>

Hi @sqren,

Can you please reply.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [December 4, 2020, 6:48pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/21 "2020-12-04T18:48:15Z")

</div>

Hi @gil,

Can you please reply on my update.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2020, 2:48pm UTC](https://discuss.elastic.co/t/unable-to-see-apm-data-in-kibana/256548/22 "2020-12-25T14:48:39Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.