Unable to see apm data in kibana

prat · November 24, 2020, 5:18pm

Hi Team,

I am unable to see apm data in kibana dashboard. Can someone please help. Getting the error as attached. this is single standalone server (dev env) where components of elk is installed.

also i am getting error 500 when clicking APM in kibana (see attachment)

  systemctl status    apm-server.service -l
● apm-server.service - Elastic APM Server
   Loaded: loaded (/usr/lib/systemd/system/apm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-11-24 21:50:26 IST; 50min ago
     Docs: https://www.elastic.co/solutions/apm
 Main PID: 3362 (apm-server)
   CGroup: /system.slice/apm-server.service
           └─3362 /usr/share/apm-server/bin/apm-server -e -c /etc/apm-server/apm-server.yml -path.home /usr/share/apm-server -path.config /etc/apm-server -path.data /var/lib/apm-server -path.logs /var/log/apm-server

Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "library_frame": true,
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "exclude_from_grouping": false,
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "line": {
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "number": 1041
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: }
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: },
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: {
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "module": "org.springframework.web.servlet",
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "function": "doDispatch",
Nov 24 22:40:40 <HOSTNAME> apm-server[3362]: "library_frame": true,

cat /etc/apm-server/apm-server.yml

# Ansible managed

logging.level: debug
apm-server:
  host: "<IP of localhost>:7200"

path.data: /var/lib/apm-server
path.logs: /var/log/apm-server


output.kafka:
  hosts: ["<IP of localhost>:3997"]
  topic: "apm"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  max_message_bytes: 1000000

both ports are showing in netstat. both service kafka and apm-server are running.

inside /var/log/apm-server, can see old timestamp log files. so it seems logs are getting generated here.

/var/log/messages

Nov 24 02:10:00 <HOSTNAME> apm-server: 2020-11-24T02:10:00.617+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "10b7c6d4-300e-4cf8-b8cf-f2702ecf8a73", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.172", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Nov 24 02:10:00 <HOSTNAME> apm-server: 2020-11-24T02:10:00.618+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "e9a9930b-caa3-48bd-b3f6-ce433e4ab329", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.173", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}
Nov 24 02:10:04 <HOSTNAME> apm-server: 2020-11-24T02:10:04.686+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "6cb983fe-ae11-400b-9f07-0626be106fdf", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.171", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}
Nov 24 02:10:04 <HOSTNAME> apm-server: 2020-11-24T02:10:04.799+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "da02a211-53d0-4050-b3ef-b72340e67efa", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.172", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Nov 24 02:10:10 <HOSTNAME> apm-server: 2020-11-24T02:10:10.618+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "9ea602ab-28d0-43ba-ae33-ddb21cf16e9d", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.172", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Nov 24 02:10:10 <HOSTNAME> apm-server: 2020-11-24T02:10:10.633+0530#011INFO#011[request]#011middleware/log_middleware.go:76#011request accepted#011{"request_id": "ab1adb2c-8676-462a-8b52-5cbd6373083d", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.191.155.173", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}

able to self telnet on 3997.

Thanks,

gil · November 24, 2020, 5:39pm

I don't see an attachment for the kibana error but guess you're running into a common issue when APM Server isn't communicating directly with Elasticsearch - the index templates are missing and won't create the APM indices as kibana expects them. Please refer to this guide to help resolve the issue. Briefly this entails:

Stop ingestion (the kafka consumer in this case)
Delete existing indices
Load index mapping templates
Start ingestion

prat · November 24, 2020, 9:37pm

Hi Gil,

Thanks for your prompt response.

Here is the attachment.

As per your reply,

Stop ingestion (the kafka consumer in this case) -
you mean to say, # systemctl stop kafka.service?
Delete existing indices -
I was able to list the indices in Dev tools in Kibana using GET /_cat/indices?v
Is it the write command to delete APM* indices using DELETE /apm*
Load index mapping templates -
i) Where can i run below command to check existence of an APM index template.
GET /_template/apm-{version}

can I run above command in Dev tools ?

and if it exists, can I proceed for below steps?

ii) How can I verify that we are not directly outputting data from APM server to Elasticsearch.
if this is true, do I need to Load the template manually?
If yes, is below command correct to load the index?

apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

in document, for above command a note is given as The examples here assume that Logstash output is enabled. How can I verify this?

Do we need to run curl -XDELETE 'http://localhost:9200/apm-server-*' to force kibana to look at newest documents?

Start ingestion -
#systemctl start Kafka-service?

Thanks once again.

prat · November 25, 2020, 8:16am

Hi,

Can someone please reply.

sqren · November 25, 2020, 2:38pm

Hey,

I also don't see the error from Kibana attached. It looks like the logs are from APM Server.
If you can find the Kibana logs, that would be very helpful.

prat · November 25, 2020, 3:50pm

the attachment is from APM tab from kibana dashboard. You can see it says Sorry, an error occured

I would like to know why there is no data shown in APM tab.

(cut the below output due to limit)

journalctl -u kibana

Nov 25 20:52:06 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:22:06Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to
Nov 25 20:52:07 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:22:07Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"

journalctl -fu kibana

-- Logs begin at Wed 2020-11-25 11:59:40 IST. --
bulks\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"bulk [default_local] reports failures when exporting documents\\\",\\\"exceptions\\\":[{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:52:57 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:22:57Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
\"used_in_bytes\\\":16347914240},\\\"uptime_in_millis\\\":738827000,\\\"platform\\\":\\\"linux\\\",\\\"platformRelease\\\":\\\"linux-3.10.0-1062.el7.x86_64\\\",\\\"distro\\\":\\\"Red Hat Linux\\\",\\\"distroRelease\\\":\\\"Red Hat Linux-7.6\\\"},\\\"process\\\":{\\\"event_loop_delay\\\":1.2810907363891602,\\\"memory\\\":{\\\"heap\\\":{\\\"total_in_bytes\\\":260448256,\\\"used_in_bytes\\\":215544528,\\\"size_limit\\\":1526909922},\\\"resident_set_size_in_bytes\\\":297082880},\\\"uptime_in_millis\\\":181768463},\\\"requests\\\":{\\\"disconnects\\\":0,\\
\",\\\"reason\\\":\\\"bulk [default_local] reports failures when exporting documents\\\",\\\"exceptions\\\":[{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:53:47 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:23:47Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
Nov 25 20:53:48 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:23:48Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":{\\\"task\\\":{\\\"taskType\\\":\\\"maps_telemetry\\\",\\\"state\\\":\\\"{}\\\",\\\"attempts\\\":1,\\\"scheduledAt\\\":\\\"2020-11-19T12:45:52.098Z\\\",\\\"startedAt\\\":\\\"2020-11-25T15:23:48.308Z\\\",\\\"retryAt\\\":\\\"2020-11-25T15:29:48.308Z\\\",\\\"runAt\\\":\\\"2020-11-24T18:30:00.000Z\\\",\\\"status\\\":\\\"running\\\"},\\\"updated_at\\\":\\\"2020-11-25T15:23:48.308Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}

ps -ef |grep kibana

kibana    9604     1  0 Nov23 ?        00:30:18 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

systemctl status kibana.service -l

  kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-11-23 18:24:19 IST; 2 days ago
 Main PID: 9604 (node)
   CGroup: /system.slice/kibana.service
           └─9604 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Nov 25 20:56:37 ITFOOPENBANKINGAPP02UAT kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[export_exception] failed to flush export bulks :: {\"path\":\"/_monitoring/bulk\",\"query\":{\"system_id\":\"kibana\",\"system_api_version\":\"6\",\"interval
\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:56:37 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
Nov 25 20:56:39 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:39Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":{\\\"task\\\":{\\\"taskType\\\":\\\"maps_telemetry\\\",\\\"state\\\":\\\"25T15:26:39.643Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}
Nov 25 20:56:47 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:47Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[export_exception] failed to flush export bulks :: {\"path\":\"/_monitoring/bulk\",\"query\":{\"system_id\":\"kibana\",\"system_api_version\":\"6\",\"interval\":\"10000ms\"},\"body\":\"{\\\"index\\\":{\\\"_type\\\":\\"used_in_bytes\\\":217684152,\\\"size_limit\\\":1526909922},\\\"resident_set_size_in_bytes\\\":296951808},\\\"uptime_in_millis\\\":181948501},\\\"requests\\\":{\\\"disconnects\\\":0,\\\"total\\\":0},\\\"response_times\\\":{\\\"average\\\":0,\\\"max\\\":0},\\\"timestamp\\\":\\\"2020-11-25T15:26:47.602Z\\\"}\\n\",\"statusCode\":500,\"response\":\"{\\\"took\\\":9,\\\"ignored\\\":false,\\\"errors\\\":true,\\\"error\\\":{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"failed to flush export bulks\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"bulk [default_local] reports failures when exporting documents\\\",\\\"exceptions\\\":[{\\\"type\\\":\\\"export_exception\\\",\\\"reason\\\":\\\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}}]}}}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:56:47 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:47Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
Nov 25 20:56:51 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:51Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":{\\\"task\\\":{\\\"taskType\\\":\\\"maps_telemetry\\\",\\\"state\\\":\\\"{\\\\\\\"runs\\\\\\\":1,\\\\\\\"stats\\\\\\\":{\\\\\\\"mapsTotalCount\\\\\\\":0,\\\\\\\"timeCaptured\\\\\\\":\\\\\\\"2020-11-23T18:30:01.884Z\\\\\\\",\\\\\\\"attributesPerMap\\\\\\\":{\\\\\"updated_at\\\":\\\"2020-11-25T15:26:51.662Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}
Nov 25 20:56:54 <HOSTNAME> kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:54Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {\"path\":\"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry\",\"query\":{\"if_seq_no\":23,\"if_primary_term\":16,\"refresh\":\"wait_for\"},\"body\":\"{\\\"doc\\\":23T18:30:01.884Z\\\\\\\",\\\\\\\"attributesPerMap\\\\\\\":{\\\\\"updated_at\\\":\\\"2020-11-25T15:26:54.668Z\\\",\\\"references\\\":[]}}\",\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"}],\\\"type\\\":\\\"cluster_block_exception\\\",\\\"reason\\\":\\\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\\\"},\\\"status\\\":403}\"}"}

cat /etc/kibana/kibana.yml

# Ansible managed

server.port: 8601
server.host: "0.0.0.0"
server.name: kibana_1

elasticsearch.requestTimeout: 30000

elasticsearch.hosts: ['http://<IP of localhost>:5200']
elasticsearch.username: "elastic"
elasticsearch.password: "Password"

prat · November 26, 2020, 6:48am

systemctl status kibana.service -l

kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-11-23 18:24:19 IST; 2 days ago
Main PID: 9604 (node)
CGroup: /system.slice/kibana.service
└─9604 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Nov 25 20:56:37 kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[export_exception] failed to flush export bulks :: {"path":"/_monitoring/bulk","query":{"system_id":"kibana","system_api_version":"6","interval":"10000ms"},"body":"{\"index\":{\"_type\":\"kibana_stats\"}}\n{\"kibana\":{\"uuid\":\"1329e789-0ad5-49e2-8b65-bb4f31a71c73\",\"name\":\"kibana_1\",\"index\":\".kibana\",\"host\":\"0.0.0.0\",\"transport_address\":\"0.0.0.0:8601\",\"version\":\"7.4.0\",\"snapshot\":false,\"status\":\"green\"},\"usage\":{\"kql\":{\"optInCount\":0,\"optOutCount\":0,\"defaultQueryLanguage\":\"default-kuery\"},\"localization\":{\"locale\":\"en\",\"integrities\":{},\"labelsCount\":0},\"ui_metric\":{\"infra_metrics\":[{\"key\":\"pageview__inventory__delayed_15000ms\",\"value\":1},{\"key\":\"pageview__inventory\",\"value\":2}],\"uptime\":[{\"key\":\"pageview__overview\",\"value\":1},{\"key\":\"pageview__overview__delayed_15000ms\",\"value\":1}]},\"index\":\".kibana\",\"dashboard\":{\"total\":95},\"visualization\":{\"total\":581},\"search\":{\"total\":57},\"index_pattern\":{\"total\":5},\"graph_workspace\":{\"total\":0},\"timelion_sheet\":{\"total\":0},\"spaces\":{\"available\":true,\"enabled\":true,\"count\":1,\"usesFeatureControls\":false,\"disabledFeatures\":{\"discover\":0,\"visualize\":0,\"dashboard\":0,\"dev_tools\":0,\"advancedSettings\":0,\"indexPatterns\":0,\"savedObjectsManagement\":0,\"graph\":0,\"monitoring\":0,\"ml\":0,\"apm\":0,\"maps\":0,\"canvas\":0,\"infrastructure\":0,\"logs\":0,\"siem\":0,\"uptime\":0}},\"ml\":{\"file_data_visualizer\":{\"index_creation_count\":0}},\"apm\":{\"has_any_services\":false,\"services_per_agent\":{}},\"maps\":{\"mapsTotalCount\":0,\"timeCaptured\":\"2020-11-23T18:30:01.884Z\",\"attributesPerMap\":{\"dataSourcesCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layersCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layerTypesCount\":{},\"emsVectorLayersCount\":{}}},\"canvas\":{},\"cloud\":{\"isCloudEnabled\":false},\"infraops\":{\"last_24_hours\":{\"hits\":{\"infraops_hosts\":0,\"infraops_docker\":0,\"infraops_kubernetes\":0,\"logs\":9}}},\"rollups\":{\"index_patterns\":{\"total\":0},\"saved_searches\":{\"total\":0},\"visualizations\":{\"total\":0,\"saved_searches\":{\"total\":0}}},\"upgrade-assistant-telemetry\":{\"ui_open\":{\"overview\":0,\"cluster\":0,\"indices\":0},\"ui_reindex\":{\"close\":0,\"open\":0,\"start\":0,\"stop\":0},\"features\":{\"deprecation_logging\":{\"enabled\":true}}},\"uptime\":{\"last_24_hours\":{\"hits\":{\"overview_page\":1,\"monitor_page\":0}}},\"visualization_types\":{\"metrics\":{\"total\":237,\"spaces_min\":237,\"spaces_max\":237,\"spaces_avg\":237},\"metric\":{\"total\":39,\"spaces_min\":39,\"spaces_max\":39,\"spaces_avg\":39},\"pie\":{\"total\":87,\"spaces_min\":87,\"spaces_max\":87,\"spaces_avg\":87},\"histogram\":{\"total\":38,\"spaces_min\":38,\"spaces_max\":38,\"spaces_avg\":38},\"table\":{\"total\":42,\"spaces_min\":42,\"spaces_max\":42,\"spaces_avg\":42},\"tagcloud\":{\"total\":9,\"spaces_min\":9,\"spaces_max\":9,\"spaces_avg\":9},\"input_control_vis\":{\"total\":14,\"spaces_min\":14,\"spaces_max\":14,\"spaces_avg\":14},\"tile_map\":{\"total\":17,\"spaces_min\":17,\"spaces_max\":17,\"spaces_avg\":17},\"area\":{\"total\":13,\"spaces_min\":13,\"spaces_max\":13,\"spaces_avg\":13},\"timelion\":{\"total\":39,\"spaces_min\":39,\"spaces_max\":39,\"spaces_avg\":39},\"line\":{\"total\":33,\"spaces_min\":33,\"spaces_max\":33,\"spaces_avg\":33},\"horizontal_bar\":{\"total\":3,\"spaces_min\":3,\"spaces_max\":3,\"spaces_avg\":3},\"markdown\":{\"total\":8,\"spaces_min\":8,\"spaces_max\":8,\"spaces_avg\":8},\"gauge\":{\"total\":1,\"spaces_min\":1,\"spaces_max\":1,\"spaces_avg\":1},\"heatmap\":{\"total\":1,\"spaces_min\":1,\"spaces_max\":1,\"spaces_avg\":1}},\"csp\":{\"strict\":false,\"warnLegacyBrowsers\":true,\"rulesChangedFromDefault\":false},\"fileUploadTelemetry\":{\"filesUploadedTotalCount\":0},\"xpack\":{\"reporting\":{\"available\":true,\"browser_type\":\"chromium\",\"enabled\":true,\"lastDay\":{\"_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"last7Days\":{\"_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}}}},\"concurrent_connections\":0,\"os\":{\"load\":{\"1m\":0.21728515625,\"5m\":0.10888671875,\"15m\":0.12841796875},\"memory\":{\"total_in_bytes\":16825806848,\"free_in_bytes\":535048192,\"used_in_bytes\":16290758656},\"uptime_in_millis\":738997000,\"platform\":\"linux\",\"platformRelease\":\"linux-3.10.0-1062.el7.x86_64\",\"distro\":\"Red Hat Linux\",\"distroRelease\":\"Red Hat Linux-7.6\"},\"process\":{\"event_loop_delay\":1.3422919511795044,\"memory\":{\"heap\":{\"total_in_bytes\":259399680,\"used_in_bytes\":215711952,\"size_limit\":1526909922},\"resident_set_size_in_bytes\":296386560},\"uptime_in_millis\":181938497},\"requests\":{\"disconnects\":0,\"total\":0},\"response_times\":{\"average\":0,\"max\":0},\"timestamp\":\"2020-11-25T15:26:37.597Z\"}\n","statusCode":500,"response":"{\"took\":8,\"ignored\":false,\"errors\":true,\"error\":{\"type\":\"export_exception\",\"reason\":\"failed to flush export bulks\",\"caused_by\":{\"type\":\"export_exception\",\"reason\":\"bulk [default_local] reports failures when exporting documents\",\"exceptions\":[{\"type\":\"export_exception\",\"reason\":\"ClusterBlockException[index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]\",\"caused_by\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.monitoring-kibana-7-2020.11.25] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}}]}}}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 25 20:56:37 kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:37Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}
Nov 25 20:56:39 kibana[9604]: {"type":"log","@timestamp":"2020-11-25T15:26:39Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: [cluster_block_exception] index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana_task_manager/_update/task%3AMaps-maps_telemetry","query":{"if_seq_no":23,"if_primary_term":16,"refresh":"wait_for"},"body":"{\"doc\":{\"task\":{\"taskType\":\"maps_telemetry\",\"state\":\"{\\\"runs\\\":1,\\\"stats\\\":{\\\"mapsTotalCount\\\":0,\\\"timeCaptured\\\":\\\"2020-11-23T18:30:01.884Z\\\",\\\"attributesPerMap\\\":{\\\"dataSourcesCount\\\":{\\\"min\\\":0,\\\"max\\\":0,\\\"avg\\\":0},\\\"layersCount\\\":{\\\"min\\\":0,\\\"max\\\":0,\\\"avg\\\":0},\\\"layerTypesCount\\\":{},\\\"emsVectorLayersCount\\\":{}}}}\",\"params\":\"{}\",\"attempts\":1,\"scheduledAt\":\"2020-11-19T12:45:52.098Z\",\"startedAt\":\"2020-11-25T15:26:39.643Z\",\"retryAt\":\"2020-11-25T15:32:39.643Z\",\"runAt\":\"2020-11-24T18:30:00.000Z\",\"status\":\"running\"},\"updated_at\":\"2020-11-25T15:26:39.643Z\",\"references\":}}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"}

prat · November 26, 2020, 6:58am

journalctl -fu kibana

-- Logs begin at Thu 2020-11-26 02:31:40 IST. --
Nov 26 12:22:33 kibana[9604]: {"type":"log","@timestamp":"2020-11-26T06:52:33Z","tags":["error","task_manager"],"pid":9604,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
Nov 26 12:22:43 kibana[9604]: {"type":"log","@timestamp":"2020-11-26T06:52:43Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"[cluster_block_exception] blocked by: [SERVICE_UNAVAILABLE/2/no master]; :: {"path":"/_monitoring/bulk","query":{"system_id":"kibana","system_api_version":"6","interval":"10000ms"},"body":"{\"index\":{\"_type\":\"kibana_stats\"}}\n{\"kibana\":{\"uuid\":\"1329e789-0ad5-49e2-8b65-bb4f31a71c73\",\"name\":\"kibana_1\",\"index\":\".kibana\",\"host\":\"0.0.0.0\",\"transport_address\":\"0.0.0.0:8601\",\"version\":\"7.4.0\",\"snapshot\":false,\"status\":\"green\"},\"usage\":{\"kql\":{\"optInCount\":0,\"optOutCount\":0,\"defaultQueryLanguage\":\"default-kuery\"},\"localization\":{\"locale\":\"en\",\"integrities\":{},\"labelsCount\":0},\"ui_metric\":{\"infra_metrics\":[{\"key\":\"pageview__inventory__delayed_15000ms\",\"value\":1},{\"key\":\"pageview__inventory\",\"value\":2}],\"uptime\":[{\"key\":\"pageview__overview\",\"value\":1},{\"key\":\"pageview__overview__delayed_15000ms\",\"value\":1}]},\"index\":\".kibana\",\"dashboard\":{\"total\":95},\"visualization\":{\"total\":581},\"search\":{\"total\":57},\"index_pattern\":{\"total\":5},\"graph_workspace\":{\"total\":0},\"timelion_sheet\":{\"total\":0},\"spaces\":{\"available\":true,\"enabled\":true,\"count\":1,\"usesFeatureControls\":false,\"disabledFeatures\":{\"discover\":0,\"visualize\":0,\"dashboard\":0,\"dev_tools\":0,\"advancedSettings\":0,\"indexPatterns\":0,\"savedObjectsManagement\":0,\"graph\":0,\"monitoring\":0,\"ml\":0,\"apm\":0,\"maps\":0,\"canvas\":0,\"infrastructure\":0,\"logs\":0,\"siem\":0,\"uptime\":0}},\"ml\":{\"file_data_visualizer\":{\"index_creation_count\":0}},\"apm\":{\"has_any_services\":false,\"services_per_agent\":{}},\"maps\":{\"mapsTotalCount\":0,\"timeCaptured\":\"2020-11-23T18:30:01.884Z\",\"attributesPerMap\":{\"dataSourcesCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layersCount\":{\"min\":0,\"max\":0,\"avg\":0},\"layerTypesCount\":{},\"emsVectorLayersCount\":{}}},\"canvas\":{},\"cloud\":{\"isCloudEnabled\":false},\"infraops\":{\"last_24_hours\":{\"hits\":{\"infraops_hosts\":0,\"infraops_docker\":0,\"infraops_kubernetes\":0,\"logs\":9}}},\"rollups\":{\"index_patterns\":{\"total\":0},\"saved_searches\":{\"total\":0},\"visualizations\":{\"total\":0,\"saved_searches\":{\"total\":0}}},\"upgrade-assistant-telemetry\":{\"ui_open\":{\"overview\":0,\"cluster\":0,\"indices\":0},\"ui_reindex\":{\"close\":0,\"open\":0,\"start\":0,\"stop\":0},\"features\":{\"deprecation_logging\":{\"enabled\":false}}},\"uptime\":{\"last_24_hours\":{\"hits\":{\"overview_page\":1,\"monitor_page\":0}}},\"visualization_types\":{\"metrics\":{\"total\":237,\"spaces_min\":237,\"spaces_max\":237,\"spaces_avg\":237},\"metric\":{\"total\":39,\"spaces_min\":39,\"spaces_max\":39,\"spaces_avg\":39},\"pie\":{\"total\":87,\"spaces_min\":87,\"spaces_max\":87,\"spaces_avg\":87},\"histogram\":{\"total\":38,\"spaces_min\":38,\"spaces_max\":38,\"spaces_avg\":38},\"table\":{\"total\":42,\"spaces_min\":42,\"spaces_max\":42,\"spaces_avg\":42},\"tagcloud\":{\"total\":9,\"spaces_min\":9,\"spaces_max\":9,\"spaces_avg\":9},\"input_control_vis\":{\"total\":14,\"spaces_min\":14,\"spaces_max\":14,\"spaces_avg\":14},\"tile_map\":{\"total\":17,\"spaces_min\":17,\"spaces_max\":17,\"spaces_avg\":17},\"area\":{\"total\":13,\"spaces_min\":13,\"spaces_max\":13,\"spaces_avg\":13},\"timelion\":{\"total\":39,\"spaces_min\":39,\"spaces_max\":39,\"spaces_avg\":39},\"line\":{\"total\":33,\"spaces_min\":33,\"spaces_max\":33,\"spaces_avg\":33},\"horizontal_bar\":{\"total\":3,\"spaces_min\":3,\"spaces_max\":3,\"spaces_avg\":3},\"markdown\":{\"total\":8,\"spaces_min\":8,\"spaces_max\":8,\"spaces_avg\":8},\"gauge\":{\"total\":1,\"spaces_min\":1,\"spaces_max\":1,\"spaces_avg\":1},\"heatmap\":{\"total\":1,\"spaces_min\":1,\"spaces_max\":1,\"spaces_avg\":1}},\"csp\":{\"strict\":false,\"warnLegacyBrowsers\":true,\"rulesChangedFromDefault\":false},\"fileUploadTelemetry\":{\"filesUploadedTotalCount\":0},\"xpack\":{\"reporting\":{\"available\":true,\"browser_type\":\"chromium\",\"enabled\":true,\"lastDay\":{\"_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"last7Days\":{\"_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}},\"_all\":0,\"status\":{\"completed\":0,\"failed\":0},\"printable_pdf\":{\"available\":false,\"total\":0,\"app\":{\"dashboard\":0,\"visualization\":0},\"layout\":{\"preserve_layout\":0,\"print\":0}},\"csv\":{\"available\":true,\"total\":0},\"PNG\":{\"available\":false,\"total\":0}}}},\"concurrent_connections\":0,\"os\":{\"load\":{\"1m\":0.01513671875,\"5m\":0.037109375,\"15m\":0.04541015625},\"memory\":{\"total_in_bytes\":16825806848,\"free_in_bytes\":280158208,\"used_in_bytes\":16545648640},\"uptime_in_millis\":794528000,\"platform\":\"linux\",\"platformRelease\":\"linux-3.10.0-1062.el7.x86_64\",\"distro\":\"Red Hat Linux\",\"distroRelease\":\"Red Hat Linux-7.6\"},\"process\":{\"event_loop_delay\":0.752547025680542,\"memory\":{\"heap\":{\"total_in_bytes\":262545408,\"used_in_bytes\":229375040,\"size_limit\":1526909922},\"resident_set_size_in_bytes\":297926656},\"uptime_in_millis\":237469036},\"requests\":{\"disconnects\":0,\"total\":0},\"response_times\":{\"average\":0,\"max\":0},\"timestamp\":\"2020-11-26T06:52:08.135Z\"}\n","statusCode":503,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [SERVICE_UNAVAILABLE/2/no master];\"}],\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [SERVICE_UNAVAILABLE/2/no master];\"},\"status\":503}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"}
Nov 26 12:22:43 kibana[9604]: {"type":"log","@timestamp":"2020-11-26T06:52:43Z","tags":["warning","monitoring","kibana-monitoring"],"pid":9604,"message":"Unable to bulk upload the stats payload to the local cluster"}

sqren · November 26, 2020, 1:07pm

This error doesn't look APM related:

blocked by: [SERVICE_UNAVAILABLE/2/no master]

Please double check that you've configured elasticsearch correctly, and that the cluster is in a healthy state:

curl localhost:9200/_cat/health

prat · November 26, 2020, 3:04pm

Hi Sqren,

Thanks for reply. Using 5200 as Elasticsearch http port.

-rw-rw---- 1 root elasticsearch 913 Nov 19 18:12 elasticsearch.yml

curl localhost:5200/_cat/health

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/health]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

prat · November 26, 2020, 3:28pm

Is below cluster.initial_master_nodes, discovery.seed_hosts correctly mentioned? or do we need to mentioned them inside [" "] and not [' '].

cat elasticsearch.yml

# Ansible managed

[node.name](http://node.name/): elasticsearch_1

path.data: /opt/elasticsearch

path.logs: /var/log/elasticsearch

http.port: 5200

transport.port: 5300

path.repo: /opt/elastic_snapshot

[cluster.name](http://cluster.name/): xxx-dev

cluster.initial_master_nodes: ['localhost ip']

discovery.seed_hosts: ['localhost ip']

network.host: 0.0.0.0

http.host: 0.0.0.0

node.master: True

node.data: True

node.ingest: True

discovery.zen.minimum_master_nodes: 2

xpack.security.transport.ssl.enabled: true

xpack.security.transport.ssl.verification_mode: certificate

xpack.security.transport.ssl.keystore.path: elasticsearch_1.p12

xpack.security.transport.ssl.truststore.path: elasticsearch_1.p12

# xpack.security.http.ssl.verification_mode: certificate

# xpack.security.http.ssl.keystore.path: elasticsearch_1.p12

# xpack.security.http.ssl.truststore.path: elasticsearch_1.p12

xpack.monitoring.collection.enabled: true

xpack.security.enabled: true

prat · November 27, 2020, 12:58pm

app related errors in /var/log/message

Nov 27 06:32:35 <HOSTNAME> <custom-app-api>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:41 <HOSTNAME> ape-server: 2020-11-27T06:32:41.619+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "8a0b9628-1ba8-4981-ad58-389bb2d9dec1", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<localhost IP>", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 503, "error": "queue is full"} Nov 27 06:32:41 <HOSTNAME> <customer-app-admin-api>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:41 <HOSTNAME> apm-server: 2020-11-27T06:32:41.620+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "a6f911cb-e424-48bb-bc77-e4ed14fab7ce", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<localhost IP>", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 503, "error": "queue is full"} Nov 27 06:32:41 <HOSTNAME> apm-server: 2020-11-27T06:32:41.620+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "74f037f1-40ff-4c80-837f-78510d046434", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<localhost IP>", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 503, "error": "queue is full"} Nov 27 06:32:41 <HOSTNAME> <custom-admin-spa>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:41 <HOSTNAME> <custom-spa>: APM Server transport error (503): Unexpected APM Server response Nov 27 06:32:50 <HOSTNAME> apm-server: 2020-11-27T06:32:50.310+0530#011ERROR#011[request]#011middleware/log_middleware.go:74#011queue is full#011{"request_id": "61cbe81d-2db7-4787-9f64-281e5983dc11", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "<pfpd-server-ip>", "user-agent": "elasticapm-java/1.14.0", "response_code": 503, "error": "queue is full"}

prat · November 30, 2020, 6:13am

Hi All,

entered the correct password.

curl localhost:5200/_cat/health -u elastic
Enter host password for user 'elastic':
1606716560 06:09:20 xxx-dev yellow 1 1 40 40 0 0 15 0 - 72.7%

@gil @sqren , Can you please update.

prat · November 30, 2020, 7:43am

In kibana, Inside HOME / Add data/ APM

for APM Server status - check APM Server status
it shows, You have correctly setup APM Server

then in APM Agents
Java
for Agent status - check agent status
it shows, Data Successfully received from one or more agents

For Load Kibana objects
It gives below error,
Unable to add 1 of 1 kibana objects, Error: index [.kibana_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

in Index Patterns - I can see apm-*.

in Discover tab, I can see apm-* in dropdown and if choose I can see data as below.

Nov 24, 2020 @ 06:55:52.734 parent.id:500e0035de9b1d6b observer.hostname: observer.id:20c18025-5e1e-454e-842a-012955875a57 observer.ephemeral_id:6c8b2636-6cd3-455a-9977-7535f1f8c4bc observer.type:apm-server observer.version:7.4.0 observer.version_major:7 trace.id:92e98e438b09ae57307a534335f0c876 agent.name:java agent.ephemeral_id:a540c853-3c1f-42bf-8df0-dd031797713b agent.version:1.14.0 @timestamp:Nov 24, 2020 @ 06:55:52.734 ecs.version:1.1.0 service.name:pingDirectory @version:1 processor.name:transaction

So why only in APM tab (left side) it shows above error (as in above attachment)

this show ok

# apm-server test config
Config OK

# apm-server test output
Kafka: <localhost ip>:3997...
  parse host... OK
  dns lookup... OK
  addresses: <localhost ip>
  dial up... ERROR dial tcp <localhost ip>:3997: connect: connection refused

telnet <localhost ip> 3997
Trying <localhost ip>...
telnet: connect to address <localhost ip>: Connection refused

netstat -antp |grep 3997

no other service is using 3997.

not sure why 3997 is not listing, apm service is up and running.

systemctl status apm-server.service  -l
 apm-server.service - Elastic APM Server
   Loaded: loaded (/usr/lib/systemd/system/apm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-11-24 21:50:26 IST; 5 days ago
     Docs: https://www.elastic.co/solutions/apm
 Main PID: 3362 (apm-server)
   CGroup: /system.slice/apm-server.service
           └─3362 /usr/share/apm-server/bin/apm-server -e -c /etc/apm-server/apm-server.yml -path.home /usr/share/apm-server -path.config /etc/apm-server -path.data /var/lib/apm-server -path.logs /var/log/apm-server

Nov 30 13:06:56 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:56.783+0530        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Nov 30 13:06:56 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:56.783+0530        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Nov 30 13:06:56 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:56.783+0530        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (1 attempts remaining)
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530        INFO        kafka/log.go:53        client/metadata fetching metadata for [apm] from broker <localhost ip>:3997
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530        INFO        kafka/log.go:53        Failed to connect to broker <localhost ip>:3997: dial tcp <localhost ip>:3997: connect: connection refused
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp <localhost ip>:3997: connect: connection refused
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.033+0530        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.634+0530        INFO        [publisher]        pipeline/retry.go:189        retryer: send unwait-signal to consumer
Nov 30 13:06:57 <HOSTNAME> apm-server[3362]: 2020-11-30T13:06:57.635+0530        INFO        [publisher]        pipeline/retry.go:191          done

sqren · November 30, 2020, 12:57pm

Hi @prat,

From you initial screenshot I see that the request to /api/apm/services was failing. Were you able to find the logs for that error?

If you open your browser's Dev tools you can inspect the API requests made. Please find the one for /api/apm/services and replay it with &_debug=true appended and see the logs for that.

prat · November 30, 2020, 3:16pm

Hi @sqren,

Thanks for reply. I can see below error in /var/log/apm-server/apm-server.7

2020-11-30T13:04:58.086+0530 ERROR instance/beat.go:878 Exiting: Index management requested but the Elasticsearch output is not configured/enabled.

Can you please tell what command need to run to inspect. Sorry I am not from developing background so not sure about this API commands.

prat · December 1, 2020, 3:05pm

Hi @sqren,

Can you please reply.

prat · December 4, 2020, 6:48pm

Hi @gil,

Can you please reply on my update.

system · December 25, 2020, 2:48pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.