APM server deployed via fleet cannot connect to Kibana

Elastic Observability APM
1

Hi,

I experience problems with APM server deployed via fleet.

Kibana version: 7.14.0

Elasticsearch version: 7.14.0

APM Server version: 7.14.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Fleet

Fresh install or upgraded from other version? fresh install

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

The APM server cannot connect to Kibana. The connection attempt is to http://localhost:5601, but I have SSL enabled and the connection to localhost in general fails, I need to provide the hostname.

Provide logs and/or server output (if relevant):

/opt/Elastic/Agent/data/elastic-agent-e127fc/logs/default/apm-server-json.log:

{"log.level":"info","@timestamp":"2021-08-25T09:34:07.817+0200","log.origin":{"file.name":"kibana/client.go","file.line":122},"message":"Kibana url: http://localhost:5601","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-08-25T09:34:07.819+0200","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":79},"message":"failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused. Response: .","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}

How would I configure the elastic agent and/or the APM integration to ignore the proxy and to use the hostname and to accept a self signed certificate?

Best regards,
Robert

2

@leprovokateur when deployed via Fleet, APM Server should not need to communicate with Kibana.

Did you do something (make an HTTP request, etc.) to trigger that error log? Otherwise, if you could provide the rest of the log we will have a better chance of understanding why that's being logged.

Are there any other issues occurring, or did you just notice the error when looking at the log and think it needed correcting?

3

Hi,
I am not aware of anything, that could have triggered that error. I just upgraded the cluster from 7.13.2 to 7.14.0.

I stopped the elastic-agent and rotated the log, so the following lines are from a restart. Please note, that I removed the timestamps as well as some lines from the beginning and some after the lines to meet the length limitation of the post.

"info",","log.origin":{"file.name":"kibana/client.go","file.line":122},"message":"Kibana url: http://localhost:5601","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"error",","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":79},"message":"failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get \"http://localhost:5601/api/status\": dial tcp 127.0.0.1:5601: connect: connection refused. Response: .","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path / added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path /assets/v1/sourcemaps added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":127},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":213},"message":"Starting apm-server [af26b5a744b6af9bcccb04fade6435241ab91f43 built 2021-07-29 19:55:24 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":113},"message":"Listening on: 192.168.58.82:8200","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":119},"message":"RUM endpoints enabled!","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"warn",","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":122},"message":"CORS related setting `apm-server.rum.allow_origins` allows all origins. Consider more restrictive setting for production use.","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":150},"message":"SSL disabled.","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.origin":{"file.name":"kibana/client.go","file.line":122},"message":"Kibana url: http://localhost:5601","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"error",","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":79},"message":"failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get \"http://localhost:5601/api/status\": dial tcp [::1]:5601: connect: connection refused. Response: .","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","service.name":"apm-server","event.dataset":"apm-server-json.log","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-python/6.3.3","source.address":"192.168.58.82","http.request.body.bytes":436,"http.request.id":"79fad178-0d2c-40ea-a446-f542ee107e57","event.duration":4205299,"http.response.status_code":202,"ecs.version":"1.6.0"}
"info",","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":143},"message":"Connecting to backoff(elasticsearch(https://q4deumsy0ka.mms-at-work.de:9200))","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":219},"message":"retryer: send unwait signal to consumer","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":223},"message":"  done","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"warn",","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":98},"message":"SSL/TLS verifications disabled.","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":273},"message":"Attempting to connect to Elasticsearch version 7.14.0","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/output.go","file.line":151},"message":"Connection to backoff(elasticsearch(https://q4deumsy0ka.mms-at-work.de:9200)) established","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"warn",","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x35bf9520, ext:63765563659, loc:(*time.Location)(nil)}, Meta:null, Fields:{\"agent\":{\"name\":\"python\",\"version\":\"6.3.3\"},\"data_stream.dataset\":\"apm.app.shortener\",\"data_stream.namespace\":\"default\",\"data_stream.type\":\"metrics\",\"ecs\":{\"version\":\"1.10.0\"},\"host\":{\"architecture\":\"x86_64\",\"hostname\":\"q4deumsy0ka\",\"ip\":\"192.168.58.82\",\"name\":\"q4deumsy0ka\",\"os\":{\"platform\":\"linux\"}},\"metricset.name\":\"app\",\"observer\":{\"ephemeral_id\":\"d49498b8-2911-4a5d-9307-0dd57761e59c\",\"hostname\":\"q4deumsy0ka\",\"id\":\"dec3cc95-bf8a-4ed2-a85b-dfa3d8e8b250\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"process\":{\"args\":[\"/usr/local/bin/flask\",\"run\"],\"pid\":46716,\"ppid\":45690},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"environment\":\"production\",\"framework\":{\"name\":\"flask\",\"version\":\"2.0.1\"},\"language\":{\"name\":\"python\",\"version\":\"3.6.8\"},\"name\":\"Shortener\",\"node\":{\"name\":\"q4deumsy0ka\"},\"runtime\":{\"name\":\"CPython\",\"version\":\"3.6.8\"}},\"system\":{\"cpu\":{\"total\":{\"norm\":{\"pct\":0.22180958811613774}}},\"memory\":{\"actual\":{\"free\":1198178304},\"total\":6067875840},\"process\":{\"cpu\":{\"total\":{\"norm\":{\"pct\":0.00016880486158001352}}},\"memory\":{\"rss\":{\"bytes\":33783808},\"size\":658227200}}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [metrics-apm.app-0.2.0] does not exist\"}","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.origin":{"file.name":"kibana/client.go","file.line":122},"message":"Kibana url: http://localhost:5601","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"error",","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":79},"message":"failed to obtain connection to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get \"http://localhost:5601/api/status\": dial tcp [::1]:5601: connect: connection refused. Response: .","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}
"info",","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":63},"message":"request accepted","service.name":"apm-server","event.dataset":"apm-server-json.log","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-python/6.3.3","source.address":"192.168.58.82","http.request.body.bytes":437,"http.request.id":"39b20075-0e74-4e91-92f7-0f5a6b077f4b","event.duration":770383,"http.response.status_code":202,"ecs.version":"1.6.0"}

If I use a web application instrumented with apm I see the following:

{"log.level":"warn","@timestamp":"2021-08-26T10:46:31.677+0200","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc041f319b99ba125, ext:754351181645, loc:(*time.Location)(0x55e26a5f9060)}, Meta:null, Fields:{\"agent\":{\"name\":\"rum-js\",\"version\":\"5.9.1\"},\"client\":{\"ip\":\"10.34.213.205\"},\"data_stream.dataset\":\"apm\",\"data_stream.namespace\":\"default\",\"data_stream.type\":\"traces\",\"ecs\":{\"version\":\"1.10.0\"},\"event\":{\"outcome\":\"unknown\"},\"observer\":{\"ephemeral_id\":\"d49498b8-2911-4a5d-9307-0dd57761e59c\",\"hostname\":\"q4deumsy0ka\",\"id\":\"dec3cc95-bf8a-4ed2-a85b-dfa3d8e8b250\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"parent\":{\"id\":\"f4377cd65b18c75b\"},\"processor\":{\"event\":\"span\",\"name\":\"transaction\"},\"service\":{\"language\":{\"name\":\"javascript\"},\"name\":\"Shorty Frontend\"},\"span\":{\"duration\":{\"us\":1000},\"id\":\"c5e4909ca724e811\",\"name\":\"Fire \\\"DOMContentLoaded\\\" event\",\"start\":{\"us\":411000},\"subtype\":\"browser-timing\",\"type\":\"hard-navigation\"},\"timestamp\":{\"us\":1629967590966500},\"trace\":{\"id\":\"186f8c59859e676734557630e3c59e1a\"},\"transaction\":{\"id\":\"f4377cd65b18c75b\"},\"user_agent\":{\"original\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36\"}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [D2fSfXsByG0FpvSKa1e7] of user [elastic/fleet-server] on indices [traces-apm-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}

I am not sure, if this is separate issue.

Best regards,
Robert

4

Hi,

I updated the Fleet integrations of Elastic APM and Elastic Agent and changed the agent policy accordingly and the connection attempt to kibana disappeared.

I still have the following line in the logs:

{"log.level":"warn","@timestamp":"2021-08-26T15:31:58.425+0200","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x52c3418, ext:63765581510, loc:(*time.Location)(nil)}, Meta:null, Fields:{\"agent\":{\"name\":\"python\",\"version\":\"6.3.3\"},\"data_stream.dataset\":\"apm.app.shortener\",\"data_stream.namespace\":\"default\",\"data_stream.type\":\"metrics\",\"ecs\":{\"version\":\"1.10.0\"},\"host\":{\"architecture\":\"x86_64\",\"hostname\":\"q4deumsy0ka\",\"ip\":\"192.168.58.82\",\"name\":\"q4deumsy0ka\",\"os\":{\"platform\":\"linux\"}},\"metricset.name\":\"app\",\"observer\":{\"ephemeral_id\":\"b15e31ae-9eda-465a-a804-bfa462e1c189\",\"hostname\":\"q4deumsy0ka\",\"id\":\"dec3cc95-bf8a-4ed2-a85b-dfa3d8e8b250\",\"type\":\"apm-server\",\"version\":\"7.14.0\",\"version_major\":7},\"process\":{\"args\":[\"/usr/local/bin/flask\",\"run\"],\"pid\":46716,\"ppid\":45690},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"environment\":\"production\",\"framework\":{\"name\":\"flask\",\"version\":\"2.0.1\"},\"language\":{\"name\":\"python\",\"version\":\"3.6.8\"},\"name\":\"Shortener\",\"node\":{\"name\":\"q4deumsy0ka\"},\"runtime\":{\"name\":\"CPython\",\"version\":\"3.6.8\"}},\"system\":{\"cpu\":{\"total\":{\"norm\":{\"pct\":0.09948743803041761}}},\"memory\":{\"actual\":{\"free\":1248878592},\"total\":6067875840},\"process\":{\"cpu\":{\"total\":{\"norm\":{\"pct\":0.00016805310478111083}}},\"memory\":{\"rss\":{\"bytes\":34172928},\"size\":658489344}}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"pipeline with id [metrics-apm.app-0.2.0] does not exist\"}","service.name":"apm-server","event.dataset":"apm-server-json.log","ecs.version":"1.6.0"}

It's just a warning, but I'd like to have it disappearing as well.

Plus, it is a bit cumbersome to have to update the integrations by hand and not being informed about the updates, and also to remove and re-add the integrations from/to the policy. It would be nice, if they would be migrated automatically.

Best regards,
Robert

5

What version of the integration package do you have installed? It should be 0.3.0.

Plus, it is a bit cumbersome to have to update the integrations by hand and not being informed about the updates, and also to remove and re-add the integrations from/to the policy. It would be nice, if they would be migrated automatically.

Agreed! We are looking into this.

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.