Hi elatic team,
I'm trying to use APM with such releases:
Kibana version: 7.2.0
Elasticsearch version: 7.2.0
APM Server version: 7.2.0
APM Agent language and version: .Net APM Agent
And have some problems, I couldn't push logs to APM server and firstly want to verify APM server status.
I configured APM as was mentioned in documentation and instructions in Kibana page, so I install APM server with RPM instructions and edit next fields only ->
I used localhost, cause APM server is located inside the same pod as Elasticsearch. Pod exist in Azure Kubernetes Service cluster.
Then I check APM-server via button ->
According to official documentation (https://www.elastic.co/guide/en/apm/server/7.4/running-with-systemd.html) we can verify status via 'systemctl status apm-server' command. After I ran it, got this message ->
[root@azure-elasticsearch-prod-0 elasticsearch]# systemctl status apm-server
Failed to get D-Bus connection: Operation not permitted
It looks like my APM server is not configured properly.
Hi @KatiukOleg. systemd is not used by the official Elastic docker container images so, assuming you're using those, you'll need an alternate approach to checking health. To simulate systemd, you should check that the process is running and tail the logs. I'd suggested enabling monitoring as well.
It sounds like apm-server->elasticsearch is ok since the kibana check is passing.
Which agent are you using? Are you able to enable logging from the agent to help narrow down the issue?
Elastic.Apm:Warning: {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (http://MyUser:MyPassword@elasticIP:8200/):
Elastic.Apm.Metrics.MetricSet
System.Net.Http.HttpRequestException: Connection refused
Elastic APM .NET Agent: [CRITICAL] Exception thrown by logging implementation. Log message: `Failed sending events. Following events were not transferred successfully to the server ({ApmServerUrl}):
In Azure I add inbound rule in Network Security Group to allow traffic to 8200 port.
I'll try to check apm-server service inside elastic pod.
is the ElasticIP that of apm-server or of Elasticsearch itself? I would not expect basic authentication for apm-server unless you had some custom proxy in front. Can you test connectivity to apm-server from your application server, if you have curl try: curl -i http://elasticIP:8200.
@gil, thanks. Yes MyUser:MyPassword part in ServerUrl is redundant. Should be removed. I think I chose bad approach initially. I installed APM-server as RPM package inside elasticssearch pod and forgot to edit elasticsearch YAML file to open port (8200) for APM-server. I'm going to set up APM-server on dedicated pod via deployment and service objects. Then return with a result.
But unfortunately I couldnt send items to APM server. Got next errors:
Exception thrown: 'Elastic.Apm.BackendComm.CentralConfigFetcher.FailedToFetchConfigException' in Elastic.Apm.dll
Exception thrown: 'Elastic.Apm.BackendComm.CentralConfigFetcher.FailedToFetchConfigException' in Elastic.Apm.dll
e[40me[37mdbuge[39me[22me[49m: Elastic.Apm[0]
{CentralConfigFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: `<null>'. URL: `http://52.157.137.240:8200/config/v1/agents?service.name=MissionControl&service.environment=Development'. Apm Server base URL: `http://52.157.137.240:8200/'. WaitInterval: 5m. dbgIterationsCount: 2.
+-> Request:
Method: GET, RequestUri: 'http://APMurl:8200/config/v1/agents?service.name=ServiceName&service.environment=Development', Version: 2.0, Content: <null>, Headers:
{
User-Agent: elasticapm-dotnet/1.1.1
User-Agent: System.Net.Http/4.6.28008.03
User-Agent: .NET_Core/2.2.7
Request-Id: |24f69f431f2ffc44b99c9276ccb8ef76.8b178e7b_
}
+-> Response:
StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
{
X-Content-Type-Options: nosniff
Date: Fri, 18 Oct 2019 12:47:51 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 52
}
+-> Response body [length: 52]:
{"error":"forbidden request: endpoint is disabled"}
Elastic.Apm.BackendComm.CentralConfigFetcher+FailedToFetchConfigException: HTTP status code is Forbidden which most likely means that APM Server supports the central configuration endpoint but Kibana connection is not enabled
at Elastic.Apm.BackendComm.CentralConfigFetcher.InterpretResponseStatusCode(HttpResponseMessage httpResponse)
at Elastic.Apm.BackendComm.CentralConfigFetcher.ProcessHttpResponse(HttpResponseMessage httpResponse, String httpResponseBody)
at Elastic.Apm.BackendComm.CentralConfigFetcher.WorkLoopIteration()
It looks like some configuration is missed ....
Any thoughts?
Hey @KatiukOleg, sorry for the delay. It looks like central configuration is not enabled, to that you should configure a kibana host. I wouldn't expect that to prevent ingestion of trace events though, do you see any data in elasticsearch?
Hi @gil, after several times I configured APM server and now can push data to elasticsearch.
I figured out that environment variables in deployment.yaml file didn't work for me (container logs in Azure were rather informative) and apm-server.yml file on host is existed in readonly mode. I couldn't edit it either.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.