Elastic APM Agent Getting HTTP 404

Elastic Stack Kibana
1

Hi,

We have setup a java agent in our server running the java base CA7 API Gateway.

Reference Log:

2019-01-31T18:50:41.279+0800 INFO 9 STDOUT : 2019-01-31 18:50:41.279 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: http://:8200/intake/v2/events, response code is 404

2019-01-31T18:50:41.280+0800 INFO 9 STDOUT : 2019-01-31 18:50:41.280 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Draft//EN">

Reference Server Details:

  • Agent using 1.3.0
  • APM Server using v6.5.4
  • CA7 Server running Java build 1.8.0_73-b02

Additional Details in Kibana:

  • Dashboard was able to get that an agent is running but, no additional details of transactions made is showing in the server

Requesting kindly for your guidance if you have encountered this issues as well.

image
image.jpg1093×609 130 KB

2

Hi!

That URL in the error looks very suspicious, did you edited it to hide the host or you get that error exactly? (http://:8200/intake/v2/events)

For reference, here it says how you can specify the APM Server URL: https://www.elastic.co/guide/en/apm/agent/java/current/configuration.html

Does that help?

3

Hi javlz,

Thank you for the reply. We deliberately edited the host in this post. The is actually a server ip address that we have.

Do note that we also did try health checks, ping to the server and it all returns fine.

4

The message <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Draft//EN"> hints that the server_urls you have configured is a URL of a WebLogic application server. Did you set it to the application you want to monitor? It is supposed to be the APM Server URL.

I hope this helps :slight_smile:

Felix

5

Hi felixbarny,

Thank you for your response. The ip address we have put in the java arguments is the actual APM Server IP.

We even check the logs and it shows on health check status that it is ok.

2019-02-06T12:12:35.041+0800 INFO 6 STDOUT : 2019-02-06 12:12:35.041 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-12-17T20:15:18Z","build_sha":"fd4f1ac006d8896e4957b6c01fb1e9b9ff04afd7","version":"6.5.4"}}

When it do try to http://APM_SERVER_IP:8200/intake/v2/events it is getting the HTTP 404 with the HTML response.

6

What happens if you do curl -XPOST http://APM_SERVER_IP:8200/intake/v2/events? Could it be that your APM Server is behind a proxy and it does not forward the intake/v2/events path?

7

Hi Felix,

Doing a curl actually return a non HTTP 404 response.

curl -v -X POST http://API_SERVER_IP:8002

About to connect() to API_SERVER_IPport 8200 (#0)

  • Trying API_SERVER_IP... connected
  • Connected to API_SERVER_IP (API_SERVER_IP) port 8200 (#0)

POST /intake/v2/events HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: APM_SERVER_IP:8200
Accept: /

< HTTP/1.1 400 Bad Request
< Connection: Close
< Content-Type: application/json
< Date: Wed, 06 Feb 2019 11:00:45 GMT
< Content-Length: 64
<

  • Closing connection #0
    {"accepted":0,"errors":[{"message":"invalid content type: ''"}]}

To answer about the APM Server could be behind proxy, we honestly believe that is not the case since other servers using Java was able to send data in intake/v2/events.

8

That curl response looks like it should.

Could it be that you added proxy settings to the JVM (-Dhttp.proxyHost system property)? See https://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html

If so, adding -Dhttp.noProxyHosts="APM_SERVER_IP" would probably help.

9

Hi Felix,

Thank you for the input. Will check on this. :smile:

10

Hi Felix,

We tried to check if any of the running JVM in the server has the -Dhttp.proxyHost argument. Somehow we don't see it.

/opt/SecureSpan/Appliance/libexec/gateway_control pc run -J-Dcom.l7tech.server.home="/opt/SecureSpan/Gateway/node/default" -J-Dcom.l7tech.server.processControllerPresent=true -J-Dcom.l7tech.server.processControllerCert=/opt/SecureSpan/Controller/var/run/pc.cer -J-Dcom.l7tech.server.processControllerIpAddress=127.0.0.1 -J-Dcom.l7tech.server.processControllerPort=8765 -J-Djava.util.logging.config.class=com.l7tech.server.log.JdkLogConfig -J-Dcom.l7tech.server.log.console=true

/bin/bash /opt/SecureSpan/Appliance/libexec/gateway_control pc run -J-Dcom.l7tech.server.home="/opt/SecureSpan/Gateway/node/default" -J-Dcom.l7tech.server.processControllerPresent=true -J-Dcom.l7tech.server.processControllerCert=/opt/SecureSpan/Controller/var/run/pc.cer -J-Dcom.l7tech.server.processControllerIpAddress=127.0.0.1 -J-Dcom.l7tech.server.processControllerPort=8765 -J-Djava.util.logging.config.class=com.l7tech.server.log.JdkLogConfig -J-Dcom.l7tech.server.log.console=true

/opt/SecureSpan/JDK/bin/java -Dcom.l7tech.server.home=/opt/SecureSpan/Gateway/node/default -Djava.ext.dirs=/opt/SecureSpan/JDK/jre/lib/ext:/opt/SecureSpan/Gateway/runtime/lib/ext -Dcom.l7tech.server.siteminder.enabled=true -server -Djava.net.preferIPv4Stack=true -Djava.security.policy=/opt/SecureSpan/Gateway/runtime/etc/ssg.policy -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF-8 -Dsun.net.inetaddr.ttl=30 -Djava.awt.headless=true -XX:CompileThreshold=1500 -Dcom.l7tech.server.defaultClusterHostname=CA7-AP-DEV.hostname.net -Djava.util.Arrays.useLegacyMergeSort=true -Xmx20232394k -Xss256k -XX:+UseParallelOldGC -XX:+UseCompressedOops -Dcom.l7tech.server.transport.reservedPorts=7001,7100 -javaagent:/home/gateway/elastic-apm-agent-1.3.0.jar -Delastic.apm.service_name=dev-ca7 -Delastic.apm.application_packages=. -Delastic.apm.server_url=http://APM_SERVER_IP:8200 -XX:MaxPermSize=256M -Dcom.l7tech.server.home="/opt/SecureSpan/Gateway/node/default" -Dcom.l7tech.server.processControllerPresent=true -Dcom.l7tech.server.processControllerCert=/opt/SecureSpan/Controller/var/run/pc.cer -Dcom.l7tech.server.processControllerIpAddress=127.0.0.1 -Dcom.l7tech.server.processControllerPort=8765 -Djava.util.logging.config.class=com.l7tech.server.log.JdkLogConfig -Dcom.l7tech.server.log.console=true -jar /opt/SecureSpan/Gateway/runtime/Gateway.jar pc run

/bin/bash /opt/SecureSpan/Appliance/libexec/gateway_control pc run -J-Dcom.l7tech.server.home="/opt/SecureSpan/Gateway/node/default" -J-Dcom.l7tech.server.processControllerPresent=true -J-Dcom.l7tech.server.processControllerCert=/opt/SecureSpan/Controller/var/run/pc.cer -J-Dcom.l7tech.server.processControllerIpAddress=127.0.0.1 -J-Dcom.l7tech.server.processControllerPort=8765 -J-Djava.util.logging.config.class=com.l7tech.server.log.JdkLogConfig -J-Dcom.l7tech.server.log.console=true

11

Have you tried setting -Dhttp.noProxyHosts="APM_SERVER_IP"?

To get some more insight, you could enable logging for HttpUrlConnection: https://stackoverflow.com/a/12339718/1125055

That should also log any proxies used for the connection.

12

Hi Felix,

Yes, we had tried putting the argument but, same issue still persist.

As additional step for troubleshooting we had installed the APM Server in same machine as the CA7 API Gateway. We also get same error (HTTP 404) when we point to localhost server of APM.

13

It seems like the requests to the APM server are proxied through the CA7 API Gateway. CA7 runs on WebLogic which prints <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Draft//EN"> on 404 pages. You need to either make sure the /intake/v2/events path is proxied correctly as well or to remove the proxy for the APM Server.

14

Hi Felix,

Let us try to check out your suggestion to our infra team. Hopefully this help us resolve the http 404. :slight_smile:

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.