Debugging information for JAVA agent

Elastic Observability APM
1

Doing my best to create some custom code to use as a tracer and add into APM. It is the intention to create a transaction and append from different locations spans to it using context. Have started APM/Elastic/kibana stack on my local machine to debug. This does not result in anything. Would have hoped some failure or error, nothing is getting cought on APM/nor on Kibana or Elastic side.

My very simple code is below:

import java.util.concurrent.TimeUnit;

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Transaction;
// import co.elastic.apm.impl.transaction.Span;
import co.elastic.apm.api.Span;

public class TestAPM{
public static void main(String args) {
for(int i=0;i<args.length;i++)
System.out.println(args.length);
String TransactionID = args[0];
String SpanID = args[1];
Transaction transaction = ElasticApm.startTransaction();
try {
transaction.setName("TransactionID");
transaction.setType(Transaction.TYPE_REQUEST);
System.out.println(TransactionID);
System.out.println(SpanID);
// create the span under the transaction
Span span = transaction.createSpan();
try {
span.setName("SpanID");
span.setType("TBD");
TimeUnit.SECONDS.sleep(5);
System.out.println("\nwaiting for 5 seconds during the span");
} catch (Exception e) {
span.captureException(e);
throw e;
} finally {
span.end();
}
} catch (Exception e) {
transaction.captureException(e);
try {
throw e;
} catch (Exception e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
} finally {
System.out.println("\nwaiting for 10 seconds to allow the creation");
System.out.println("of a span in a different process");
try {
TimeUnit.SECONDS.sleep(10);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
transaction.end();
}
}
}

Have also looked at all possible environment variables documented here: https://www.elastic.co/guide/en/apm/agent/java/current/configuration.html

What is the best approach to debug ??

1 Like
2

Hi and thanks for trying out the Java agent.

Did you maybe forget to add the -javaagent flag?

--
Felix

3

Have tried all different combinations of environment variables. The code is about as simple as anything can be. If this does not work APM-agent does not work. Which other locations can you suggest for looking into potential issues. Would love something like a segmentation fault, or an entry in an error log, like this I am at a loss.

4

I admit we have a possibility for an anti-virus kicking in. But where can I possibly identify this is indeed what is happening

5

The thing is that even when you are using the API, you need to attach the javaagent. Otherwise nothing will happen the API does not include an implementation. The impel will be injected into the API only if the javaagent is added.

See here how to enable debug logging: https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html

6

Have managed to define the javaagent as "VM arguments" inside eclipse. Don't understand what the other environment variables are for, mentioned in the documentation.
Nevertheless I get a different error now:

2018-12-17 13:58:55.530 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.1.0 as TestAPM on Java 1.8.0_181 (Oracle Corporation) Windows 7 6.1
Error: Could not find or load main class \

But at least I got an error.

7

The good news is that it is probably configuration, as I can see this all the way in Kibana now:

error id icon

8

Sounds like an Eclipse run configuration problem. Let me know if you need any further help.

9

Don't think Eclipse is the problem.
Now I make it through to the apm-agent call,
but it fails to find the APM-server.
It is running fine,
in apm-server.yml:

Defines the host and port the server is listening on. use "unix:/path/to.sock" to listen on a unix domain socket.

host: "localhost:8200"
When in Cygwin I run this:
netstat -na | grep 8200
TCP 127.0.0.1:8200 0.0.0.0:0 LISTENING
Which is what I expect.
But still getting stuck:
2018-12-17 16:25:47.297 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.1.0 as TestAPM on Java 1.8.0_181 (Oracle Corporation) Windows 7 6.1
Error: Could not find or load main class \

10

Have removed these settings on the Kibana side, one of the posts suggested that:
#apm_oss.errorIndices: apm-*
#apm_oss.spanIndices: apm-*
#apm_oss.transactionIndices: apm-*
#apm_oss.onboardingIndices: apm-*

Now Eclipse is back to
2018-12-17 16:33:38.568 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.1.0 as TestAPM on Java 1.8.0_181 (Oracle Corporation) Windows 7 6.1
Error: Could not find or load main class \

The Kibana side mentions in a repeatable way:
December 17th 2018, 16:33:21.398
listening:127.0.0.1:8200 @timestamp:December 17th 2018, 16:33:21.398 host.name:XXXYYYZZZ beat.hostname:XXXYYYZZZ beat.name:XXXYYYZZZ beat.version:6.5.1 processor.name:onboarding processor.event:onboarding _id:MbzMvGcB2hbqiKRM4C3f _type:doc _index:apm-6.5.1-onboarding-2018.12.17 _score: - view errors: - error id icon: -

Definately not Eclipse,
Kibana and APM just don't work together,
how can I fix that

11

Seems like Java is trying to find a class named \ which it obviously can't find. Could you post a screenshot of your Eclipse run configuration?

Thanks,
Felix

12

Which TAB on the Eclipse config are you looking for.
In fact I managed to get the below JSON visible on the kibana side,
but I still get the class not found error on eclipse:

{
"_index": "apm-6.5.1-onboarding-2018.12.18",
"_type": "doc",
"_id": "I39hwGcBz0kHYb-tITkq",
"_version": 1,
"_score": null,
"_source": {
"listening": "127.0.0.1:8200",
"@timestamp": "2018-12-18T08:14:09.082Z",
"beat": {
"hostname": "XXXYYYZZZ",
"name": "XXXYYYZZZ",
"version": "6.5.1"
},
"host": {
"name": "XXXYYYZZZ"
},
"processor": {
"name": "onboarding",
"event": "onboarding"
}
},
"fields": {
"view errors": [
null
],
"error id icon": [
null
],
"@timestamp": [
"2018-12-18T08:14:09.082Z"
]
},
"sort": [
1545120849082
]
}

13

2018-12-18 09:42:56.206 [apm-server-healthcheck] WARN co.elastic.apm.report.ApmServerHealthChecker - Elastic APM server is not available (Unexpected end of file from server)
2018-12-18 09:42:57.073 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.1.0 as TestAPM on Java 1.8.0_181 (Oracle Corporation) Windows 7 6.1
Error: Could not find or load main class \

The first error "Elastic APM server is not available" is not true, it is listing on the correct port, towards which my program is sending data. Even the data is arriving at Kibana.

The VM arguments on eclipse are below:
-javaagent:H:\elastic-apm-agent-1.1.0.jar
-Delastic.apm.service_name=TestAPM
-Delastic.apm.server_url=http://XXXYYYZZZ:8200
-Delastic.apm.application_packages=org.db
-Delastic.apm.log_level=INFO

14

When I configure like this:
-javaagent:H:\elastic-apm-agent-1.1.0.jar
-Delastic.apm.service_name=TestAPM
-Delastic.apm.server_url=http://localhost:8200
-Delastic.apm.application_packages=org.db
-Delastic.apm.log_level=INFO

Instead of hostname, I get a different error,
but still the same final result

2018-12-18 09:48:43.562 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.1.0 as TestAPM on Java 1.8.0_181 (Oracle Corporation) Windows 7 6.1
2018-12-18 09:48:43.641 [apm-server-healthcheck] INFO co.elastic.apm.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-11-16T01:21:45Z","build_sha":"e9b22404feb418663d86f059b833bf1487d9379e","version":"6.5.1"}}
Error: Could not find or load main class \

15

If you remove the JVM arguments, do you still get this error?

Instead of -javaagent:H:\elastic-apm-agent-1.1.0.jar, could you try -javaagent:H:/elastic-apm-agent-1.1.0.jar ?

16

Hi felixbarny,
I got everything working but, there is nothing showing in the APM services.

2018-12-18 15:18:46.335 [apm-server-healthcheck] DEBUG co.elastic.apm.report.ApmServerHealthChecker - Starting healthcheck to http://127.0.0.1:8200/
2018-12-18 15:18:46.353 [apm-server-healthcheck] INFO co.elastic.apm.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-11-29T22:56:26Z","build_sha":"ea0bf4863f638b35ecb6ced94f02015bab8bcb47","version":"6.5.2"}}
2018-12-18 15:18:46.380 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.1.0 as standalonetest on Java 1.7.0_80 (Oracle Corporation) Windows 8.1 6.3

Thanks.

17

Eclipse showed a \ after the .jar
when I removed it things started working more reliably

18

The UI only shows services which have sent transactions in the specified timeframe. Judging from your logs, the application has not sent any.

See the troubleshooting guide on how the logs are supposed to look like when transactions are sent.

19

Thanks for your reply felixbarny, but will this work only with web projects in java?, I'm querying from DB but it still does not show up anything, but if used in a web project I can see the metrics. If there is a sample for standalone java project(not web based) it would be grateful. Thanks a lot

20

You can use our public api to manually create a transaction. The auto instrumentation will then add DB spans to the transaction.