APM Java agent not passing data from the "service"

[mihai2018 (stoian)]

Hello.

I've written several small Java TCs (DB test, HTTP test, etc), in plain Java (no framework below), everything seems fine, as far as configuration is concerned (APM confirms every item, including successful data transmission by the agent). However all I get in the server log is the following:

2018-10-19T16:45:19.797+0300 INFO [request] beater/handlers.go:282 handled request {"request_id": "e733913f-46fb-44cb-bdc1-79403b4e7900", "method": "GET", "URL": "/healthcheck", "content_length": 0, "remote_address": "192.168.56.101", "user-agent": "co/elastic/apm/shaded/okhttp/3.9.1", "response_code": 200}

And no data displaying in Kibana.
Can you share a thought, especially regarding the "content_length":0 thing ?

I've also activated DEBUG, below a sample output from one of the runs:
2018-10-19 16:19:37.222 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 0.7.0 as getpost on Java 1.8.0_141 (Oracle Corporation) Linux 4.1.12-112.14.2.el7uek.x86_64
2018-10-19 16:19:37.222 [main] DEBUG co.elastic.apm.configuration.StartupInfo - server_urls: 'http://elk-node:8200' (source: Java System Properties)
2018-10-19 16:19:37.223 [main] DEBUG co.elastic.apm.configuration.StartupInfo - application_packages: '-1' (source: Java System Properties)
2018-10-19 16:19:37.223 [main] DEBUG co.elastic.apm.configuration.StartupInfo - log_level: 'TRACE' (source: Java System Properties)
...
2018-10-19 16:19:37.274 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.ElasticApmApiInstrumentation$CaptureExceptionInstrumentation
2018-10-19 16:19:37.274 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.TransactionInstrumentation$SetNameInstrumentation
2018-10-19 16:19:37.275 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.TransactionInstrumentation$SetTypeInstrumentation
2018-10-19 16:19:37.275 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.TransactionInstrumentation$AddTagInstrumentation
2018-10-19 16:19:37.276 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.TransactionInstrumentation$SetUserInstrumentation
...

• Applying advice co.elastic.apm.plugin.api.SpanInstrumentation$DoCreateSpanInstrumentation
  2018-10-19 16:19:37.280 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.SpanInstrumentation$EndInstrumentation
  2018-10-19 16:19:37.280 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.SpanInstrumentation$CaptureExceptionInstrumentation
  2018-10-19 16:19:37.280 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.SpanInstrumentation$GetIdInstrumentation
  2018-10-19 16:19:37.281 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.SpanInstrumentation$AddTagInstrumentation
  2018-10-19 16:19:37.281 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.CaptureExceptionInstrumentation
  2018-10-19 16:19:37.282 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.jdbc.ConnectionInstrumentation
  2018-10-19 16:19:37.282 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.jdbc.StatementInstrumentation
  ...
  2018-10-19 16:19:37.286 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.opentracing.impl.ApmSpanInstrumentation$BaggageItemsInstrumentation
  2018-10-19 16:19:37.287 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.opentracing.impl.ApmSpanBuilderInstrumentation$CreateSpanInstrumentation
  ...
  2018-10-19 16:19:37.298 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.servlet.AsyncInstrumentation
  2018-10-19 16:19:37.299 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.resttemplate.SpringRestTemplateInstrumentation
  2018-10-19 16:19:37.299 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.spring.webmvc.SpringTransactionNameInstrumentation
  2018-10-19 16:19:37.300 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applied 37 advices
  JSON String Result { "userId": 4, "id": 32, "title": "doloremque illum aliquid sunt", "body": "deserunt eos nobis asperiores et hic\nest debitis repellat molestiae optio\nnihil ratione ut eos beatae quibusdam distinctio maiores\nearum voluptates et aut adipisci ea maiores voluptas maxime"}
  {
  "userId": 101,
  "id": 101,
  "title": "Test Title",
  "body": "Test Body"
  }
  2018-10-19 16:19:38.218 [apm-reporter] TRACE co.elastic.apm.report.IntakeV1ReportingEventHandler - Receiving FLUSH event (sequence 0)
  2018-10-19 16:19:38.218 [apm-reporter] TRACE co.elastic.apm.report.IntakeV1ReportingEventHandler - Finished processing FLUSH event (sequence 0)
  POST Response Code : 201
  POST Response Message : Created
  { "userId": 101, "id": 101, "title": "Test Title", "body": "Test Body"}

Thx really.
Mihai

[Eyal_Koren]

Hi Mihai,

Thanks a lot for trying out the Elastic APM Java agent.
The log entry with content_length:0 is not related to data sending, it is related to the healthcheck.
Your DB queries and HTTP calls may be recorded by the agent as Spans, but in order for those to be sent, stored and viewed, they need to be part of Transactions. The Java agent would create Transactions out of the box when using one of the supported frameworks, but since you are not relying on such, you would need to programatically create them, meaning starting and ending them from your code. Read through the documentation how to setup the dependency and using the API.

I hope you would find it easy to setup.
Eyal.

[mihai2018 (stoian)]

Hi Eyal,

And thanks for the reply.

However I have 1 remark and 1 question:

Remark:

With all due respect for this interesting product, the documentation for "using the API" is really bad. And I really mean it.

Question:

I put together the different "hints" on your site and assembled the following simple code:

package apmfirst;

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Transaction;
import co.elastic.apm.api.Span;

public class apmfirst {
public static void main(String[] args) {

      Transaction transaction = ElasticApm.startTransaction();
      System.out.println("ID of current transaction: "+ transaction.getId());
      System.out.println("ID length of current transaction: "+ transaction.getId().length());

      try {
          transaction.setName("First transaction name");
          transaction.setType("First transaction test");
         
          // create some span
          Span span = transaction.createSpan();
          System.out.println("ID of current span: "+ span.getId());
          System.out.println("ID length of current span: "+ span.getId().length());

          try {
              span.setName("Second span's name");
              span.setType("Second span's type");
          } catch (Exception e) {
              span.captureException(e);
              throw e;
          } finally {
              span.end();
          }
         
      } catch (Exception e) {
          transaction.captureException(e);
          throw e;
      } finally {
          transaction.end();
      }
 }

}

Running it in my Eclipse IDE I get the output below:

ID of current transaction:
ID length of current transaction: 0
ID of current span:
ID length of current span: 0

So, seemingly no transaction and no span are created (or at least they've got no IDs ...).

Can you please comment on these ?

PS. As a very minimum, to my mind, a very simple "Hello APM !", instrumented with a custom transaction/span, should have been on your site.

Thanks for your time,

Mihai

[Eyal_Koren]

Mihai,

Thanks for your input about the documentation, we will take your feedback into account.

Regarding your test code - seems to be correct, so maybe the documentation is not so bad after all . Did you also start this app with the agent itself? Because the jar you added to your code contains only the API, but the actual execution requires a full agent to be installed on the JVM.

So, just to clarify- as long as you are using supported technologies, the spans should be automatically captured by the agent, however, we cannot show them without relating them to a transaction, so what you will have to do is only the part of starting and ending transactions where it makes sense in your code.

I hope this helps.
Eyal.