Java Apm agent for multiple tomcats

[Chandan1144 (Chandan)]

How to configure multiple tomcats running on same machine to Apm server ? With single tomcat I m able to start APMUI. How to achieve for multiple tomcats

[Eyal_Koren]

Hi and thanks for trying out our agent!
We really need more details - generally there should be no problem configuring multiple Tomcat instances with an agent sending data to the same APM server. Did you try and didn't work?

[Chandan1144 (Chandan)]

Thanks for the reply.
Yes I tried still no luck.
Here are steps I followed-

1. Installed apm server.
2. Added Java agent to one tomcat.
3. APM UI showed that tomact metrics.
4. For other tomcat added Java agent flag. Checked on UI but couldn't find the newly added tomcat..

[Eyal_Koren]

OK, thanks.
The more info you can provide, the easier it can be to try and figure out what the problem is. For example:

• Are the tomcats working OK? Logs are clear and apps are functioning as expected?
• Please provide the settings you used for you agents. Did you give a service name for each? Was it the same?
• Can you see in the second tomcat logs whether it succeeds communicating with the server?
• If you only run the second tomcat- what happens?
• Anything in APM server logs?
• Anything in Kibana logs?

Thanks

[Chandan1144 (Chandan)]

Below are the answers -

1. Yes Tomcats are working well . functional

2. Here are my settings - -javaagent:/apps/elastic-apm-agent-1.2.0.jar -Delastic.apm.service_name=3dspaceCAS -Delastic.apm.application_packages=org.example,org.another.example -Delastic.apm.server_urls=http://localhost:8200 I have used same settings for both tomcats. AOM server and ELK stack with apm are on same machine.

3. From second tomcat logs -
   [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.2.0 as 3dspaceCAS on Java 1.8.0_152 (Oracle Corporation) Linux 3.10.0-862.14.4.el7.x86_64
   2019-01-01 12:35:20.943 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-12-17T20:15:18Z","build_sha":"fd4f1ac006d8896e4957b6c01fb1e9b9ff04afd7","version":"6.5.4"}

4. If I stop first tomcat and run only second tomcat, I m able to use metric changes on kibana UI .i.e response , requests are getting collected from 2nd tomcat.

5. In APM logs I could see http response codes 202 for all transactions. No error messages.

6. Kibana logs output is mixed up of 304 and 200 http response codes.

May be I m missing some configuration ? First I tried with different service name for both tomcats, it was also unsuccessful.

[Eyal_Koren]

You don't seem to be missing something. Unfortunately I can't reproduce this problem - I started two tomcats (8.5.31) on my machine configured identically to one another (besides the port of course) and I can see data from both agents. I am not sure what you mean by this:

If I stop first tomcat and run only second tomcat, I m able to use metric changes on kibana UI .i.e response , requests are getting collected from 2nd tomcat.

See if you can find two transaction instances from two different tomcats collected when both are reporting to the APM server, something like this view:

If you can't find such, please try to see if you can find data produced by the two agents when working concurrently in the APM Elasticsearch index. The basic query to get all APM traces would be GET apm-*/_search. From that, try to come up with a query that looks for data from both tomcats using a time filter where they worked concurrently.
Other than that, please keep an open eye on the APM server and Kibana logs and see if you find something interesting.

Lastly, regardless to the issue in hand, you would probably want to set elastic.apm.application_packages to the base packages of your application code.

[Chandan1144 (Chandan)]

I tried GET apm-*/_search I could see increase in results whenever I perform actions on any of the tomcats, meaning both tomcats are collecting metrics. As sent in your screenshot I couldn't find transactions duration distribution tab. Apart from these I get following error when operating on 2nd tomcat ( for which I m facing issue).

[Eyal_Koren]

Please try to run a query as such:

GET apm-*/_search
{
  "size": 20,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "context.service.name": "<your-service-name>"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2019-01-03T09:50:00",
              "lte": "2019-01-03T09:55:00",
              "time_zone": "+02:00"
            }
          }
        }
      ],
      "must_not": {
        "term": {
          "processor.event": {
            "value": "metric"
          }
        }
      }
    }
  }
}

Where you should update your service name and the time filter to match where two tomcats are working together and see if there are documents from both or only one.

Regarding the error- seems like the agent captured a problem with your servlet which ended execution with Exception. Looks like an issue with the app, not related to the current issue.