How to confirm if my apm-server is collecting the data from apm agent

Hi Team,

I have installed APM-Server and ElasticSearch on machine with default configurations.
APM agent resides on another machine and i have added the agent to JVM startup.

I have tested the APM server config and output and its working fine.

How can i confirm if my apm-server is collecting the data from apm agent. I need to see some metrics from log files. Also, by what name is log file saved ?

Thanks,
Sachin

I suggest installing Kibana and checking the APM App or Discover app.

Otherwise quick way from the same host you might need to substitute the hostname or IP for localhost

curl -X GET "http://localhost:9200/_cat/indices/apm-*?v&s=index"

Note: you might need to substitute the hostname or IP for localhost
Should result in something like

health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   apm-7.2.0-onboarding-2019.06.25  IgIBRTnXRimMufSlUovdOQ   1   1          2            0     12.2kb         12.2kb
yellow open   apm-7.2.0-span-2019.06.25        LJYn7yRaQRaYyJmS3YDzjg   1   1         12            0     92.2kb         92.2kb
yellow open   apm-7.2.0-transaction-2019.06.25 6vHdBxkRTDGx0IBSjHMt_g   1   1         12            0    136.2kb        136.2kb

Thank you Stephen, Installation of Kibana is already in progress and in process of official approvals :slight_smile: . I will check the other solution from you and will update here.

Thanks,
Sachin

Hi Stephen,

I used the curl command and got the below output

I think, this means we can see the output in elasticsearch. Correct me if i am wrong.

Also, thus it means APM agent is working.

Regards,
Sachin

Thanks and first in the future please paste logs / command results as formatted text, images are not searchable and hard to read.

Actually that tells me the apm server connected but I do not see any actual transactions or spans which take the form

yellow open   apm-7.2.0-span-2019.06.25        LJYn7yRaQRaYyJmS3YDzjg   1   1         12            0     92.2kb         92.2kb
yellow open   apm-7.2.0-transaction-2019.06.25 6vHdBxkRTDGx0IBSjHMt_g   1   1         12            0    136.2kb        136.2kb

So at the point it does not look like you are generating / receiving any actual APM transaction or spans from an Elastic APM Agent from your apps you are instrumenting but perhaps you have not got that far.

Thanks Stephen, I will traverse some flows in my application and see if get any transactions like you mentioned above.

So at this point , i reckon

  1. My Apm-server and elasticsearch is working fine
  2. If i dont see any transaction traces then i will check for APM agent configuration

Regards,
Sachin

Yup that would be the next place to look...
What Agent / Language and Application Framework are you working with? Let us know what you find.

I am using elastic-apm-agent-1.5.0 agent and application is based on Spring Hibernate framework.

I hit some transactions yesterday but didn't see any changes in logs. I will revisit the agent configuration and see if agent is working fine with my application.

Thanks,
Sachin

Hi Stephen,

So i checked the logs after restarting my app. This is the output for it

04:29:18,686 INFO  [stdout] (apm-reporter)      at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.startRequest(IntakeV2ReportingEventHandler.java:236)
04:29:18,686 INFO  [stdout] (apm-reporter)      at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:162)
04:29:18,686 INFO  [stdout] (apm-reporter)      at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:140)
04:29:18,686 INFO  [stdout] (apm-reporter)      at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:58)
04:29:18,686 INFO  [stdout] (apm-reporter)      at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
04:29:18,686 INFO  [stdout] (apm-reporter)      at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
04:29:18,686 INFO  [stdout] (apm-reporter)      at java.lang.Thread.run(Thread.java:748)
04:29:18,686 INFO  [stdout] (apm-reporter) 2019-07-19 04:29:18.686 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
04:29:54,825 INFO  [stdout] (apm-reporter) 2019-07-19 04:29:54.825 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICS event (sequence 213)
04:29:54,825 INFO  [stdout] (apm-reporter) 2019-07-19 04:29:54.825 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://server:8200/intake/v2/events
04:29:59,831 INFO  [stdout] (apm-reporter) 2019-07-19 04:29:59.831 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
04:29:59,831 INFO  [stdout] (apm-reporter) 2019-07-19 04:29:59.831 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Event handling failure

and from server.log

00:00:19,718 INFO  [stdout] (apm-reporter) 2019-07-19 00:00:19.718 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:00:19,721 INFO  [stdout] (apm-reporter) 2019-07-19 00:00:19.721 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:00:57,245 INFO  [stdout] (apm-reporter) 2019-07-19 00:00:57.245 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:00:57,245 INFO  [stdout] (apm-reporter) 2019-07-19 00:00:57.245 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:01:40,129 INFO  [stdout] (apm-reporter) 2019-07-19 00:01:40.129 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:01:40,129 INFO  [stdout] (apm-reporter) 2019-07-19 00:01:40.129 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:02:21,618 INFO  [stdout] (apm-reporter) 2019-07-19 00:02:21.618 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:02:21,618 INFO  [stdout] (apm-reporter) 2019-07-19 00:02:21.618 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:03:03,593 INFO  [stdout] (apm-reporter) 2019-07-19 00:03:03.593 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:03:03,593 INFO  [stdout] (apm-reporter) 2019-07-19 00:03:03.593 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:03:48,184 INFO  [stdout] (apm-reporter) 2019-07-19 00:03:48.184 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:03:48,184 INFO  [stdout] (apm-reporter) 2019-07-19 00:03:48.184 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)

These are my startup parameters -

"-javaagent:/opt/user_projects/myapp/configuration/elastic-apm-agent-1.5.0.jar -Delastic.apm.log_level=debug -Delastic.apm.service_name=MagicPerf -Delastic.apm.application_packages=com.sony.spe -Delastic.apm.server_url=http://myserver:8200 -Xms1024m -Xmx1024m -XX:MaxNewSize=448m -XX:NewSize=448m -XX:SurvivorRatio=6 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.server.home.url=file:/opt/user_projects/myapp"

Any idea what could be wrong here ?

Thanks,
Sachin

Hi can you post two more things can you restart the APM server and post the first 50 lines or so from the APM server looks like it is having trouble connecting to elasticsearch ... Also can you post the output.elasticsearch section of the apm-server.yml. Thanks

Please make sure you are using a supported technology
https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html

And also are you properly configuring JBoss
https://www.elastic.co/guide/en/apm/agent/java/current/setup-javaagent.html#setup-jboss-wildfly

Also can you please take a look at this

Hi Stephen,

I have checked the other thread earlier as well, I will check the supported tech* and configuration on JBOSS. Will update here as soon as i get something.

Thanks,
Sachin

Hi Stephen,

  1. For APM server logs - I referred this path /var/log/apm-server and could see multiple log files like this
    apm-server apm-server.1 apm-server.2 apm-server.3 apm-server.4 apm-server.5 apm-server.6 apm-server.7

So i have attached logs for apm-server and apm-server.1

apm-server

2019-07-15T06:40:44.311-0700    INFO    instance/beat.go:571    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2019-07-15T06:40:44.312-0700    INFO    instance/beat.go:579    Beat ID: dccbe5b9-66e2-4da3-b394-f1a7cef2700d
2019-07-15T06:40:44.312-0700    INFO    [index-management.ilm]  ilm/ilm.go:129  Policy name: apm-server-7.0.0
2019-07-15T06:40:44.312-0700    INFO    [index-management]      idxmgmt/std.go:165      Set output.elasticsearch.index to 'apm-server-7.0.0' as ILM is enabled.
2019-07-15T06:40:44.312-0700    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-07-15T06:40:44.452-0700    INFO    elasticsearch/client.go:734     Attempting to connect to Elasticsearch version 7.0.0

apm-server.1

2019-07-15T06:40:44.311-0700    INFO    instance/beat.go:571    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2019-07-15T06:40:44.312-0700    INFO    instance/beat.go:579    Beat ID: dccbe5b9-66e2-4da3-b394-f1a7cef2700d
2019-07-15T06:40:44.312-0700    INFO    [index-management.ilm]  ilm/ilm.go:129  Policy name: apm-server-7.0.0
2019-07-15T06:40:44.312-0700    INFO    [index-management]      idxmgmt/std.go:165      Set output.elasticsearch.index to 'apm-server-7.0.0' as ILM is enabled.
2019-07-15T06:40:44.312-0700    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-07-15T06:40:44.452-0700    INFO    elasticsearch/client.go:734     Attempting to connect to Elasticsearch version 7.0.0
-bash-4.2$ cat apm-server.1
2019-07-15T06:40:37.261-0700    INFO    instance/beat.go:571    Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2019-07-15T06:40:37.262-0700    INFO    instance/beat.go:579    Beat ID: dccbe5b9-66e2-4da3-b394-f1a7cef2700d
2019-07-15T06:40:37.262-0700    INFO    [beat]  instance/beat.go:827    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/apm-server", "data": "/var/lib/apm-server", "home": "/usr/share/apm-server", "logs": "/var/log/apm-server"}, "type": "apm-server", "uuid": "dccbe5b9-66e2-4da3-b394-f1a7cef2700d"}}}
2019-07-15T06:40:37.262-0700    INFO    [beat]  instance/beat.go:836    Build info      {"system_info": {"build": {"commit": "1f7ad059a0bbf0174b90e07c80c0274d152f1155", "libbeat": "7.0.0", "time": "2019-04-05T21:50:06.000Z", "version": "7.0.0"}}}
2019-07-15T06:40:37.262-0700    INFO    [beat]  instance/beat.go:839    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.11.5"}}}
2019-07-15T06:40:37.263-0700    INFO    [beat]  instance/beat.go:843    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-07-03T22:01:40-07:00","containerized":true,"name":"usdl653.spe.sony.com","ip":["127.0.0.1/8","10.192.18.77/23"],"kernel_version":"3.10.0-957.21.3.el7.x86_64","mac":["00:50:56:8c:4e:91"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.6 (Maipo)","major":7,"minor":6,"patch":0,"codename":"Maipo"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"f555496539ee4417a495b39cb0766313"}}}
2019-07-15T06:40:37.263-0700    INFO    [beat]  instance/beat.go:872    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/var/log/apm-server", "exe": "/usr/share/apm-server/bin/apm-server", "name": "apm-server", "pid": 5175, "ppid": 2888, "seccomp": {"mode":"disabled"}, "start_time": "2019-07-15T06:40:36.400-0700"}}}
2019-07-15T06:40:37.263-0700    INFO    instance/beat.go:280    Setup Beat: apm-server; Version: 7.0.0
2019-07-15T06:40:37.264-0700    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-07-15T06:40:37.264-0700    INFO    [publisher]     pipeline/module.go:97   Beat name: myservername
2019-07-15T06:40:37.265-0700    INFO    [beater]        beater/beater.go:118    No pipeline callback registered
  1. Elastic search output

     #-------------------------- Elasticsearch output ------------------------------
     output.elasticsearch:
       # Array of hosts to connect to.
       # Scheme and port can be left out and will be set to the default (http and 9200)
       # In case you specify and additional path, the scheme is required: http://localhost:9200/path
       # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
       hosts: ["localhost:9200"]
    
       # Boolean flag to enable or disable the output module.
       #enabled: true
    
       # Set gzip compression level.
       #compression_level: 0
    
       # Optional protocol and basic auth credentials.
       #protocol: "https"
       #username: "elastic"
       #password: "changeme"
    
       # Dictionary of HTTP parameters to pass within the url with index operations.
       #parameters:
         #param1: value1
         #param2: value2
    
       # Number of workers per Elasticsearch host.
       #worker: 1
    
       # By using the configuration below, apm documents are stored to separate indices,
       # depending on their `processor.event`:
       # - error
       # - transaction
       # - span
       # - sourcemap
       #
       # The indices are all prefixed with `apm-%{[observer.version]}`.
       # To allow managing indices based on their age, all indices (except for sourcemaps)
       # end with the information of the day they got indexed.
       # e.g. "apm-6.3.0-transaction-2018.03.20"
       #
       # Be aware that you can only specify one Elasticsearch template.
       # In case you modify the index patterns you must also update those configurations accordingly,
       # as they need to be aligned:
       # * `setup.template.name`
       # * `setup.template.pattern`
       #index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
       #indices:
       #  - index: "apm-%{[observer.version]}-sourcemap"
       #    when.contains:
       #      processor.event: "sourcemap"
       #
       #  - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
       #    when.contains:
       #      processor.event: "error"
       #
       #  - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
       #    when.contains:
       #      processor.event: "transaction"
       #
       #  - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
       #    when.contains:
       #      processor.event: "span"
       #
       #  - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
       #    when.contains:
       #      processor.event: "metric"
       #
       #  - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
       #    when.contains:
       #      processor.event: "onboarding"
    
       # A pipeline is a definition of processors applied to documents when writing them to Elasticsearch.
       # APM Server comes with a default pipeline definition, located at `ingest/pipeline/definition.json`.
       # Pipelines are disabled by default. To make use of them you have to:
       # (1) ensure pipelines are registered in Elasticsearch, see `apm-server.register.ingest.pipeline`
       # (2) enable the following:
       #pipelines:
       #- pipeline: "apm_user_agent"
    
       # Optional HTTP Path
       #path: "/elasticsearch"
    
       # Custom HTTP headers to add to each request
       #headers:
       #  X-My-Header: Contents of the header
    
       # Proxy server url
       #proxy_url: http://proxy:3128
    
       # The number of times a particular Elasticsearch index operation is attempted. If
       # the indexing operation doesn't succeed after this many retries, the events are
       # dropped. The default is 3.
       #max_retries: 3
    
       # The maximum number of events to bulk in a single Elasticsearch bulk API index request.
       # The default is 50.
       #bulk_max_size: 50
    
       # The number of seconds to wait before trying to reconnect to Elasticsearch
       # after a network error. After waiting backoff.init seconds, apm-server
       # tries to reconnect. If the attempt fails, the backoff timer is increased
       # exponentially up to backoff.max. After a successful connection, the backoff
       # timer is reset. The default is 1s.
       #backoff.init: 1s
    
       # The maximum number of seconds to wait before attempting to connect to
       # Elasticsearch after a network error. The default is 60s.
       #backoff.max: 60s
    
       # Configure http request timeout before failing an request to Elasticsearch.
       #timeout: 90
       
         # Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
       #ssl.enabled: true
    
       # Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.
       # Configure SSL verification mode. If `none` is configured, all server hosts
       # and certificates will be accepted. In this mode, SSL based connections are
       # susceptible to man-in-the-middle attacks. Use only for testing. Default is
       # `full`.
       #ssl.verification_mode: full
    
       # List of supported/valid TLS versions. By default all TLS versions 1.0 up to
       # 1.2 are enabled.
       #ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
    
       # List of root certificates for HTTPS server verifications
       #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
    
       # Certificate for SSL client authentication
       #ssl.certificate: "/etc/pki/client/cert.pem"
    
       # Client Certificate Key
       #ssl.key: "/etc/pki/client/cert.key"
    
       # Optional passphrase for decrypting the Certificate Key.
       # It is recommended to use the provided keystore instead of entering the passphrase in plain text.
       #ssl.key_passphrase: ''
    
       # Configure cipher suites to be used for SSL connections
       #ssl.cipher_suites: []
    
       # Configure curve types for ECDHE based cipher suites
       #ssl.curve_types: []
    
       # Configure what types of renegotiation are supported. Valid options are
       # never, once, and freely. Default is never.
       #ssl.renegotiation: never
    
  2. As per the supported technology info, for us it is supported.

I am curious about the Elastic, Kibana and APM Server versions you are using. The logs suggest you use APM Server 7.0.0, but they also show some ILM configuration, which was only supported from 7.2.0 on in APM Server. It would also be helpful if you could post any custom settings you configured for the APM Server.

Hi,

Thanks for your response.

I didnt do any custom settings for APM server.

Regards,
Sachin

Can you try this sample app and see if it works...

Specifically just try the simple java standalone apps

# Clone Petclinic Repo Java Spring Boot App
git clone https://github.com/bvader/spring-petclinic.git

cd spring-petclinic

# Build
./mvnw package -Dmaven.test.skip=true

# Get the Elastic Java APM agent
curl -O  https://search.maven.org/remotecontent?filepath=co/elastic/apm/elastic-apm-agent/1.7.0/elastic-apm-agent-1.7.0.jar

# Run Petclinc with APM, 
# NOTE: this enables method tracing 
# Unix / Mac OS
java -javaagent:./elastic-apm-agent-1.7.0.jar \
-Delastic.apm.server_urls="http://localhost:8200" \
-Delastic.apm.service_name="spring-petclinic-monolith" \
-Delastic.apm.application_packages="org.springframework.samples" \
-Delastic.apm.trace_methods="org.springframework.samples.petclinic.*" \
-jar target/spring-petclinic-2.1.0.BUILD-SNAPSHOT.jar

Hi Stephen,

Apologies for the delay in response. Due to access issue i wont be able to try the sample application.
Though i figured out some issue with port connectivity between my application and apm server. Firewall for 8200 port is open for communication. The message i received from network team that it is only listening on Locp back. 127.0.0.1, so i am going to update my host to IP address.

I will update once i get more information about it.

Regards,
Sachin