Hi Team,
I have installed APM-Server and ElasticSearch on machine with default configurations.
APM agent resides on another machine and i have added the agent to JVM startup.
I have tested the APM server config and output and its working fine.
How can i confirm if my apm-server is collecting the data from apm agent. I need to see some metrics from log files. Also, by what name is log file saved ?
Thanks,
Sachin
I suggest installing Kibana and checking the APM App or Discover app.
Otherwise quick way from the same host you might need to substitute the hostname or IP for localhost
curl -X GET "http://localhost:9200/_cat/indices/apm-*?v&s=index"
Note: you might need to substitute the hostname or IP for localhost
Should result in something like
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open apm-7.2.0-onboarding-2019.06.25 IgIBRTnXRimMufSlUovdOQ 1 1 2 0 12.2kb 12.2kb
yellow open apm-7.2.0-span-2019.06.25 LJYn7yRaQRaYyJmS3YDzjg 1 1 12 0 92.2kb 92.2kb
yellow open apm-7.2.0-transaction-2019.06.25 6vHdBxkRTDGx0IBSjHMt_g 1 1 12 0 136.2kb 136.2kbThank you Stephen, Installation of Kibana is already in progress and in process of official approvals 
. I will check the other solution from you and will update here.
Thanks,
Sachin
Hi Stephen,
I used the curl command and got the below output
I think, this means we can see the output in elasticsearch. Correct me if i am wrong.
Also, thus it means APM agent is working.
Regards,
Sachin
Thanks and first in the future please paste logs / command results as formatted text, images are not searchable and hard to read.
Actually that tells me the apm server connected but I do not see any actual transactions or spans which take the form
yellow open apm-7.2.0-span-2019.06.25 LJYn7yRaQRaYyJmS3YDzjg 1 1 12 0 92.2kb 92.2kb
yellow open apm-7.2.0-transaction-2019.06.25 6vHdBxkRTDGx0IBSjHMt_g 1 1 12 0 136.2kb 136.2kb
So at the point it does not look like you are generating / receiving any actual APM transaction or spans from an Elastic APM Agent from your apps you are instrumenting but perhaps you have not got that far.
Thanks Stephen, I will traverse some flows in my application and see if get any transactions like you mentioned above.
So at this point , i reckon
Regards,
Sachin
Yup that would be the next place to look...
What Agent / Language and Application Framework are you working with? Let us know what you find.
I am using elastic-apm-agent-1.5.0 agent and application is based on Spring Hibernate framework.
I hit some transactions yesterday but didn't see any changes in logs. I will revisit the agent configuration and see if agent is working fine with my application.
Thanks,
Sachin
Hi Stephen,
So i checked the logs after restarting my app. This is the output for it
04:29:18,686 INFO [stdout] (apm-reporter) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.startRequest(IntakeV2ReportingEventHandler.java:236)
04:29:18,686 INFO [stdout] (apm-reporter) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:162)
04:29:18,686 INFO [stdout] (apm-reporter) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:140)
04:29:18,686 INFO [stdout] (apm-reporter) at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:58)
04:29:18,686 INFO [stdout] (apm-reporter) at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
04:29:18,686 INFO [stdout] (apm-reporter) at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
04:29:18,686 INFO [stdout] (apm-reporter) at java.lang.Thread.run(Thread.java:748)
04:29:18,686 INFO [stdout] (apm-reporter) 2019-07-19 04:29:18.686 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
04:29:54,825 INFO [stdout] (apm-reporter) 2019-07-19 04:29:54.825 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICS event (sequence 213)
04:29:54,825 INFO [stdout] (apm-reporter) 2019-07-19 04:29:54.825 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://server:8200/intake/v2/events
04:29:59,831 INFO [stdout] (apm-reporter) 2019-07-19 04:29:59.831 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
04:29:59,831 INFO [stdout] (apm-reporter) 2019-07-19 04:29:59.831 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Event handling failure
and from server.log
00:00:19,718 INFO [stdout] (apm-reporter) 2019-07-19 00:00:19.718 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:00:19,721 INFO [stdout] (apm-reporter) 2019-07-19 00:00:19.721 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:00:57,245 INFO [stdout] (apm-reporter) 2019-07-19 00:00:57.245 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:00:57,245 INFO [stdout] (apm-reporter) 2019-07-19 00:00:57.245 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:01:40,129 INFO [stdout] (apm-reporter) 2019-07-19 00:01:40.129 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:01:40,129 INFO [stdout] (apm-reporter) 2019-07-19 00:01:40.129 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:02:21,618 INFO [stdout] (apm-reporter) 2019-07-19 00:02:21.618 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:02:21,618 INFO [stdout] (apm-reporter) 2019-07-19 00:02:21.618 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:03:03,593 INFO [stdout] (apm-reporter) 2019-07-19 00:03:03.593 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:03:03,593 INFO [stdout] (apm-reporter) 2019-07-19 00:03:03.593 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
00:03:48,184 INFO [stdout] (apm-reporter) 2019-07-19 00:03:48.184 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: connect timed out
00:03:48,184 INFO [stdout] (apm-reporter) 2019-07-19 00:03:48.184 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (±10%)
These are my startup parameters -
"-javaagent:/opt/user_projects/myapp/configuration/elastic-apm-agent-1.5.0.jar -Delastic.apm.log_level=debug -Delastic.apm.service_name=MagicPerf -Delastic.apm.application_packages=com.sony.spe -Delastic.apm.server_url=http://myserver:8200 -Xms1024m -Xmx1024m -XX:MaxNewSize=448m -XX:NewSize=448m -XX:SurvivorRatio=6 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.server.home.url=file:/opt/user_projects/myapp"
Any idea what could be wrong here ?
Thanks,
Sachin
Hi can you post two more things can you restart the APM server and post the first 50 lines or so from the APM server looks like it is having trouble connecting to elasticsearch ... Also can you post the output.elasticsearch section of the apm-server.yml. Thanks
Please make sure you are using a supported technology
And also are you properly configuring JBoss
Also can you please take a look at this
Hi Stephen,
I have checked the other thread earlier as well, I will check the supported tech* and configuration on JBOSS. Will update here as soon as i get something.
Thanks,
Sachin
Hi Stephen,
apm-server apm-server.1 apm-server.2 apm-server.3 apm-server.4 apm-server.5 apm-server.6 apm-server.7
So i have attached logs for apm-server and apm-server.1
apm-server
2019-07-15T06:40:44.311-0700 INFO instance/beat.go:571 Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2019-07-15T06:40:44.312-0700 INFO instance/beat.go:579 Beat ID: dccbe5b9-66e2-4da3-b394-f1a7cef2700d
2019-07-15T06:40:44.312-0700 INFO [index-management.ilm] ilm/ilm.go:129 Policy name: apm-server-7.0.0
2019-07-15T06:40:44.312-0700 INFO [index-management] idxmgmt/std.go:165 Set output.elasticsearch.index to 'apm-server-7.0.0' as ILM is enabled.
2019-07-15T06:40:44.312-0700 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200
2019-07-15T06:40:44.452-0700 INFO elasticsearch/client.go:734 Attempting to connect to Elasticsearch version 7.0.0
apm-server.1
2019-07-15T06:40:44.311-0700 INFO instance/beat.go:571 Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2019-07-15T06:40:44.312-0700 INFO instance/beat.go:579 Beat ID: dccbe5b9-66e2-4da3-b394-f1a7cef2700d
2019-07-15T06:40:44.312-0700 INFO [index-management.ilm] ilm/ilm.go:129 Policy name: apm-server-7.0.0
2019-07-15T06:40:44.312-0700 INFO [index-management] idxmgmt/std.go:165 Set output.elasticsearch.index to 'apm-server-7.0.0' as ILM is enabled.
2019-07-15T06:40:44.312-0700 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200
2019-07-15T06:40:44.452-0700 INFO elasticsearch/client.go:734 Attempting to connect to Elasticsearch version 7.0.0
-bash-4.2$ cat apm-server.1
2019-07-15T06:40:37.261-0700 INFO instance/beat.go:571 Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2019-07-15T06:40:37.262-0700 INFO instance/beat.go:579 Beat ID: dccbe5b9-66e2-4da3-b394-f1a7cef2700d
2019-07-15T06:40:37.262-0700 INFO [beat] instance/beat.go:827 Beat info {"system_info": {"beat": {"path": {"config": "/etc/apm-server", "data": "/var/lib/apm-server", "home": "/usr/share/apm-server", "logs": "/var/log/apm-server"}, "type": "apm-server", "uuid": "dccbe5b9-66e2-4da3-b394-f1a7cef2700d"}}}
2019-07-15T06:40:37.262-0700 INFO [beat] instance/beat.go:836 Build info {"system_info": {"build": {"commit": "1f7ad059a0bbf0174b90e07c80c0274d152f1155", "libbeat": "7.0.0", "time": "2019-04-05T21:50:06.000Z", "version": "7.0.0"}}}
2019-07-15T06:40:37.262-0700 INFO [beat] instance/beat.go:839 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.11.5"}}}
2019-07-15T06:40:37.263-0700 INFO [beat] instance/beat.go:843 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-07-03T22:01:40-07:00","containerized":true,"name":"usdl653.spe.sony.com","ip":["127.0.0.1/8","10.192.18.77/23"],"kernel_version":"3.10.0-957.21.3.el7.x86_64","mac":["00:50:56:8c:4e:91"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.6 (Maipo)","major":7,"minor":6,"patch":0,"codename":"Maipo"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"f555496539ee4417a495b39cb0766313"}}}
2019-07-15T06:40:37.263-0700 INFO [beat] instance/beat.go:872 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/var/log/apm-server", "exe": "/usr/share/apm-server/bin/apm-server", "name": "apm-server", "pid": 5175, "ppid": 2888, "seccomp": {"mode":"disabled"}, "start_time": "2019-07-15T06:40:36.400-0700"}}}
2019-07-15T06:40:37.263-0700 INFO instance/beat.go:280 Setup Beat: apm-server; Version: 7.0.0
2019-07-15T06:40:37.264-0700 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200
2019-07-15T06:40:37.264-0700 INFO [publisher] pipeline/module.go:97 Beat name: myservername
2019-07-15T06:40:37.265-0700 INFO [beater] beater/beater.go:118 No pipeline callback registeredElastic search output
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
# Scheme and port can be left out and will be set to the default (http and 9200)
# In case you specify and additional path, the scheme is required: http://localhost:9200/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["localhost:9200"]
# Boolean flag to enable or disable the output module.
#enabled: true
# Set gzip compression level.
#compression_level: 0
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
# Dictionary of HTTP parameters to pass within the url with index operations.
#parameters:
#param1: value1
#param2: value2
# Number of workers per Elasticsearch host.
#worker: 1
# By using the configuration below, apm documents are stored to separate indices,
# depending on their `processor.event`:
# - error
# - transaction
# - span
# - sourcemap
#
# The indices are all prefixed with `apm-%{[observer.version]}`.
# To allow managing indices based on their age, all indices (except for sourcemaps)
# end with the information of the day they got indexed.
# e.g. "apm-6.3.0-transaction-2018.03.20"
#
# Be aware that you can only specify one Elasticsearch template.
# In case you modify the index patterns you must also update those configurations accordingly,
# as they need to be aligned:
# * `setup.template.name`
# * `setup.template.pattern`
#index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
#indices:
# - index: "apm-%{[observer.version]}-sourcemap"
# when.contains:
# processor.event: "sourcemap"
#
# - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "error"
#
# - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "transaction"
#
# - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "span"
#
# - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "metric"
#
# - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
# when.contains:
# processor.event: "onboarding"
# A pipeline is a definition of processors applied to documents when writing them to Elasticsearch.
# APM Server comes with a default pipeline definition, located at `ingest/pipeline/definition.json`.
# Pipelines are disabled by default. To make use of them you have to:
# (1) ensure pipelines are registered in Elasticsearch, see `apm-server.register.ingest.pipeline`
# (2) enable the following:
#pipelines:
#- pipeline: "apm_user_agent"
# Optional HTTP Path
#path: "/elasticsearch"
# Custom HTTP headers to add to each request
#headers:
# X-My-Header: Contents of the header
# Proxy server url
#proxy_url: http://proxy:3128
# The number of times a particular Elasticsearch index operation is attempted. If
# the indexing operation doesn't succeed after this many retries, the events are
# dropped. The default is 3.
#max_retries: 3
# The maximum number of events to bulk in a single Elasticsearch bulk API index request.
# The default is 50.
#bulk_max_size: 50
# The number of seconds to wait before trying to reconnect to Elasticsearch
# after a network error. After waiting backoff.init seconds, apm-server
# tries to reconnect. If the attempt fails, the backoff timer is increased
# exponentially up to backoff.max. After a successful connection, the backoff
# timer is reset. The default is 1s.
#backoff.init: 1s
# The maximum number of seconds to wait before attempting to connect to
# Elasticsearch after a network error. The default is 60s.
#backoff.max: 60s
# Configure http request timeout before failing an request to Elasticsearch.
#timeout: 90
# Enable custom SSL settings. Set to false to ignore custom SSL settings for secure communication.
#ssl.enabled: true
# Optional SSL configuration options. SSL is off by default, change the `protocol` option if you want to enable `https`.
# Configure SSL verification mode. If `none` is configured, all server hosts
# and certificates will be accepted. In this mode, SSL based connections are
# susceptible to man-in-the-middle attacks. Use only for testing. Default is
# `full`.
#ssl.verification_mode: full
# List of supported/valid TLS versions. By default all TLS versions 1.0 up to
# 1.2 are enabled.
#ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# Optional passphrase for decrypting the Certificate Key.
# It is recommended to use the provided keystore instead of entering the passphrase in plain text.
#ssl.key_passphrase: ''
# Configure cipher suites to be used for SSL connections
#ssl.cipher_suites: []
# Configure curve types for ECDHE based cipher suites
#ssl.curve_types: []
# Configure what types of renegotiation are supported. Valid options are
# never, once, and freely. Default is never.
#ssl.renegotiation: never
As per the supported technology info, for us it is supported.
I am curious about the Elastic, Kibana and APM Server versions you are using. The logs suggest you use APM Server 7.0.0, but they also show some ILM configuration, which was only supported from 7.2.0 on in APM Server. It would also be helpful if you could post any custom settings you configured for the APM Server.
Hi,
Thanks for your response.
I didnt do any custom settings for APM server.
Regards,
Sachin
Can you try this sample app and see if it works...
Specifically just try the simple java standalone apps
# Clone Petclinic Repo Java Spring Boot App
git clone https://github.com/bvader/spring-petclinic.git
cd spring-petclinic
# Build
./mvnw package -Dmaven.test.skip=true
# Get the Elastic Java APM agent
curl -O https://search.maven.org/remotecontent?filepath=co/elastic/apm/elastic-apm-agent/1.7.0/elastic-apm-agent-1.7.0.jar
# Run Petclinc with APM,
# NOTE: this enables method tracing
# Unix / Mac OS
java -javaagent:./elastic-apm-agent-1.7.0.jar \
-Delastic.apm.server_urls="http://localhost:8200" \
-Delastic.apm.service_name="spring-petclinic-monolith" \
-Delastic.apm.application_packages="org.springframework.samples" \
-Delastic.apm.trace_methods="org.springframework.samples.petclinic.*" \
-jar target/spring-petclinic-2.1.0.BUILD-SNAPSHOT.jarHi Stephen,
Apologies for the delay in response. Due to access issue i wont be able to try the sample application.
Though i figured out some issue with port connectivity between my application and apm server. Firewall for 8200 port is open for communication. The message i received from network team that it is only listening on Locp back. 127.0.0.1, so i am going to update my host to IP address.
I will update once i get more information about it.
Regards,
Sachin
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.