1. Apm server is not creating index in ElasticSearch

[vikassharma123 (vikas)]

1. Apm server is not creating index in ElasticSearch.
2. Apm server logs getting 503 and 202 responses with error Queue is Full from APM agent.
3. Without ElasticSearch index not get APM in Kibana.

command
./apm-server test output

elasticsearch: http://IP
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: IP
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 7.3.0
elasticsearch: http://IP..
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: IP
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 7.3.0

ElasticSearch Index

green open .kibana qcymOwbiR-agbnGCO0ytEg 1 1 4 0 66.3kb 36.4kb
green open metricbeat-7.3.0-2019.08.18-000001 J-IdzmtMSnKMXPiLTG6cPw 1 1 125852 0 98.6mb 33.1mb
green open .monitoring-es-7-mb-2019.08.18 oLvxrPRlTf-IJF1zuWll5w 1 1 72 0 689.5kb 259.5kb
green open .monitoring-beats-7-2019.08.18 whVMQcOXT8C0ZqoD84bCJw 1 1 94 0 612.5kb 307.1kb

[jalvz (Juan Alvarez)]

Hi, can you have you look at the troubleshooting guide, if you didn't already?

https://www.elastic.co/guide/en/apm/server/current/common-problems.html#queue-full

https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html might also give some valuable information.

If that doesn't help, can you paste your apm-server configuration? What version of apm-server are you using? How much memory do your apm-server(s) and elasticsearch instances have?
In Kibana, can you browse to /app/kibana#/home/tutorial/apm?_g=()? What does it say when you click "Check APM Server status" and "Check agent status"?

Juan

[vikassharma123 (vikas)]

@jalvz

What version of apm-server are you using? How much memory do your apm-server(s) and elasticsearch instances

Apm-server:
apm-server-7.3.0-linux-x86_64

How much memory do your apm-server
output.elasticsearch.workers: 5
output.elasticsearch.bulk_max_size: 5120
queue.mem.events: 26500

elasticsearch instances:2

Check APM Server status

APM Server has still not connected to Elasticsearch

Check agent status
No data has been received from agents yet

[jalvz (Juan Alvarez)]

Hello, your "Check APM Server status" button in Kibana says:

@vikassharma123

APM Server has still not connected to Elasticsearch

However, in your first comment the output from ./apm-server test output looks fine.

It looks like you have configured APM Server to point to a different elasticsearch instance than the one you want, could that be the case? Either that or one of your Elasticsearch instances is not connected properly to Kibana. Can you please double check the elasticsearch.hosts setting in your kibana.yml file?

Juan

[vikassharma123 (vikas)]

Hi,

with same elasticsearch cluster metricbeat is working.

metricbeat-7.3.0-2019.08.19-000001

Issue is that suddenly my Apm server and elasticsearch down for while . when elasticsearch and Apm server started by me. i have get 503 response in Apm server logs. and Now I have not find any Apm index in elasticsearch .

even i have reconfigure elasticsearch ,Apm and kibana. but still I have not stop Apm agent.

[vikassharma123 (vikas)]

Do not want to restart Apm agent in Production.

My Apm Agent configuration in tomcat

Using elastic-apm-agent-1.8.0.jar, elasticapm.properties files to tomcat bin and added following to setenv.bat:CATALINA_OPTS="$CATALINA_OPTS -javaagent:/home/tomcat/elastic-apm-agent-1.8.0.jar -Delastic.apm.service_name=P_24_CA -Delastic.apm.server_url=http://IP -Delastic.apm.environment=PR -Delastic.apm.capture_body=all"