Trouble "reconnecting" APM after upgrade to 6.6.0

[bigdamhero (Patrick)]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

Kibana version: 6.6.0

Elasticsearch version: 6.6.0

APM Server version: 6.6.0

APM Agent language and version:

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version? upgrade from ver 6.5.4

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Install is on my TEST environment (lowest environment) No security single Elastic Node and APM on a separate machine.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

service apm-server status shows :

apm-server.service - Elastic APM Server
Loaded: loaded (/lib/systemd/system/apm-server.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2019-02-08 09:41:35 CST; 44min ago
Docs: https://www.elastic.co/solutions/apm
Main PID: 94215 (apm-server)
Tasks: 10
Memory: 9.0M
CPU: 609ms
CGroup: /system.slice/apm-server.service
└─94215 /usr/share/apm-server/bin/apm-server -c /etc/apm-server/apm-server.yml -path.home /usr/share/apm-server -path.config /etc/apm-server -path.data /var/lib/apm-server -path.logs /

when I go into setup to check server status it must see the indexes from the past and says its connected but then it is not showing any new indexes.

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
I only saw a few things in the syslog and I cleaned up those issues but still no connection. Is there a different logging area? is there a way to turn up the logging to see more details?
As always thank you for any assistance.

[bigdamhero (Patrick)]

I found the settings for turning up logging levels.

2019-02-08T11:00:50.838-0600    DEBUG   [monitoring]    elasticsearch/client.go:49      
Monitoring client: connect.
2019-02-08T11:00:50.839-0600    DEBUG   [elasticsearch] elasticsearch/client.go:739     GET 
https://dhpxtlmd4:9300/_xpack?filter_path=features.monitoring.enabled  <nil>
2019-02-08T11:00:50.855-0600    DEBUG   [monitoring]    elasticsearch/elasticsearch.go:237      
Monitoring could not connect to elasticsearch, failed with X-Pack capabilities query failed with: 
Get https://dhpxtlmd4:9300/_xpack?filter_path=features.monitoring.enabled: net/http: 
HTTP/1.x transport connection broken: malformed HTTP status code "is"
2019-02-08T11:00:52.851-0600    ERROR   pipeline/output.go:100  Failed to connect to 
backoff(elasticsearch(https://dhpxtlmd4:9300)): Get https://dhpxtlmd4:9300: net/http: 
HTTP/1.x transport connection broken: malformed HTTP status code "is"
2019-02-08T11:00:52.851-0600    INFO    pipeline/output.go:93   Attempting to reconnect to 
backoff(elasticsearch(https://dhpxtlmd4:9300)) with 5 reconnect attempt(s)
2019-02-08T11:00:52.851-0600    DEBUG   [elasticsearch] elasticsearch/client.go:697     ES 
Ping(url=https://dhpxtlmd4:9300)
2019-02-08T11:00:52.869-0600    DEBUG   [elasticsearch] elasticsearch/client.go:701     Ping 
request failed with: Get https://dhpxtlmd4:9300: net/http: HTTP/1.x transport connection 
broken: malformed HTTP status code "is"
2019-02-08T11:01:20.855-0600    DEBUG   [monitoring]    elasticsearch/client.go:49      
Monitoring client: connect.
2019-02-08T11:01:20.856-0600    DEBUG   [elasticsearch] elasticsearch/client.go:739     GET 
https://dhpxtlmd4:9300/_xpack?filter_path=features.monitoring.enabled  <nil>
2019-02-08T11:01:20.868-0600    DEBUG   [monitoring]    elasticsearch/elasticsearch.go:237      
Monitoring could not connect to elasticsearch, failed with X-Pack capabilities query failed with: 
Get https://dhpxtlmd4:9300/_xpack?filter_path=features.monitoring.enabled: net/http: 
HTTP/1.x transport connection broken: malformed HTTP status code "is"
2019-02-08T11:01:24.869-0600    ERROR   pipeline/output.go:100  Failed to connect to 
backoff(elasticsearch(https://dhpxtlmd4:9300)): Get https://dhpxtlmd4:9300: net/http: 
HTTP/1.x transport connection broken: malformed HTTP status code "is"
2019-02-08T11:01:24.869-0600    INFO    pipeline/output.go:93   Attempting to reconnect to 
backoff(elasticsearch(https://dhpxtlmd4:9300)) with 6 reconnect attempt(s)
2019-02-08T11:01:24.869-0600    DEBUG   [elasticsearch] elasticsearch/client.go:697     ES 
Ping(url=https://dhpxtlmd4:9300)
2019-02-08T11:01:24.882-0600    DEBUG   [elasticsearch] elasticsearch/client.go:701     Ping 
request failed with: Get https://dhpxtlmd4:9300: net/http: HTTP/1.x transport connection 
broken: malformed HTTP status code "is"
2019-02-08T11:01:50.868-0600    DEBUG   [monitoring]    elasticsearch/client.go:49      
Monitoring client: connect.
2019-02-08T11:01:50.868-0600    DEBUG   [elasticsearch] elasticsearch/client.go:739     GET 
https://dhpxtlmd4:9300/_xpack?filter_path=features.monitoring.enabled  <nil>
2019-02-08T11:01:50.882-0600    DEBUG   [monitoring]    elasticsearch/elasticsearch.go:237      
Monitoring could not connect to elasticsearch, failed with X-Pack capabilities query failed with: 
Get https://dhpxtlmd4:9300/_xpack?filter_path=features.monitoring.enabled: net/http: 
HTTP/1.x transport connection broken: malformed HTTP status code "is"

[gil (Gil Raphaelli)]

@bigdamhero This is suspicious as port 9300 is typically used by elastisearch for internal communication, while 9200 is the default for queries:

2019-02-08T11:00:52.851-0600    ERROR   pipeline/output.go:100  Failed to connect to 
backoff(elasticsearch(https://dhpxtlmd4:9300)): Get https://dhpxtlmd4:9300: net/http: 

Can you double check that configuration?

[bigdamhero (Patrick)]

You are correct I was trying to see if that would throw a different error. Another question, I was pointing this to a coordinating node and I needed to change that to one of my master/data nodes. Does this sound correct?

[bigdamhero (Patrick)]

I changed the port to :9200 and I change the FQDN of the server back to the host name and its showing up in monitoring now. Thank you !!