X-Pack monitoring caches DNS indefinitely?

Elastic Stack Elasticsearch
1

I've encountered the same problem as described in https://discuss.elastic.co/t/consistantly-losing-monitoring-data-for-clusters/92512: I have a separate monitoring cluster, and sometimes my cluster stops sending data to it (or, in one case, actually posted the data to an unrelated cluster).

I believe this is because the monitoring cluster's IP address changes and X-Pack doesn't honour the DNS TTL. Running tcpdump on one of my cluster hosts has shown that the DNS name for the monitoring cluster is not being regularly looked up, although its TTL is 60 seconds.

Following the advice to set networkaddress.cache.ttl doesn't seem to have made a difference for me. Has this worked for others? i.e. can anyone confirm whether X-Pack's HTTP exporter tries to look up the DNS name before each request or whether it only looks it up on initialisation? And if this only happens on initialisation, is there a suggested solution for communicating with a monitoring cluster that may change IP addresses.

(Edit: I'm running Elasticsearch from the docker.elastic.co/elasticsearch/elasticsearch:6.1.1 Docker image, which includes X-Pack.)

2

Hi, can you provide the lines of dockerfile configuration where networkaddress.cache.ttl is set?

The java.security file that needs to have the setting is in the Java home in the container.

The important thing to clarify is whether or not the setting is applied in the system Java security policy, as opposed to a system property or anything like that. It must be done at the system level.

3

My Dockerfile contains this line:

COPY log4j2.properties java.security config/

Then when I run the container, I set

ES_JAVA_OPTS: "-Djava.security.properties=/usr/share/elasticsearch/config/java.security -Xms4g -Xmx4g"

4

and the java.security file that gets added into the image contains networkaddress.cache.ttl=1.

5

Hi, sorry for the delay in response.

I asked an Elasticsearch engineer to take a look at this topic, and neither of us see anything wrong with your setup. We are going to dig further into this to see if there is any issue with the monitoring HTTP exporter itself having an issue.

6

Hi,

I think that networkaddress.cache.ttl is working correctly and that I misunderstood what was happening when I attempted to monitor it with tcpdump the first time.

I'll post again if I have further problems, but it may be that no further action is needed.

Thanks for your help.

7

Thank you for the followup!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.