I have an Elasticsearch cluster (lets call this the data cluster) setup on AWS EC2 servers. I log all calls made to that ES cluster into another AWS ES cluster (lets call this the logging cluster) for tracking/analysis purposes using logstash.
I am currently using the ES log4j socketappender as the mechanism to send logs entries from the ES data cluster to a central logstash instance, which then saves those logs to the ES logging cluster.
- the log4j socketappender only supports IP addresses for the remoteHost field, not DNS names. That is a problem on AWS as IP addresses can and do change (ie when logstash gets updated, or when we need multiple logstash instances behind an ELB)
- it doesnt allow for multiple logstash servers to deal with high loads or with logstash server failure
note that I am using centos 7 EC2 instances, running ES 1.7.2 and logstash 1.5
Is there a way to configure the socketappender for DNS names instead of IP addresses?
Does anyone have any recommendations re how to do this sort of logging on AWS?