Best way to log ES calls in a cloud environment

Elastic Stack Elasticsearch
1

hi all.

I have an Elasticsearch cluster (lets call this the data cluster) setup on AWS EC2 servers. I log all calls made to that ES cluster into another AWS ES cluster (lets call this the logging cluster) for tracking/analysis purposes using logstash.

I am currently using the ES log4j socketappender as the mechanism to send logs entries from the ES data cluster to a central logstash instance, which then saves those logs to the ES logging cluster.

However:

  1. the log4j socketappender only supports IP addresses for the remoteHost field, not DNS names. That is a problem on AWS as IP addresses can and do change (ie when logstash gets updated, or when we need multiple logstash instances behind an ELB)
  2. it doesnt allow for multiple logstash servers to deal with high loads or with logstash server failure

note that I am using centos 7 EC2 instances, running ES 1.7.2 and logstash 1.5

Is there a way to configure the socketappender for DNS names instead of IP addresses?
Does anyone have any recommendations re how to do this sort of logging on AWS?

Thanks

2

I found the syslog appender not to be reliable with network glitches. If
logging stalls due to issues with the network, then the search thread never
finishes, potentially exhausting the search thread pool. I switched to
default logging to a file and having a logstash client tailing that file.

Ivan

3

Thanks Ivan.

Are you are logging to local files on the EC2 instance(s) or to S3 or ?

if local, how are you dealing with autoscaling issues (which we need to consider)? ie how would logstash know where to look for those files if servers are added/removed dynamically as load changes?

if S3, are you finding any speed issues in doing so? reason I ask, is that we are looking at anywhere from 6000 to 10000 log entries per second, and wonder if S3 would be able to keep up.

4

I did not use AWS. The default log4j file output does log rotation IIRC, so
Logstash just needs to keep track of one file. You would need to create all
these rules before creating an AMI I guess.

Ivan