I have the requirement of collecting logs and then sending some (based on source log file path name) to one elasticsearch server and the rest to another one.
In order to accomplish this, I'm using Filebeat to send to a Logstash server. The Logstash server is collecting, filtering (grok, tagging, etc) and outputting logs to the two Elasticsearch servers.
On Logstash, in the filtering stage, I'm tagging certain logs that need to go to the first ES server and then using an if statement in the output to send tagged logs one way and non-tagged ones the other way.
The primary (untagged) ES server is on the same network as the Logstash server, as is a
test ES server. When I'm sending logs to these two servers, everything stays in sync and up to date.
The real second (tagged) ES server is accessed over the internet. When I change the output in Logstash from the
test ES server to the
real one over the internet, the logs start getting very delayed and never catch up.
First of all, if I'm doing this a dumb way, let me know a better way.
Secondly, it seems to me that the problem is the
real ES server across the internet, but the owners of that server say they have the resources to handle what I'm sending. Is there anything on my end that could be contributing to this problem?
I wondered if the problem was related to changing the ES server in the output. i.e. is Logstash trying to "catch up" the
real ES server when switching away from the
test one? Or does Logstash just send logs to the output based on the time the output was changed?
I have XPack monitoring working, so below are graphs that look relevant to me. If there are better ones that would help, let me know. The large jump in the graphs at 15:19 are when I changed the output from the
real tagged ES servers.