The question I have is that , I have two Logstash Instances, running on same configuration VMs, with filebeat forwarding data to both the logstash instances. But I could find that one of the VM is indexing 22 MB data while the other VM is only indexing 9 MB.
The filebeat configs are as follows:
#=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - input_type: log enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/prod_logs/XS/basatlxs01/*.txt #- c:\programdata\elasticsearch\logs\* # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ["^DBG"] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. #include_lines: ["^ERR", "^WARN"] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #exclude_files: [".gz$"] # Optional additional fields. These field can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 ### Multiline options # Mutiline can be used for log messages spanning multiple lines. This is common # for Java Stack Traces or C-Line Continuation # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ multiline.pattern: ^([0-9]+.[0-9]+.[0-9]+) # Defines if the pattern set under pattern should be negated or not. Default is false. multiline.negate: true # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern # that was (not) matched before or after or as long as a pattern is not matched based on negate. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash multiline.match: after clean_*: true # Files for the modification data is older then clean_inactive the state from the registry is removed # By default this is disabled. clean_inactive: 0 # Removes the state for file which cannot be found on disk anymore immediately clean_removed: true filebeat.spool_size: 2048 #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: [ "184.108.40.206:5044", "220.127.116.11:5044"] loadbalance: true index: filebeat bulk_max_size: 1024 # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key"
As far as Logstash is concerned the filters and configs are same.
Please provide some input on the working of filebeat load balancing so that I can design my node accordingly.