While shipping the server logs through filebeat to elasticsearch via logstash , most of the entries are missing

[Alok]

We are trying to ship logs through filebeat to elasticsearch but unfortunately all the logs are getting ingested to Elasticsearch. We have tried both ways - (a)Filebeat to Elastic search directly and (b) From Filebeat to Logstash to Elasticsearch.
The log size is 150 GB and some 70 million logs were shipped but at elasticsearch we are seeing only 30 Million in case (a) and 0.2 million in case (b)

Versions : Elasticsearch 6.4
filebeat 6.6 and Logstash 6.2.4

Filebeat.yml

filebeat.prospectors:

• type: log

  enabled: true

  • /var/Syslog22/logs/2019.08.20/IP.log

harvester_limit: 0

input_type: log
document_type: log

spool_size: 30720

fields:

setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "IP:5601"
output.logstash.hosts: ["IP:5044"]

Logstash config file
input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => "http://IP:9200"
manage_template => false
index => "filebeatdd-%{+YYYY-MM-dd}"

}
stdout {codec=>rubydebug}
}

We have kept hep size to 32 GB on both elastiocsearch and logstash config/jvm.options
as the memory size of the server is 64GB

Please let me know if we have missed any settings somewhere.

Thanks in advance.

[faec (Fae Charlton)]

Hi,

Sorry for the trouble. Could you repost your configuration in a quote block (begin / end with ``` on a line) to preserve formatting? It's hard to tell where some of the parameters are being applied (e.g. I'm not sure what the loose fields: line applies to, if anything). Thanks!

[Alok]

Good Morning !

I am copy pasting the relevant portion from Filebeat.yml

filebeat.prospectors:
- type: log
  enabled: true
  paths:
     - /var/Syslog22/logs/2019.08.26/<IP>.log
  input_type: log
  document_type: log
 # spool_size: 30720
 # Optional additional fields. These fields can be freely picked
 # to add additional information to the crawled log files for filtering
 #fields:
  fields:
    level: debug
  ignore_older: 24h
  scan_frequency: 1500s
  backoff: 1s

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: true
#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1


#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "<IP>:5601"
#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
 hosts: ["IP:9200"]

 processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

With current confguration , we are now getting 341 million lines out of 343

The server is of decent configuration ( 16 vCPU and 64 GB Mem)

Regards
Alok Pathak