We are trying to ship logs through filebeat to elasticsearch but unfortunately all the logs are getting ingested to Elasticsearch. We have tried both ways - (a)Filebeat to Elastic search directly and (b) From Filebeat to Logstash to Elasticsearch.
The log size is 150 GB and some 70 million logs were shipped but at elasticsearch we are seeing only 30 Million in case (a) and 0.2 million in case (b)
Versions : Elasticsearch 6.4
filebeat 6.6 and Logstash 6.2.4
Sorry for the trouble. Could you repost your configuration in a quote block (begin / end with ``` on a line) to preserve formatting? It's hard to tell where some of the parameters are being applied (e.g. I'm not sure what the loose fields: line applies to, if anything). Thanks!
I am copy pasting the relevant portion from Filebeat.yml
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/Syslog22/logs/2019.08.26/<IP>.log
input_type: log
document_type: log
# spool_size: 30720
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
fields:
level: debug
ignore_older: 24h
scan_frequency: 1500s
backoff: 1s
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "<IP>:5601"
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["IP:9200"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
With current confguration , we are now getting 341 million lines out of 343
The server is of decent configuration ( 16 vCPU and 64 GB Mem)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.