I configured Filebeat, Logstash, ES and Kibana, to gather nginx-ingress logs from Kubernetes.
Since there's a lot of logs (10 hits in a second), the presented data in Kibana is good, but behind the current time.
For example , shown logs are for 17:05:24 but now is 17:15, and this delta is getting bigger, because of the amount of logs.
I tried to use scan_frequency and close_inactive in filebeat config
apiVersion: v1 kind: ConfigMap metadata: name: filebeat-prospectors namespace: kube-system labels: k8s-app: filebeat kubernetes.io/cluster-service: "true" data: kubernetes.yml: |- - type: docker containers.ids: - "*" processors: - add_kubernetes_metadata: in_cluster: true scan_frequency: 10s close_inactive: 1m
But it didn't really helps, any best practise for such case?