# Beats **URL:** https://discuss.elastic.co/c/elastic-stack/beats/28.md?page=281 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 282 --- ## [Can I use “bulk\_max\_size” & “bulk\_flush\_frequency” to limit how much data Filebeat can send per min?](https://discuss.elastic.co/t/can-i-use-bulk-max-size-bulk-flush-frequency-to-limit-how-much-data-filebeat-can-send-per-min/216930)
**Author:** [@wouyang408](https://discuss.elastic.co/u/wouyang408)\ **Replies:** 4\ **Last updated:** [January 29, 2020, 10:51pm UTC](https://discuss.elastic.co/t/can-i-use-bulk-max-size-bulk-flush-frequency-to-limit-how-much-data-filebeat-can-send-per-min/216930 "2020-01-29T22:51:23Z")
Hi all, Our data pipeline uses Filebeat to send logs to Kafka. Unfortunately, every once a while the amount of data sent by Filebeat instances exceeds the amount of the data that can be handled by Kafka. Therefore, we … --- ## [Filebeat - iptables module](https://discuss.elastic.co/t/filebeat-iptables-module/217062)
**Author:** [@sentient](https://discuss.elastic.co/u/sentient)\ **Replies:** 2\ **Last updated:** [January 29, 2020, 9:54pm UTC](https://discuss.elastic.co/t/filebeat-iptables-module/217062 "2020-01-29T21:54:22Z")
Quick question, does the filebeat iptables module work when using ufw ? If is my basic understanding the ufw works on top of iptables. I do see the ufw events in the syslog, but I don't get any iptables events into SI… --- ## [Exception: Key 'my\_field' found in event is not documented!](https://discuss.elastic.co/t/exception-key-my-field-found-in-event-is-not-documented/216902)
**Author:** [@Jim\_Ivey](https://discuss.elastic.co/u/Jim_Ivey)\ **Replies:** 3\ **Last updated:** [January 29, 2020, 6:09pm UTC](https://discuss.elastic.co/t/exception-key-my-field-found-in-event-is-not-documented/216902 "2020-01-29T18:09:49Z")
I'm getting this error in testing my module: Exception: Key 'my\_field' found in event is not documented! Mr. Google only finds discussion in github about PRs related to this. I don't know what this error means. Here's… --- ## [Single module for multiple similar logs](https://discuss.elastic.co/t/single-module-for-multiple-similar-logs/216931)
**Author:** [@Jim\_Ivey](https://discuss.elastic.co/u/Jim_Ivey)\ **Replies:** 3\ **Last updated:** [January 29, 2020, 5:07pm UTC](https://discuss.elastic.co/t/single-module-for-multiple-similar-logs/216931 "2020-01-29T17:07:15Z")
We have multiple instances of the same service. Consider, for example: sudo service my\_service\_1 start sudo service my\_service\_2 start sudo service my\_service\_3 start with log files respectively at: /var/log/my\_servi… --- ## [Dynamic value of the add\_field processor](https://discuss.elastic.co/t/dynamic-value-of-the-add-field-processor/216827)
**Author:** [@nahiko2000](https://discuss.elastic.co/u/nahiko2000)\ **Replies:** 2\ **Last updated:** [January 29, 2020, 3:56pm UTC](https://discuss.elastic.co/t/dynamic-value-of-the-add-field-processor/216827 "2020-01-29T15:56:25Z")
Hi! In a Filebeat input I have a processor in Filebeat like this: processors: - add\_fields: when: regexp: message: '\[0-9\]{8}\\-?\[a-zA-Z\]' fileds: newfield: XXX So, when a new line gets … --- ## [Beats on VM vs on Kubernetes](https://discuss.elastic.co/t/beats-on-vm-vs-on-kubernetes/217023)
**Author:** [@Mauricio\_Borges](https://discuss.elastic.co/u/Mauricio_Borges)\ **Replies:** 0\ **Last updated:** [January 29, 2020, 3:04pm UTC](https://discuss.elastic.co/t/beats-on-vm-vs-on-kubernetes/217023 "2020-01-29T15:04:08Z")
Hi team! I exploring Beats and have some doubts about differences and limit between Beats installed over Operation System and ones over Kubernetes. 1 - At first look, appears Beats installed over Host/VM will collect m… --- ## [Top-level configuration not working](https://discuss.elastic.co/t/top-level-configuration-not-working/216961)
**Author:** [@ELK4Life](https://discuss.elastic.co/u/ELK4Life)\ **Replies:** 2\ **Last updated:** [January 29, 2020, 2:33pm UTC](https://discuss.elastic.co/t/top-level-configuration-not-working/216961 "2020-01-29T14:33:38Z")
Dear Logstash community, I have challenged myself to capture my network data using TShark and make custom dashboards in Kibana. I have the following setup: 1 Ubuntu 18.04 VM with ElasticSearch, Logstash and Kibana do… --- ## [FileBeat Processing @timestamp instead of log @timestamp](https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979)
**Author:** [@salunkerahul](https://discuss.elastic.co/u/salunkerahul)\ **Replies:** 2\ **Last updated:** [January 29, 2020, 2:20pm UTC](https://discuss.elastic.co/t/filebeat-processing-timestamp-instead-of-log-timestamp/216979 "2020-01-29T14:20:02Z")
I'm using the following flow FileBeat-\>Elastic-\>Kibana on Windows-7 using v7.5.2 and the timestamp as seen in the Kibana Discover window always corresponds to the time the log was processed rather than represent the log'… --- ## [Debugging test.log-expected.json](https://discuss.elastic.co/t/debugging-test-log-expected-json/216738)
**Author:** [@Jim\_Ivey](https://discuss.elastic.co/u/Jim_Ivey)\ **Replies:** 4\ **Last updated:** [January 29, 2020, 1:38pm UTC](https://discuss.elastic.co/t/debugging-test-log-expected-json/216738 "2020-01-29T13:38:39Z")
I'm trying to create a module for filebeat. The tests involving test.log and test.log-expected.json are failing. I'm having a hard time figuring out why. Are there files in build that would give hints? Is there a way… --- ## [What is the difference between kube-state-metric and metricbeat](https://discuss.elastic.co/t/what-is-the-difference-between-kube-state-metric-and-metricbeat/216811)
**Author:** [@kasim123](https://discuss.elastic.co/u/kasim123)\ **Replies:** 2\ **Last updated:** [January 29, 2020, 5:52am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-kube-state-metric-and-metricbeat/216811 "2020-01-29T05:52:00Z")
Hi Team, I would like to know the difference between kube-state-metric and metricbeat. I have EFK stack installed in k8s cluster. I would like to get metrics from containers, pod, cpu usage. Which tools is suggested… --- ## [Filebeat: Failed to connect to backoff async dial tcp connect connection refused](https://discuss.elastic.co/t/filebeat-failed-to-connect-to-backoff-async-dial-tcp-connect-connection-refused/216556)
**Author:** [@mcmeathouse](https://discuss.elastic.co/u/mcmeathouse)\ **Replies:** 3\ **Last updated:** [January 29, 2020, 5:22am UTC](https://discuss.elastic.co/t/filebeat-failed-to-connect-to-backoff-async-dial-tcp-connect-connection-refused/216556 "2020-01-29T05:22:33Z")
I've gone through other posts with this same issue, and I have come up empty handed. I've set up ELK rigs before, and this is the first time encountering this problem. General Info: \*New cluster with 4 nodes. 1 for kib… --- ## [Logs of Different Indices are merging with each other](https://discuss.elastic.co/t/logs-of-different-indices-are-merging-with-each-other/216610)
**Author:** [@mohammadawaisjavaid](https://discuss.elastic.co/u/mohammadawaisjavaid)\ **Replies:** 2\ **Last updated:** [January 29, 2020, 5:21am UTC](https://discuss.elastic.co/t/logs-of-different-indices-are-merging-with-each-other/216610 "2020-01-29T05:21:50Z")
Hello all. I am sending logs of windows to a index using winlogbeat and also I am sending logs of Icinga to other index using icingabeat. The problem I am facing is that the logs are not going to specific index that i h… --- ## [MSSQL Deadlocks metric](https://discuss.elastic.co/t/mssql-deadlocks-metric/216869)
**Author:** [@razgrim](https://discuss.elastic.co/u/razgrim)\ **Replies:** 2\ **Last updated:** [January 28, 2020, 11:27pm UTC](https://discuss.elastic.co/t/mssql-deadlocks-metric/216869 "2020-01-28T23:27:39Z")
Proposing feature per https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html Our DBA is looking to have MSSQL deadlocks included in metricbeat mssql performance metricset. The following query ge… --- ## [Metricbeat Data Output through HTTP](https://discuss.elastic.co/t/metricbeat-data-output-through-http/216342)
**Author:** [@Yiw](https://discuss.elastic.co/u/Yiw)\ **Replies:** 4\ **Last updated:** [January 28, 2020, 9:39pm UTC](https://discuss.elastic.co/t/metricbeat-data-output-through-http/216342 "2020-01-28T21:39:13Z")
Hi, I am using Metricbeat to get system data. Can my server get these data as json from Metricbeat through http directly without using Elasticsearch. What I mean is I plan to write a server and listen to a port, say 800… --- ## [Unable to see filebeat in kibana](https://discuss.elastic.co/t/unable-to-see-filebeat-in-kibana/216856)
**Author:** [@Enzo\_baker](https://discuss.elastic.co/u/Enzo_baker)\ **Replies:** 1\ **Last updated:** [January 28, 2020, 5:14pm UTC](https://discuss.elastic.co/t/unable-to-see-filebeat-in-kibana/216856 "2020-01-28T17:14:52Z")
Filebeat yml: filebeat.inputs: Each - is an input. Most options can be set at the input level, so you can use different inputs for various configurations. Below are the input specific configurations. type: log Chang… --- ## [How to get/download the filebeat 7.5.2 source code?](https://discuss.elastic.co/t/how-to-get-download-the-filebeat-7-5-2-source-code/216780)
**Author:** [@davidz101](https://discuss.elastic.co/u/davidz101)\ **Replies:** 2\ **Last updated:** [January 28, 2020, 4:37pm UTC](https://discuss.elastic.co/t/how-to-get-download-the-filebeat-7-5-2-source-code/216780 "2020-01-28T16:37:13Z")
Hi, I am new to filebeat, I just downloaded filebeat 7.5.2 version, how can I checkout the corresponding source code for filebeat version 7.5.2? Thanks -David --- ## [Metricbeat and kibana synchronisation!](https://discuss.elastic.co/t/metricbeat-and-kibana-synchronisation/216436)
**Author:** [@Annonyme1](https://discuss.elastic.co/u/Annonyme1)\ **Replies:** 16\ **Last updated:** [January 28, 2020, 9:47am UTC](https://discuss.elastic.co/t/metricbeat-and-kibana-synchronisation/216436 "2020-01-28T09:47:20Z")
I'am new in ELK tools ! i'am starting to minitor my own server with @172.16.0.179 ! my elasticsearch @172.16.0.179:9200 my kibana @172.16.0.179:5601 i' already installed metricbeat to collect metrics from my system an… --- ## [Unable to fetch system.filesystem.used.pct from metricbeat index](https://discuss.elastic.co/t/unable-to-fetch-system-filesystem-used-pct-from-metricbeat-index/216825)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 5\ **Last updated:** [January 28, 2020, 1:56pm UTC](https://discuss.elastic.co/t/unable-to-fetch-system-filesystem-used-pct-from-metricbeat-index/216825 "2020-01-28T13:56:07Z")
Hi Team, I am working on creating watcher alert (which would trigger alert when the threshold percent ins exceeded from the target server). My metricbeat index contains almost 30 hostname's in them,but i am now concer… --- ## [High Metricbeat CPU usage](https://discuss.elastic.co/t/high-metricbeat-cpu-usage/216745)
**Author:** [@premierpsp](https://discuss.elastic.co/u/premierpsp)\ **Replies:** 3\ **Last updated:** [January 28, 2020, 12:47pm UTC](https://discuss.elastic.co/t/high-metricbeat-cpu-usage/216745 "2020-01-28T12:47:13Z")
Hi, I’m using Metricbeat 7.5.x (rpm) on a RHEL, OEL 5.x, 6.x with system module enabled each 10s per period. The CPU reaches 85, 100% and I don't think they are normal values, isn’t? I’m using ELK on a CentOS 7. I woul… --- ## [Syslog.input and logstash output](https://discuss.elastic.co/t/syslog-input-and-logstash-output/216824)
**Author:** [@HaZet](https://discuss.elastic.co/u/HaZet)\ **Replies:** 0\ **Last updated:** [January 28, 2020, 11:15am UTC](https://discuss.elastic.co/t/syslog-input-and-logstash-output/216824 "2020-01-28T11:15:49Z")
When I enable the filebeat.inputs.type syslog to receive sysslog messages over a socket and send it to logstash I get the following error in the filebeat.log: 2020-01-28T11:03:29.279+0100 ERROR logstash/async.go:256 Fa… --- ## [Filebeat Scheduling in logstash](https://discuss.elastic.co/t/filebeat-scheduling-in-logstash/216807)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 0\ **Last updated:** [January 28, 2020, 10:21am UTC](https://discuss.elastic.co/t/filebeat-scheduling-in-logstash/216807 "2020-01-28T10:21:03Z")
Hi Team, can we fetch data from Logstash while pushing data through filebeat and scheduling in logstash configuration file. Thanks and Regards, Sagar --- ## [Writing of registry returned error:](https://discuss.elastic.co/t/writing-of-registry-returned-error/216651)
**Author:** [@Robin020](https://discuss.elastic.co/u/Robin020)\ **Replies:** 4\ **Last updated:** [January 28, 2020, 10:10am UTC](https://discuss.elastic.co/t/writing-of-registry-returned-error/216651 "2020-01-28T10:10:13Z")
Hi, I get the following error constantly no matter what I do: Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or … --- ## [Filebeat ignore\_older](https://discuss.elastic.co/t/filebeat-ignore-older/216774)
**Author:** [@Anindita\_Chavan](https://discuss.elastic.co/u/Anindita_Chavan)\ **Replies:** 1\ **Last updated:** [January 28, 2020, 10:09am UTC](https://discuss.elastic.co/t/filebeat-ignore-older/216774 "2020-01-28T10:09:01Z")
Hello community, I am sending logs from filebeat to logstash, once I send logs from a particular file to logstash via filebeat, it won't run again for the same file. I have tried setting ignore\_older = 0 in filebeat, ho… --- ## [Cannot get Filebeat to stream](https://discuss.elastic.co/t/cannot-get-filebeat-to-stream/216529)
**Author:** [@arutale](https://discuss.elastic.co/u/arutale)\ **Replies:** 3\ **Last updated:** [January 28, 2020, 8:45am UTC](https://discuss.elastic.co/t/cannot-get-filebeat-to-stream/216529 "2020-01-28T08:45:23Z")
I'm trying out elastic.co I have installed Beats on my server and i'm trying to configure the filebeat.yml file. Under cloud, i have applied the configuration following these instructions: Configure the output for the… --- ## [Docker Logs not sent to ES](https://discuss.elastic.co/t/docker-logs-not-sent-to-es/216133)
**Author:** [@rorixrebel](https://discuss.elastic.co/u/rorixrebel)\ **Replies:** 4\ **Last updated:** [January 28, 2020, 3:41am UTC](https://discuss.elastic.co/t/docker-logs-not-sent-to-es/216133 "2020-01-28T03:41:57Z")
So i got a home cluster running on Docker which is working fine and i can push data to it from filebeat and metricbeat, but im having issues understanding why my docker logs are not being sent to the cluster. i have the… --- ## [Metricbeat: How to monitor podman resources](https://discuss.elastic.co/t/metricbeat-how-to-monitor-podman-resources/216619)
**Author:** [@giraffe](https://discuss.elastic.co/u/giraffe)\ **Replies:** 2\ **Last updated:** [January 28, 2020, 2:00am UTC](https://discuss.elastic.co/t/metricbeat-how-to-monitor-podman-resources/216619 "2020-01-28T02:00:45Z")
Hi Team environment: CentOS 7.7 metricbeat 7.5.2 podman 1.4.4-4 I'm using podman (https://podman.io/) for run containers. Podman is a daemonless container engine for developing, managing, and running OCI Containers… --- ## [Multiple 'Multiline' blocks in same filebeat.yml - Filebeats](https://discuss.elastic.co/t/multiple-multiline-blocks-in-same-filebeat-yml-filebeats/215922)
**Author:** [@klang](https://discuss.elastic.co/u/klang)\ **Replies:** 2\ **Last updated:** [January 21, 2020, 5:49pm UTC](https://discuss.elastic.co/t/multiple-multiline-blocks-in-same-filebeat-yml-filebeats/215922 "2020-01-21T17:49:13Z")
I'm looking to understand if I may have more than 1 multiline.pattern defined in a filebeat configuration of which these multiline configurations would be against the same log file. When I run something similar to the be… --- ## [RabbitMQ redeliveries metrics?](https://discuss.elastic.co/t/rabbitmq-redeliveries-metrics/215527)
**Author:** [@IamaBase](https://discuss.elastic.co/u/IamaBase)\ **Replies:** 2\ **Last updated:** [January 27, 2020, 10:49pm UTC](https://discuss.elastic.co/t/rabbitmq-redeliveries-metrics/215527 "2020-01-27T22:49:50Z")
I notice that these used to be provided in the exchange metricset: https://github.com/elastic/beats/pull/6607/files/34b28f231537933ee1e0296393f6e726bc6d060d But it appears they are not included as of 7.5.1. Is there a w… --- ## [Filebeat configuration in main config or seperate?](https://discuss.elastic.co/t/filebeat-configuration-in-main-config-or-seperate/215724)
**Author:** [@hlamsc](https://discuss.elastic.co/u/hlamsc)\ **Replies:** 1\ **Last updated:** [January 27, 2020, 11:01pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-main-config-or-seperate/215724 "2020-01-27T23:01:31Z")
Hello there, we want to use the "close\_renamed: true" option for our filebeat. Im currently asking myself if I could set it "globally" in the filebeat.yml (so every config in conf.d is affected) or do I have to set it s… --- ## [Calculate total uptime/downtime](https://discuss.elastic.co/t/calculate-total-uptime-downtime/216627)
**Author:** [@Matthias\_Seidl](https://discuss.elastic.co/u/Matthias_Seidl)\ **Replies:** 2\ **Last updated:** [January 27, 2020, 5:58pm UTC](https://discuss.elastic.co/t/calculate-total-uptime-downtime/216627 "2020-01-27T17:58:14Z")
Hi, I'm thinking about setting up Hearbeat to monitor the uptime of my hosts. Is it possible with Hearbeat to calculate the time that a host has been up or down over a month or a year? This functionality would be really… [Previous page](https://discuss.elastic.co/c/elastic-stack/beats/28.md?page=280) [Next page](https://discuss.elastic.co/c/elastic-stack/beats/28.md?page=282)