Limiting number of CPU usage in filebeat

Elastic Stack Beats
1

I want to limit the CPU usage by filebeat . I have tried using "max_procs" but it doesn't seem to work . This high CPU usage may affect the performance of my app in production .
My filebeat.yml is same as this link but i have made the changes as specified in the solution, And it doesn't seem to solve the issue.
(Filebeat CPU usage 400%+)

filebeat.yml

filebeat.inputs:
- type: log 

enabled: true
  max_procs: 2

# Paths that should be crawled and fetched. Glob based paths.
  paths:
       - /var/log/nx/access.log
       - /var/log/nx/error.log
       - /var/log/xyz/web.log

scan_frequency: 5m
close_renamed: true

output.logstash:
# The Logstash hosts
hosts: ["127.0.0.1:8000"] #Not real IP
protocol: "http"

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~

Can you suggest some solution ?

Thank You.

2

Indentation in your configuration file looks off. Can you share your properly formatted config? (See The configuration file format documentation).

Also max_procs is no per input setting, but a global setting. It similar to setting the GOMAXPROCS environment variable:

I guess your config should look more like this:

filebeat.inputs:
- type: log 
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/error.log
    - /var/log/nferx/web.log
  scan_frequency: 5m
  close_renamed: true

max_procs: 2

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~

output.logstash:
  hosts: ["127.0.0.1:8000"] #Not real IP
3

I tried placing max_procs as global setting it is not working for me will share the screenshot with you

Screenshot%20from%202018-12-27%2018-13-39
Screenshot from 2018-12-27 18-13-39.png957×378 64 KB

the filebeat process is using more than two CPU

4

What is CPU usage saying?

How do you start your beat?

max_procs sets the GOMAXPROCS environment variable. This configures the maximum number of active OS threads. If an OS thread is blocked, the runtime might start another thread. But with max_procs: 2 CPU usage should (almost never) pass the 200% mark. If it is still at 400% check how filebeat is started and if filebeat is not started with another configuration file by accident.

5

sudo service filebeat start
sudo filebeat -e -c /etc/filebeat/filebeat.yml

i am using htop to see the number of cores its using.
According to me it should not use more than two cores. Am i wrong ?

max_procs sets the GOMAXPROCS environment variable. This configures the maximum number of active OS threads. If an OS thread is blocked, the runtime might start another thread. But with max_procs: 2 CPU usage should (almost never) pass the 200% mark. If it is still at 400% check how filebeat is started and if filebeat is not started with another configuration file by accident.

I have not sent logs in large number so for me its giving me 0.0% usage

6

i am using htop to see the number of cores its using.
According to me it should not use more than two cores. Am i wrong ?

The runtime might use more OS threads (cores). The max_procs setting configures the maximum number of 'ACTIVE' OS threads. If an OS Threads blocks (e.g. during syscalls), it becomes inactive. In this case the runtime might use another OS Threads, so other go-routines can still work.

1 Like
7

Thank You for the help . its working properly.:+1:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.