Hi Champs,
My filebeat consumtipn is very high and using more than 4GB of RAM, only log files which I am pushing are some app logs , /var/log/messages and /var/log/secure. Npt sure why this happening. Please help
[root@ip-10-1-16-7 ~]# service filebeat status
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-07-07 12:53:30 UTC; 20h ago
Docs: https://www.elastic.co/beats/filebeat
Main PID: 1400956 (filebeat)
Tasks: 12 (limit: 101212)
Memory: 4.0G
CGroup: /system.slice/filebeat.service
└─1400956 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat>
Jul 08 09:27:27 ip-10-1-16-7.eastus.tessell.internal filebeat[1400956]: {"log.level":"info","@timestamp":"2023-07-08T09:27:27.725Z","log.>
Jul 08 09:27:27 ip-10-1-16-7.eastus.tessell.internal filebeat[1400956]: {"log.level":"error","@timestamp":"2023-07-08T09:27:27.725Z","log>
Top output
top - 09:29:01 up 8 days, 18:43, 1 user, load average: 1.03, 1.26, 1.31
Tasks: 300 total, 2 running, 297 sleeping, 0 stopped, 1 zombie
%Cpu0 : 21.9 us, 5.0 sy, 0.0 ni, 72.8 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st
%Cpu1 : 27.6 us, 1.7 sy, 0.0 ni, 70.1 id, 0.3 wa, 0.3 hi, 0.0 si, 0.0 st
%Cpu2 : 7.0 us, 1.7 sy, 0.0 ni, 91.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st
%Cpu3 : 37.1 us, 1.7 sy, 0.0 ni, 60.5 id, 0.0 wa, 0.7 hi, 0.0 si, 0.0 st
MiB Mem : 15873.1 total, 660.3 free, 13134.7 used, 2078.1 buff/cache
MiB Swap: 20480.0 total, 19253.1 free, 1226.9 used. 1406.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1400956 root 20 0 6633268 4.1g 36772 S 89.7 26.2 1200:03 filebeat
Logs size is
[root@ip-10-1-16-7 ~]# du -sh /var/log/secure
283M /var/log/secure
[root@ip-10-1-16-7 ~]# du -sh /var/log/messages
2.0G /var/log/messages
[root@ip-10-1-16-7 ~]# du -sh /tmp/tessell-logs/*/*.log
4.0K /tmp/tessell-logs/0024e778-1b5a-11ee-8c2d-6271dd09f167/genie_disable.log
4.0K /tmp/tessell-logs/0024e778-1b5a-11ee-8c2d-6271dd09f167/genie_enable.log
8.0K /tmp/tessell-logs/0024e778-1b5a-11ee-8c2d-6271dd09f167/python-task-executor-0024e778-1b5a-11ee-8c2d-6271dd09f167-0024e778-1b5a-11ee-8c2d-6271dd09f167_0.log
0 /tmp/tessell-logs/0024e778-1b5a-11ee-8c2d-6271dd09f167/task-0024e778-1b5a-11ee-8c2d-6271dd09f167-0024e778-1b5a-11ee-8c2d-6271dd09f167_0.log
32K /tmp/tessell-logs/0024e778-1b5a-11ee-8c2d-6271dd09f167/tessell-lambda.log
4.0K /tmp/tessell-logs/1e26afac-f3cf-42e0-83fa-c0151f7f84fc/python-task-executor-1e26afac-f3cf-42e0-83fa-c0151f7f84fc-jobBackupCreation_01b51c4a-5a90-4f68-bdba-f7bc563a026d.log
64K /tmp/tessell-logs/1e26afac-f3cf-42e0-83fa-c0151f7f84fc/task-1e26afac-f3cf-42e0-83fa-c0151f7f84fc-jobBackupCreation_01b51c4a-5a90-4f68-bdba-f7bc563a026d.log
52K /tmp/tessell-logs/1e26afac-f3cf-42e0-83fa-c0151f7f84fc/tessell-lambda.log
4.0K /tmp/tessell-logs/35375835-f89f-4be9-94e2-c78ebd51365b/python-task-executor-35375835-f89f-4be9-94e2-c78ebd51365b-jobBackupCreation_70a81de1-c9aa-43b0-83fe-10130329410d.log
64K /tmp/tessell-logs/35375835-f89f-4be9-94e2-c78ebd51365b/task-35375835-f89f-4be9-94e2-c78ebd51365b-jobBackupCreation_70a81de1-c9aa-43b0-83fe-10130329410d.log
52K /tmp/tessell-logs/35375835-f89f-4be9-94e2-c78ebd51365b/tessell-lambda.log
4.0K /tmp/tessell-logs/482c4e54-97b3-41f4-be89-acb60b2d84f9/python-task-executor-482c4e54-97b3-41f4-be89-acb60b2d84f9-jobBackupCreation_417ee8b0-7708-40f0-a0be-e5d09131342a.log
64K /tmp/tessell-logs/482c4e54-97b3-41f4-be89-acb60b2d84f9/task-482c4e54-97b3-41f4-be89-acb60b2d84f9-jobBackupCreation_417ee8b0-7708-40f0-a0be-e5d09131342a.log
52K /tmp/tessell-logs/482c4e54-97b3-41f4-be89-acb60b2d84f9/tessell-lambda.log
4.0K /tmp/tessell-logs/6ade1c58-5ff5-4cc4-a264-69d027a157c1/genie_enable.log
4.0K /tmp/tessell-logs/6ade1c58-5ff5-4cc4-a264-69d027a157c1/python-task-executor-6ade1c58-5ff5-4cc4-a264-69d027a157c1-6ade1c58-5ff5-4cc4-a264-69d027a157c1_0.log
0 /tmp/tessell-logs/6ade1c58-5ff5-4cc4-a264-69d027a157c1/task-6ade1c58-5ff5-4cc4-a264-69d027a157c1-6ade1c58-5ff5-4cc4-a264-69d027a157c1_0.log
20K /tmp/tessell-logs/6ade1c58-5ff5-4cc4-a264-69d027a157c1/tessell-lambda.log
116K /tmp/tessell-logs/common/LogArchivalCleanup.log
84K /tmp/tessell-logs/common/LogSweep.log
60K /tmp/tessell-logs/common/recycle_files.log