Filebeat logs stored in /tmp are causing pod eviction

Elastic Stack Beats
1

Hi all,
i am seeing an issue on my environment which is using filebeat for logging and monitoring where files with large amounts of storage used are getting created. I presume these are log files created by filebeat and these files are getting stored in my pods /tmp directory. I am using openshift 4.11.9 with kubernetes 1.24.

total 680892
-rw-r--r--. 1 1000680000 1000680000 45518518 Mar  7 01:35 monitoring-stdout---supervisor-jti9vqc6.log
-rw-r-----. 1 1000680000 1000680000  5803121 Mar  7 01:35 pch4543779432628550477.history.gz
-rw-r-----. 1 1000680000 1000680000 54381482 Mar  7 01:35 pch4543779432628550477.history
-rw-r-----. 1 1000680000 1000680000  2185531 Mar  7 01:35 pch6805951925590054337.history
drwxrwsrwx. 4 root       1000680000     4096 Mar  7 01:28 .
-rw-r--r--. 1 1000680000 1000680000 52428996 Mar  7 01:28 monitoring-stdout---supervisor-jti9vqc6.log.1
-rw-r-----. 1 1000680000 1000680000    72101 Mar  7 01:26 pch6805951925590054337.history.gz
-rw-r--r--. 1 1000680000 1000680000 52428802 Mar  7 01:20 monitoring-stdout---supervisor-jti9vqc6.log.2
-rw-r--r--. 1 1000680000 1000680000 52428984 Mar  7 01:12 monitoring-stdout---supervisor-jti9vqc6.log.3
-rw-r--r--. 1 1000680000 1000680000 52428907 Mar  7 01:04 monitoring-stdout---supervisor-jti9vqc6.log.4
-rw-r--r--. 1 1000680000 1000680000 52429189 Mar  7 00:55 monitoring-stdout---supervisor-jti9vqc6.log.5
-rw-r--r--. 1 1000680000 1000680000 52428807 Mar  7 00:47 monitoring-stdout---supervisor-jti9vqc6.log.6
-rw-r--r--. 1 1000680000 1000680000 52428867 Mar  7 00:39 monitoring-stdout---supervisor-jti9vqc6.log.7
-rw-r--r--. 1 1000680000 1000680000 52429055 Mar  7 00:31 monitoring-stdout---supervisor-jti9vqc6.log.8
-rw-r--r--. 1 1000680000 1000680000 52428920 Mar  7 00:23 monitoring-stdout---supervisor-jti9vqc6.log.9
-rw-r--r--. 1 1000680000 1000680000 52428974 Mar  7 00:15 monitoring-stdout---supervisor-jti9vqc6.log.10
-rw-r-----. 1 1000680000 1000680000   173586 Mar  6 23:41 pch783860385935287176.history.gz
-rw-r-----. 1 1000680000 1000680000 11674655 Mar  6 23:41 pch7390231010312236403.history.gz
drwxr-s---. 2 1000680000 1000680000     4096 Mar  6 19:41 axis2-tmp-597419792942874660.tmp
-rw-r-----. 1 1000680000 1000680000        0 Mar  6 19:41 axis2-tmp-597419792942874660.tmp.lck
drwxrwxrwt. 4 1000680000 root             64 Mar  6 19:40 .com_ibm_tools_attach
-rw-------. 1 1000680000 1000680000        0 Mar  6 19:39 filebeat-syslog-stderr---supervisor-j80_3v5j.log
-rw-------. 1 1000680000 1000680000        0 Mar  6 19:39 filebeat-syslog-stdout---supervisor-hq39mt_v.log
-rw-------. 1 1000680000 1000680000        0 Mar  6 19:39 filebeat-was-stderr---supervisor-en73_cgc.log
-rw-------. 1 1000680000 1000680000        0 Mar  6 19:39 filebeat-was-stdout---supervisor-v0comu9p.log
-rw-------. 1 1000680000 1000680000        0 Mar  6 19:39 filebeat-cpe-stderr---supervisor-v8837rxx.log
-rw-------. 1 1000680000 1000680000        0 Mar  6 19:39 filebeat-cpe-stdout---supervisor-_0l469zf.log
-rw-r--r--. 1 1000680000 1000680000        3 Mar  6 19:39 supervisord.pid
srwx------. 1 1000680000 1000680000        0 Mar  6 19:39 supervisor.sock
dr-xr-xr-x. 1 root       root             28 Mar  6 19:39 ..

As you can see these are the files being created and its the files starting from monitoring-*.log that i am concerned about.
i was wondering if there were any solutions that could be provided to any of the following points

  1. Reduce the file size
  2. change the directory from /tmp to some other directory
  3. Reduce the amount of time these files take to wrap the content around each other.

any thoughts on this would be gladly appreciated

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.