My logstash ingest log from /var/log/audit/audit.log After running for one day, the cpu usage is very high (80%). I found that i have many same events. It is wired.
I have delete file in /var/log/audit/ and sincedb file. And i restart the logstash. After running for one day, cpu issue happened again. Let me show the inode info and the sincedb info.
1.Inode info and sincedb info when first audit.log generated
File: /var/log/audit/audit.log'
Size: 13944 Blocks: 32 IO Block: 4096 regular file
Device: 806h/2054d Inode: 131082 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-04 14:41:38.415725119 +0000
Modify: 2018-09-04 14:41:37.418731519 +0000
Change: 2018-09-04 14:41:37.418731519 +0000
131082 0 2054 19022 1536072122.454 /var/log/audit/audit.log
2.Inode info and sincedb info when second audit.log generated
File: /var/log/audit/audit.log.1
Size: 6291731 Blocks: 12304 IO Block: 4096 regular file
Device: 806h/2054d Inode: 131082 Links: 1
Access: (0400/-r--------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-04 21:41:38.461620647 +0000
Modify: 2018-09-04 21:41:37.459627660 +0000
Change: 2018-09-04 21:41:37.459627660 +0000
File: /var/log/audit/audit.log
Size: 2305644 Blocks: 4512 IO Block: 4096 regular file
Device: 806h/2054d Inode: 131086 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-04 21:41:37.459627660 +0000
Modify: 2018-09-05 00:25:23.189746347 +0000
Change: 2018-09-05 00:25:23.189746347 +0000
131082 0 2054 6291731 1536107170.932 /var/log/audit/audit.log
3.Inode info and sincedb info when third audit.log generated
File: /var/log/audit/audit.log.1
Size: 6291489 Blocks: 12304 IO Block: 4096 regular file
Device: 806h/2054d Inode: 131086 Links: 1
Access: (0400/-r--------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-05 01:10:40.251275094 +0000
Modify: 2018-09-05 05:06:54.922262012 +0000
Change: 2018-09-05 05:06:54.923262005 +0000
File: /var/log/audit/audit.log.2
Size: 6291731 Blocks: 12304 IO Block: 4096 regular file
Device: 806h/2054d Inode: 131082 Links: 1
Access: (0400/-r--------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-05 05:06:55.924254999 +0000
Modify: 2018-09-04 21:41:37.459627660 +0000
Change: 2018-09-05 05:06:54.923262005 +0000
File: /var/log/audit/audit.log
Size: 820601 Blocks: 1616 IO Block: 4096 regular file
Device: 806h/2054d Inode: 131179 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-09-05 05:06:54.923262005 +0000
Modify: 2018-09-05 06:04:35.958931936 +0000
Change: 2018-09-05 06:04:35.958931936 +0000
131082 0 2054 6291731 1536127569.392 /var/log/audit/audit.log
From the info above. It seems that the sincedb not updated prperly. It caused logstash ingest logs from file with inode number 131082 repeatedly which make event rate up to 10k/s.