Logstash filling up disk space beyond queue.max_bytes

Sindhu_Bandi · June 19, 2023, 3:37pm

Logstash persistent volume size is increasing beyond configured queue.max_bytes
Scenario:
Logstash : 7.17.3
Env : On Kubernetes cluster
Persistence: Enabled

logstash.yml:
----
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
xpack.monitoring.enabled: false
queue.type: persisted
queue.max_bytes: 2gb

Persistent disk is filled up beyond configured queue.max_bytes

$ sudo du -sh  /mnt/onl/sdpon/data/logstash/data/
15G     /mnt/onl/sdpon/data/logstash/data/

A total of 236 page files were created

leandrojmp · June 19, 2023, 6:07pm

Are you sure this is recent data?

The date on the files you shared are from Aug 2022.

This may be some leftover. Can you liste the files in the directory and paste it here as plain text, not as a screenshot?

Sindhu_Bandi · June 20, 2023, 3:10am

Hi @leandrojmp, pasted the directory contents.

oltausr@localhost:~$ ls -hlrta /mnt/onl/sdpon/data/logstash/data/queue/main/
total 15G
drwxrwsr-x 3 root root 4.0K Aug  9  2022 ..
-rw-rw-r-- 1 root root    0 Aug  9  2022 .lock
-rw-rw-r-- 1 root root    4 Aug  9  2022 .queue-version
-rw-rw-r-- 1 root root  64M Aug  9  2022 page.0
-rw-rw-r-- 1 root root   34 Aug  9  2022 checkpoint.0
-rw-rw-r-- 1 root root  64M Aug 10  2022 page.1
-rw-rw-r-- 1 root root   34 Aug 10  2022 checkpoint.1
-rw-rw-r-- 1 root root  64M Aug 10  2022 page.2
-rw-rw-r-- 1 root root   34 Aug 10  2022 checkpoint.2
-rw-rw-r-- 1 root root  64M Aug 10  2022 page.3
-rw-rw-r-- 1 root root   34 Aug 10  2022 checkpoint.3
-rw-rw-r-- 1 root root  64M Aug 11  2022 page.4
-rw-rw-r-- 1 root root   34 Aug 11  2022 checkpoint.4
-rw-rw-r-- 1 root root  64M Aug 11  2022 page.5
-rw-rw-r-- 1 root root   34 Aug 11  2022 checkpoint.5
-rw-rw-r-- 1 root root  64M Aug 11  2022 page.6
-rw-rw-r-- 1 root root   34 Aug 11  2022 checkpoint.6
-rw-rw-r-- 1 root root  64M Aug 11  2022 page.7
-rw-rw-r-- 1 root root   34 Aug 11  2022 checkpoint.7
-rw-rw-r-- 1 root root  64M Aug 12  2022 page.8
-rw-rw-r-- 1 root root   34 Aug 12  2022 checkpoint.8
-rw-rw-r-- 1 root root  64M Aug 12  2022 page.9
-rw-rw-r-- 1 root root   34 Aug 12  2022 checkpoint.9
-rw-rw-r-- 1 root root  64M Aug 12  2022 page.10
-rw-rw-r-- 1 root root   34 Aug 12  2022 checkpoint.10
-rw-rw-r-- 1 root root  64M Aug 12  2022 page.11
-rw-rw-r-- 1 root root   34 Aug 12  2022 checkpoint.11
-rw-rw-r-- 1 root root  64M Aug 13  2022 page.12
-rw-rw-r-- 1 root root   34 Aug 13  2022 checkpoint.12
-rw-rw-r-- 1 root root  64M Aug 13  2022 page.13
-rw-rw-r-- 1 root root   34 Aug 13  2022 checkpoint.13
-rw-rw-r-- 1 root root  64M Aug 13  2022 page.14
-rw-rw-r-- 1 root root   34 Aug 13  2022 checkpoint.14
-rw-rw-r-- 1 root root  64M Aug 13  2022 page.15
-rw-rw-r-- 1 root root   34 Aug 13  2022 checkpoint.15
-rw-rw-r-- 1 root root  64M Aug 14  2022 page.16
-rw-rw-r-- 1 root root   34 Aug 14  2022 checkpoint.16
-rw-rw-r-- 1 root root  64M Aug 14  2022 page.17
-rw-rw-r-- 1 root root   34 Aug 14  2022 checkpoint.17
-rw-rw-r-- 1 root root  64M Aug 14  2022 page.18
-rw-rw-r-- 1 root root   34 Aug 14  2022 checkpoint.18
-rw-rw-r-- 1 root root  64M Aug 14  2022 page.19
-rw-rw-r-- 1 root root   34 Aug 14  2022 checkpoint.19
-rw-rw-r-- 1 root root  64M Aug 15  2022 page.20
-rw-rw-r-- 1 root root   34 Aug 15  2022 checkpoint.20
-rw-rw-r-- 1 root root  64M Aug 15  2022 page.21
-rw-rw-r-- 1 root root   34 Aug 15  2022 checkpoint.21
-rw-rw-r-- 1 root root  64M Aug 15  2022 page.22
-rw-rw-r-- 1 root root   34 Aug 15  2022 checkpoint.22
-rw-rw-r-- 1 root root  64M Aug 16  2022 page.23
-rw-rw-r-- 1 root root   34 Aug 16  2022 checkpoint.23
-rw-rw-r-- 1 root root  64M Aug 16  2022 page.24
-rw-rw-r-- 1 root root   34 Aug 16  2022 checkpoint.24
-rw-rw-r-- 1 root root  64M Aug 16  2022 page.25
-rw-rw-r-- 1 root root   34 Aug 16  2022 checkpoint.25
-rw-rw-r-- 1 root root  64M Aug 16  2022 page.26
-rw-rw-r-- 1 root root   34 Aug 16  2022 checkpoint.26
-rw-rw-r-- 1 root root  64M Aug 17  2022 page.27
-rw-rw-r-- 1 root root   34 Aug 17  2022 checkpoint.27
-rw-rw-r-- 1 root root  64M Aug 17  2022 page.28
-rw-rw-r-- 1 root root   34 Aug 17  2022 checkpoint.28
-rw-rw-r-- 1 root root  64M Aug 17  2022 page.29
-rw-rw-r-- 1 root root   34 Aug 17  2022 checkpoint.29
-rw-rw-r-- 1 root root  64M Aug 17  2022 page.30
-rw-rw-r-- 1 root root   34 Aug 17  2022 checkpoint.30
-rw-rw-r-- 1 root root  64M Aug 18  2022 page.31
-rw-rw-r-- 1 root root   34 Aug 18  2022 checkpoint.31
-rw-rw-r-- 1 root root  64M Aug 18  2022 page.32
-rw-rw-r-- 1 root root   34 Oct  5  2022 checkpoint.32
-rw-rw-r-- 1 root root  64M Oct  5  2022 page.33
-rw-rw-r-- 1 root root   34 Oct 10  2022 checkpoint.33
-rw-rw-r-- 1 root root  64M Oct 10  2022 page.34
-rw-rw-r-- 1 root root   34 Oct 11  2022 checkpoint.34
-rw-rw-r-- 1 root root  64M Oct 11  2022 page.35
-rw-rw-r-- 1 root root   34 Oct 11  2022 checkpoint.35
-rw-rw-r-- 1 root root  64M Oct 11  2022 page.36
-rw-rw-r-- 1 root root   34 Oct 15  2022 checkpoint.36
-rw-rw-r-- 1 root root  64M Oct 15  2022 page.37
-rw-rw-r-- 1 root root   34 Oct 15  2022 checkpoint.37
-rw-rw-r-- 1 root root  64M Oct 15  2022 page.38
-rw-rw-r-- 1 root root   34 Oct 16  2022 checkpoint.38
-rw-rw-r-- 1 root root  64M Oct 16  2022 page.39
-rw-rw-r-- 1 root root   34 Oct 18  2022 checkpoint.39
-rw-rw-r-- 1 root root  64M Oct 18  2022 page.40
-rw-rw-r-- 1 root root   34 Oct 18  2022 checkpoint.40
-rw-rw-r-- 1 root root  64M Oct 18  2022 page.41
-rw-rw-r-- 1 root root   34 Oct 24  2022 checkpoint.41
-rw-rw-r-- 1 root root  64M Oct 24  2022 page.42
-rw-rw-r-- 1 root root   34 Oct 24  2022 checkpoint.42
-rw-rw-r-- 1 root root  64M Oct 24  2022 page.43
-rw-rw-r-- 1 root root   34 Oct 25  2022 checkpoint.43
-rw-rw-r-- 1 root root  64M Oct 25  2022 page.44
-rw-rw-r-- 1 root root   34 Oct 25  2022 checkpoint.44
-rw-rw-r-- 1 root root  64M Oct 25  2022 page.45
-rw-rw-r-- 1 root root   34 Oct 28  2022 checkpoint.45
-rw-rw-r-- 1 root root  64M Oct 28  2022 page.46
-rw-rw-r-- 1 root root   34 Oct 29  2022 checkpoint.46
-rw-rw-r-- 1 root root  64M Oct 29  2022 page.47
-rw-rw-r-- 1 root root   34 Oct 31  2022 checkpoint.47
-rw-rw-r-- 1 root root  64M Oct 31  2022 page.48
-rw-rw-r-- 1 root root   34 Nov  2  2022 checkpoint.48
-rw-rw-r-- 1 root root  64M Nov  2  2022 page.49
-rw-rw-r-- 1 root root   34 Nov  3  2022 checkpoint.49
-rw-rw-r-- 1 root root  64M Nov  3  2022 page.50
-rw-rw-r-- 1 root root   34 Nov  5  2022 checkpoint.50
-rw-rw-r-- 1 root root  64M Nov  5  2022 page.51
-rw-rw-r-- 1 root root   34 Nov  7  2022 checkpoint.51
-rw-rw-r-- 1 root root  64M Nov  7  2022 page.52
-rw-rw-r-- 1 root root   34 Nov  8  2022 checkpoint.52
-rw-rw-r-- 1 root root  64M Nov  8  2022 page.53
-rw-rw-r-- 1 root root   34 Nov  8  2022 checkpoint.53
-rw-rw-r-- 1 root root  64M Nov  8  2022 page.54
-rw-rw-r-- 1 root root   34 Nov  9  2022 checkpoint.54
-rw-rw-r-- 1 root root  64M Nov  9  2022 page.55
-rw-rw-r-- 1 root root   34 Nov 10  2022 checkpoint.55
-rw-rw-r-- 1 root root  64M Nov 10  2022 page.56
-rw-rw-r-- 1 root root   34 Nov 10  2022 checkpoint.56
-rw-rw-r-- 1 root root  64M Nov 10  2022 page.57
-rw-rw-r-- 1 root root   34 Nov 10  2022 checkpoint.57
-rw-rw-r-- 1 root root  64M Nov 10  2022 page.58
-rw-rw-r-- 1 root root   34 Nov 11  2022 checkpoint.58
-rw-rw-r-- 1 root root  64M Nov 11  2022 page.59
-rw-rw-r-- 1 root root   34 Nov 12  2022 checkpoint.59
-rw-rw-r-- 1 root root  64M Nov 12  2022 page.60
-rw-rw-r-- 1 root root   34 Nov 12  2022 checkpoint.60
-rw-rw-r-- 1 root root  64M Nov 12  2022 page.61
-rw-rw-r-- 1 root root   34 Nov 15  2022 checkpoint.61
-rw-rw-r-- 1 root root  64M Nov 15  2022 page.62
-rw-rw-r-- 1 root root   34 Nov 15  2022 checkpoint.62
-rw-rw-r-- 1 root root  64M Nov 15  2022 page.63
-rw-rw-r-- 1 root root   34 Nov 16  2022 checkpoint.63
-rw-rw-r-- 1 root root  64M Nov 16  2022 page.64
-rw-rw-r-- 1 root root   34 Nov 16  2022 checkpoint.64
-rw-rw-r-- 1 root root  64M Nov 16  2022 page.65
-rw-rw-r-- 1 root root   34 Nov 16  2022 checkpoint.65
-rw-rw-r-- 1 root root  64M Nov 16  2022 page.66
-rw-rw-r-- 1 root root   34 Nov 17  2022 checkpoint.66
-rw-rw-r-- 1 root root  64M Nov 17  2022 page.67
-rw-rw-r-- 1 root root   34 Nov 18  2022 checkpoint.67
-rw-rw-r-- 1 root root  64M Nov 18  2022 page.68
-rw-rw-r-- 1 root root   34 Nov 18  2022 checkpoint.68
-rw-rw-r-- 1 root root  64M Nov 18  2022 page.69
-rw-rw-r-- 1 root root   34 Nov 18  2022 checkpoint.69
-rw-rw-r-- 1 root root  64M Nov 18  2022 page.70
-rw-rw-r-- 1 root root   34 Nov 18  2022 checkpoint.70
-rw-rw-r-- 1 root root  64M Nov 18  2022 page.71
-rw-rw-r-- 1 root root   34 Nov 19  2022 checkpoint.71
-rw-rw-r-- 1 root root  64M Nov 19  2022 page.72
-rw-rw-r-- 1 root root   34 Nov 20  2022 checkpoint.72
-rw-rw-r-- 1 root root  64M Nov 20  2022 page.73
-rw-rw-r-- 1 root root   34 Nov 22  2022 checkpoint.73
-rw-rw-r-- 1 root root  64M Nov 22  2022 page.74
-rw-rw-r-- 1 root root   34 Nov 22  2022 checkpoint.74
-rw-rw-r-- 1 root root  64M Nov 22  2022 page.75
-rw-rw-r-- 1 root root   34 Nov 22  2022 checkpoint.75
-rw-rw-r-- 1 root root  64M Nov 22  2022 page.76
-rw-rw-r-- 1 root root   34 Nov 22  2022 checkpoint.76
-rw-rw-r-- 1 root root  64M Nov 22  2022 page.77
-rw-rw-r-- 1 root root   34 Nov 23  2022 checkpoint.77
-rw-rw-r-- 1 root root  64M Nov 23  2022 page.78
-rw-rw-r-- 1 root root   34 Nov 24  2022 checkpoint.78
-rw-rw-r-- 1 root root  64M Nov 24  2022 page.79
-rw-rw-r-- 1 root root   34 Nov 25  2022 checkpoint.79
-rw-rw-r-- 1 root root  64M Nov 25  2022 page.80
-rw-rw-r-- 1 root root   34 Nov 25  2022 checkpoint.80
-rw-rw-r-- 1 root root  64M Nov 25  2022 page.81
-rw-rw-r-- 1 root root   34 Nov 25  2022 checkpoint.81
-rw-rw-r-- 1 root root  64M Nov 25  2022 page.82
-rw-rw-r-- 1 root root   34 Nov 26  2022 checkpoint.82
-rw-rw-r-- 1 root root  64M Nov 26  2022 page.83
-rw-rw-r-- 1 root root   34 Nov 26  2022 checkpoint.83
-rw-rw-r-- 1 root root  64M Nov 26  2022 page.84
-rw-rw-r-- 1 root root   34 Nov 26  2022 checkpoint.84
-rw-rw-r-- 1 root root  64M Nov 26  2022 page.85
-rw-rw-r-- 1 root root   34 Nov 26  2022 checkpoint.85
-rw-rw-r-- 1 root root  64M Nov 26  2022 page.86
-rw-rw-r-- 1 root root   34 Nov 27  2022 checkpoint.86
-rw-rw-r-- 1 root root  64M Nov 27  2022 page.87
-rw-rw-r-- 1 root root   34 Nov 27 20:55 checkpoint.87
-rw-rw-r-- 1 root root  64M Nov 27 20:55 page.88
-rw-rw-r-- 1 root root   34 Nov 28 01:13 checkpoint.88
-rw-rw-r-- 1 root root  64M Nov 28 01:13 page.89
-rw-rw-r-- 1 root root   34 Nov 28 11:34 checkpoint.89
-rw-rw-r-- 1 root root  64M Nov 28 11:34 page.90
-rw-rw-r-- 1 root root   34 Nov 28 13:38 checkpoint.90
-rw-rw-r-- 1 root root  64M Nov 28 13:38 page.91
-rw-rw-r-- 1 root root   34 Nov 29 01:33 checkpoint.91
-rw-rw-r-- 1 root root  64M Nov 29 01:33 page.92
-rw-rw-r-- 1 root root   34 Nov 29 02:47 checkpoint.92
-rw-rw-r-- 1 root root  64M Nov 29 02:47 page.93
-rw-rw-r-- 1 root root   34 Nov 29 06:45 checkpoint.93
-rw-rw-r-- 1 root root  64M Nov 29 06:45 page.94
-rw-rw-r-- 1 root root   34 Nov 29 12:04 checkpoint.94
-rw-rw-r-- 1 root root  64M Nov 29 12:04 page.95
-rw-rw-r-- 1 root root   34 Nov 30 07:37 checkpoint.95
-rw-rw-r-- 1 root root  64M Nov 30 07:37 page.96
-rw-rw-r-- 1 root root   34 Nov 30 11:55 checkpoint.96
-rw-rw-r-- 1 root root  64M Nov 30 11:55 page.97
-rw-rw-r-- 1 root root   34 Nov 30 13:39 checkpoint.97
-rw-rw-r-- 1 root root  64M Nov 30 13:39 page.98
-rw-rw-r-- 1 root root   34 Dec  1 03:39 checkpoint.98
-rw-rw-r-- 1 root root  64M Dec  1 03:39 page.99
-rw-rw-r-- 1 root root   34 Dec  1 14:09 checkpoint.99
-rw-rw-r-- 1 root root  64M Dec  1 14:09 page.100
-rw-rw-r-- 1 root root   34 Dec  2 03:29 checkpoint.100
-rw-rw-r-- 1 root root  64M Dec  2 03:29 page.101
-rw-rw-r-- 1 root root   34 Dec  2 10:52 checkpoint.101
-rw-rw-r-- 1 root root  64M Dec  2 10:52 page.102
-rw-rw-r-- 1 root root   34 Dec  3 09:37 checkpoint.102
-rw-rw-r-- 1 root root  64M Dec  3 09:37 page.103
-rw-rw-r-- 1 root root   34 Dec  4 11:19 checkpoint.103
-rw-rw-r-- 1 root root  64M Dec  4 11:19 page.104
-rw-rw-r-- 1 root root   34 Dec  5 10:26 checkpoint.104
-rw-rw-r-- 1 root root  64M Dec  5 10:26 page.105
-rw-rw-r-- 1 root root   34 Dec  5 21:11 checkpoint.105
-rw-rw-r-- 1 root root  64M Dec  5 21:11 page.106
-rw-rw-r-- 1 root root   34 Dec  5 23:56 checkpoint.106
-rw-rw-r-- 1 root root  64M Dec  5 23:56 page.107
-rw-rw-r-- 1 root root   34 Dec  6 01:09 checkpoint.107
-rw-rw-r-- 1 root root  64M Dec  6 01:09 page.108
-rw-rw-r-- 1 root root   34 Dec  6 01:29 checkpoint.108
-rw-rw-r-- 1 root root  64M Dec  6 01:29 page.109
-rw-rw-r-- 1 root root   34 Dec  6 13:54 checkpoint.109
-rw-rw-r-- 1 root root  64M Dec  6 13:54 page.110
-rw-rw-r-- 1 root root   34 Dec  6 19:34 checkpoint.110
-rw-rw-r-- 1 root root  64M Dec  6 19:34 page.111
-rw-rw-r-- 1 root root   34 Dec  7 05:14 checkpoint.111
-rw-rw-r-- 1 root root  64M Dec  7 05:14 page.112
-rw-rw-r-- 1 root root   34 Dec  7 11:59 checkpoint.112
-rw-rw-r-- 1 root root  64M Dec  7 11:59 page.113
-rw-rw-r-- 1 root root   34 Dec  7 18:29 checkpoint.113
-rw-rw-r-- 1 root root  64M Dec  7 18:29 page.114
-rw-r--r-- 1 root root   34 Dec  7 20:38 checkpoint.114
-rw-r--r-- 1 root root  64M Dec  7 20:38 page.115
-rw-r--r-- 1 root root   34 Jan 23 03:26 checkpoint.115
-rw-r--r-- 1 root root  64M Jan 23 03:26 page.116
-rw-r--r-- 1 root root   34 Jan 26 00:20 checkpoint.116
-rw-r--r-- 1 root root  64M Jan 26 00:20 page.117
-rw-r--r-- 1 root root   34 Jan 28 23:47 checkpoint.117
-rw-r--r-- 1 root root  64M Jan 28 23:47 page.118
-rw-r--r-- 1 root root   34 Jan 30 14:40 checkpoint.118
-rw-r--r-- 1 root root  64M Jan 30 14:40 page.119
-rw-r--r-- 1 root root   34 Jan 31 10:03 checkpoint.119
-rw-r--r-- 1 root root  64M Jan 31 10:03 page.120
-rw-r--r-- 1 root root   34 Feb  3 17:27 checkpoint.120
-rw-r--r-- 1 root root  64M Feb  3 17:27 page.121
-rw-r--r-- 1 root root   34 Feb  4 09:45 checkpoint.121
-rw-r--r-- 1 root root  64M Feb  4 09:45 page.122
-rw-r--r-- 1 root root   34 Feb  4 18:44 checkpoint.122
-rw-r--r-- 1 root root  64M Feb  4 18:44 page.123
-rw-r--r-- 1 root root   34 Feb  8 01:50 checkpoint.123
-rw-r--r-- 1 root root  64M Feb  8 01:50 page.124
-rw-r--r-- 1 root root   34 Feb  8 07:41 checkpoint.124
-rw-r--r-- 1 root root  64M Feb  8 07:41 page.125
-rw-r--r-- 1 root root   34 Feb  8 14:46 checkpoint.125
-rw-r--r-- 1 root root  64M Feb  8 14:46 page.126
-rw-r--r-- 1 root root   34 Feb  9 08:27 checkpoint.126
-rw-r--r-- 1 root root  64M Feb  9 08:27 page.127
-rw-r--r-- 1 root root   34 Feb  9 17:56 checkpoint.127
-rw-r--r-- 1 root root  64M Feb  9 17:56 page.128
-rw-r--r-- 1 root root   34 Feb 10 09:11 checkpoint.128
-rw-r--r-- 1 root root  64M Feb 10 09:11 page.129
-rw-r--r-- 1 root root   34 Feb 14 12:00 checkpoint.129
-rw-r--r-- 1 root root  64M Feb 14 12:00 page.130
-rw-r--r-- 1 root root   34 Feb 16 12:55 checkpoint.130
-rw-r--r-- 1 root root  64M Feb 16 12:55 page.131
-rw-r--r-- 1 root root   34 Feb 16 17:36 checkpoint.131
-rw-r--r-- 1 root root  64M Feb 16 17:36 page.132
-rw-r--r-- 1 root root   34 Feb 16 18:59 checkpoint.132
-rw-r--r-- 1 root root  64M Feb 16 18:59 page.133
-rw-r--r-- 1 root root   34 Feb 17 08:59 checkpoint.133
-rw-r--r-- 1 root root  64M Feb 17 08:59 page.134
-rw-r--r-- 1 root root   34 Feb 19 12:30 checkpoint.134
-rw-r--r-- 1 root root  64M Feb 19 12:30 page.135
-rw-r--r-- 1 root root   34 Feb 20 01:19 checkpoint.135
-rw-r--r-- 1 root root  64M Feb 20 01:19 page.136
-rw-r--r-- 1 root root   34 Feb 20 17:55 checkpoint.136
-rw-r--r-- 1 root root  64M Feb 20 17:55 page.137
-rw-r--r-- 1 root root   34 Feb 21 00:49 checkpoint.137
-rw-r--r-- 1 root root  64M Feb 21 00:49 page.138
-rw-r--r-- 1 root root   34 Feb 22 13:19 checkpoint.138
-rw-r--r-- 1 root root  64M Feb 22 13:19 page.139
-rw-r--r-- 1 root root   34 Feb 23 15:38 checkpoint.139
-rw-r--r-- 1 root root  64M Feb 23 15:38 page.140
-rw-r--r-- 1 root root   34 Feb 24 18:30 checkpoint.140
-rw-r--r-- 1 root root  64M Feb 24 18:30 page.141
-rw-r--r-- 1 root root   34 Feb 27 12:33 checkpoint.141
-rw-r--r-- 1 root root  64M Feb 27 12:33 page.142
-rw-r--r-- 1 root root   34 Feb 27 20:31 checkpoint.142
-rw-r--r-- 1 root root  64M Feb 27 20:31 page.143
-rw-r--r-- 1 root root   34 Feb 28 11:44 checkpoint.143
-rw-r--r-- 1 root root  64M Feb 28 11:44 page.144
-rw-r--r-- 1 root root   34 Mar  2 07:40 checkpoint.144
-rw-r--r-- 1 root root  64M Mar  2 07:40 page.145
-rw-r--r-- 1 root root   34 Mar  2 20:49 checkpoint.145
-rw-r--r-- 1 root root  64M Mar  2 20:49 page.146
-rw-r--r-- 1 root root   34 Mar  3 07:53 checkpoint.146
-rw-r--r-- 1 root root  64M Mar  3 07:53 page.147
-rw-r--r-- 1 root root   34 Mar  3 14:38 checkpoint.147
-rw-r--r-- 1 root root  64M Mar  3 14:38 page.148
-rw-r--r-- 1 root root   34 Mar  4 01:45 checkpoint.148
-rw-r--r-- 1 root root  64M Mar  4 01:45 page.149
-rw-r--r-- 1 root root   34 Mar  4 17:40 checkpoint.149
-rw-r--r-- 1 root root  64M Mar  4 17:40 page.150
-rw-r--r-- 1 root root   34 Mar  5 10:06 checkpoint.150
-rw-r--r-- 1 root root  64M Mar  5 10:06 page.151
-rw-r--r-- 1 root root   34 Mar  5 11:39 checkpoint.151
-rw-r--r-- 1 root root  64M Mar  5 11:39 page.152
-rw-r--r-- 1 root root   34 Mar  6 08:15 checkpoint.152
-rw-r--r-- 1 root root  64M Mar  6 08:15 page.153
-rw-r--r-- 1 root root   34 Mar  6 16:43 checkpoint.153
-rw-r--r-- 1 root root  64M Mar  6 16:43 page.154
-rw-r--r-- 1 root root   34 Mar  6 18:47 checkpoint.154
-rw-r--r-- 1 root root  64M Mar  6 18:47 page.155
-rw-r--r-- 1 root root   34 Mar  7 07:46 checkpoint.155
-rw-r--r-- 1 root root  64M Mar  7 07:46 page.156
-rw-r--r-- 1 root root   34 Mar  7 13:27 checkpoint.156
-rw-r--r-- 1 root root  64M Mar  7 13:27 page.157
-rw-r--r-- 1 root root   34 Mar  8 08:28 checkpoint.157
-rw-r--r-- 1 root root  64M Mar  8 08:28 page.158
-rw-r--r-- 1 root root   34 Mar  8 17:59 checkpoint.158
-rw-r--r-- 1 root root  64M Mar  8 17:59 page.159
-rw-r--r-- 1 root root   34 Mar  9 07:58 checkpoint.159
-rw-r--r-- 1 root root  64M Mar  9 07:58 page.160
-rw-r--r-- 1 root root   34 Mar  9 22:00 checkpoint.160
-rw-r--r-- 1 root root  64M Mar  9 22:00 page.161
-rw-r--r-- 1 root root   34 Mar 10 22:42 checkpoint.161
-rw-r--r-- 1 root root  64M Mar 10 22:42 page.162
-rw-r--r-- 1 root root   34 Mar 11 02:09 checkpoint.162
-rw-r--r-- 1 root root  64M Mar 11 02:09 page.163
-rw-r--r-- 1 root root   34 Mar 11 02:29 checkpoint.163
-rw-r--r-- 1 root root  64M Mar 11 02:29 page.164
-rw-r--r-- 1 root root   34 Mar 11 09:24 checkpoint.164
-rw-r--r-- 1 root root  64M Mar 11 09:24 page.165
-rw-r--r-- 1 root root   34 Mar 11 20:49 checkpoint.165
-rw-r--r-- 1 root root  64M Mar 11 20:49 page.166
-rw-r--r-- 1 root root   34 Mar 12 07:52 checkpoint.166
-rw-r--r-- 1 root root  64M Mar 12 07:52 page.167
-rw-r--r-- 1 root root   34 Mar 12 16:42 checkpoint.167
-rw-r--r-- 1 root root  64M Mar 12 16:42 page.168
-rw-r--r-- 1 root root   34 Mar 12 19:19 checkpoint.168
-rw-r--r-- 1 root root  64M Mar 12 19:19 page.169
-rw-r--r-- 1 root root   34 Mar 13 09:51 checkpoint.169
-rw-r--r-- 1 root root  64M Mar 13 09:51 page.170
-rw-r--r-- 1 root root   34 Mar 13 22:29 checkpoint.170
-rw-r--r-- 1 root root  64M Mar 13 22:29 page.171
-rw-r--r-- 1 root root   34 Mar 14 00:02 checkpoint.171
-rw-r--r-- 1 root root  64M Mar 14 00:02 page.172
-rw-r--r-- 1 root root   34 Mar 14 03:30 checkpoint.172
-rw-r--r-- 1 root root  64M Mar 14 03:30 page.173
-rw-r--r-- 1 root root   34 Mar 14 10:57 checkpoint.173
-rw-r--r-- 1 root root  64M Mar 14 10:57 page.174
-rw-r--r-- 1 root root   34 Mar 16 22:16 checkpoint.174
-rw-r--r-- 1 root root  64M Mar 16 22:16 page.175
-rw-r--r-- 1 root root   34 Mar 18 06:47 checkpoint.175
-rw-r--r-- 1 root root  64M Mar 18 06:47 page.176
-rw-r--r-- 1 root root   34 Mar 18 09:14 checkpoint.176
-rw-r--r-- 1 root root  64M Mar 18 09:14 page.177
-rw-r--r-- 1 root root   34 Mar 19 04:24 checkpoint.177
-rw-r--r-- 1 root root  64M Mar 19 04:24 page.178
-rw-r--r-- 1 root root   34 Mar 19 14:48 checkpoint.178
-rw-r--r-- 1 root root  64M Mar 19 14:48 page.179
-rw-r--r-- 1 root root   34 Mar 19 15:19 checkpoint.179
-rw-r--r-- 1 root root  64M Mar 19 15:19 page.180
-rw-r--r-- 1 root root   34 Mar 20 05:38 checkpoint.180
-rw-r--r-- 1 root root  64M Mar 20 05:38 page.181
-rw-r--r-- 1 root root   34 Mar 21 13:07 checkpoint.181
-rw-r--r-- 1 root root  64M Mar 21 13:07 page.182
-rw-r--r-- 1 root root   34 Mar 21 21:57 checkpoint.182
-rw-r--r-- 1 root root  64M Mar 21 21:57 page.183
-rw-r--r-- 1 root root   34 Mar 22 22:00 checkpoint.183
-rw-r--r-- 1 root root  64M Mar 22 22:00 page.184
-rw-r--r-- 1 root root   34 Mar 22 23:24 checkpoint.184
-rw-r--r-- 1 root root  64M Mar 22 23:24 page.185
-rw-r--r-- 1 root root   34 Mar 23 11:09 checkpoint.185
-rw-r--r-- 1 root root  64M Mar 23 11:09 page.186
-rw-r--r-- 1 root root   34 Mar 23 16:00 checkpoint.186
-rw-r--r-- 1 root root  64M Mar 23 16:00 page.187
-rw-r--r-- 1 root root   34 Mar 23 22:46 checkpoint.187
-rw-r--r-- 1 root root  64M Mar 23 22:46 page.188
-rw-r--r-- 1 root root   34 Mar 24 04:40 checkpoint.188
-rw-r--r-- 1 root root  64M Mar 24 04:40 page.189
-rw-r--r-- 1 root root   34 Mar 24 15:22 checkpoint.189
-rw-r--r-- 1 root root  64M Mar 24 15:22 page.190
-rw-r--r-- 1 root root   34 Mar 25 16:37 checkpoint.190
-rw-r--r-- 1 root root  64M Mar 25 16:37 page.191
-rw-r--r-- 1 root root   34 Mar 25 17:50 checkpoint.191
-rw-r--r-- 1 root root  64M Mar 25 17:50 page.192
-rw-r--r-- 1 root root   34 Mar 25 22:40 checkpoint.192
-rw-r--r-- 1 root root  64M Mar 25 22:40 page.193
-rw-r--r-- 1 root root   34 Mar 25 22:51 checkpoint.193
-rw-r--r-- 1 root root  64M Mar 25 22:51 page.194
-rw-r--r-- 1 root root   34 Mar 26 08:42 checkpoint.194
-rw-r--r-- 1 root root  64M Mar 26 08:42 page.195
-rw-r--r-- 1 root root   34 Mar 26 15:37 checkpoint.195
-rw-r--r-- 1 root root  64M Mar 26 15:37 page.196
-rw-r--r-- 1 root root   34 Mar 27 09:48 checkpoint.196
-rw-r--r-- 1 root root  64M Mar 27 09:48 page.197
-rw-r--r-- 1 root root   34 Mar 28 02:22 checkpoint.197
-rw-r--r-- 1 root root  64M Mar 28 02:22 page.198
-rw-r--r-- 1 root root   34 Mar 28 13:47 checkpoint.198
-rw-r--r-- 1 root root  64M Mar 28 13:47 page.199
-rw-r--r-- 1 root root   34 Mar 29 16:39 checkpoint.199
-rw-r--r-- 1 root root  64M Mar 29 16:39 page.200
-rw-r--r-- 1 root root   34 Mar 30 03:44 checkpoint.200
-rw-r--r-- 1 root root  64M Mar 30 03:44 page.201
-rw-r--r-- 1 root root   34 Mar 30 04:25 checkpoint.201
-rw-r--r-- 1 root root  64M Mar 30 04:25 page.202
-rw-r--r-- 1 root root   34 Mar 31 00:38 checkpoint.202
-rw-r--r-- 1 root root  64M Mar 31 00:38 page.203
-rw-r--r-- 1 root root   34 Mar 31 09:09 checkpoint.203
-rw-r--r-- 1 root root  64M Mar 31 09:09 page.204
-rw-r--r-- 1 root root   34 Apr  1 16:25 checkpoint.204
-rw-r--r-- 1 root root  64M Apr  1 16:25 page.205
-rw-r--r-- 1 root root   34 Apr  2 05:54 checkpoint.205
-rw-r--r-- 1 root root  64M Apr  2 05:54 page.206
-rw-r--r-- 1 root root   34 Apr  2 11:25 checkpoint.206
-rw-r--r-- 1 root root  64M Apr  2 11:25 page.207
-rw-r--r-- 1 root root   34 Apr  5 07:02 checkpoint.207
-rw-r--r-- 1 root root  64M Apr  5 07:02 page.208
-rw-r--r-- 1 root root   34 Apr  5 08:36 checkpoint.208
-rw-r--r-- 1 root root  64M Apr  5 08:36 page.209
-rw-r--r-- 1 root root   34 Apr  6 02:55 checkpoint.209
-rw-r--r-- 1 root root  64M Apr  6 02:55 page.210
-rw-r--r-- 1 root root   34 Apr  6 09:50 checkpoint.210
-rw-r--r-- 1 root root  64M Apr  6 09:50 page.211
-rw-r--r-- 1 root root   34 Apr  6 17:17 checkpoint.211
-rw-r--r-- 1 root root  64M Apr  6 17:17 page.212
-rw-r--r-- 1 root root   34 Apr  7 01:56 checkpoint.212
-rw-r--r-- 1 root root  64M Apr  7 01:56 page.213
-rw-r--r-- 1 root root   34 Apr  7 05:41 checkpoint.213
-rw-r--r-- 1 root root  64M Apr  7 05:41 page.214
-rw-r--r-- 1 root root   34 Apr  7 13:41 checkpoint.214
-rw-r--r-- 1 root root  64M Apr  7 13:41 page.215
-rw-r--r-- 1 root root   34 Apr  8 08:54 checkpoint.215
-rw-r--r-- 1 root root  64M Apr  8 08:54 page.216
-rw-r--r-- 1 root root   34 Apr  8 10:27 checkpoint.216
-rw-r--r-- 1 root root  64M Apr  8 10:27 page.217
-rw-r--r-- 1 root root   34 Apr  8 18:45 checkpoint.217
-rw-r--r-- 1 root root  64M Apr  8 18:45 page.218
-rw-r--r-- 1 root root   34 Apr  9 01:09 checkpoint.218
-rw-r--r-- 1 root root  64M Apr  9 01:09 page.219
-rw-r--r-- 1 root root   34 Apr  9 10:21 checkpoint.219
-rw-r--r-- 1 root root  64M Apr  9 10:21 page.220
-rw-r--r-- 1 root root   34 Apr 11 07:08 checkpoint.220
-rw-r--r-- 1 root root  64M Apr 11 07:08 page.221
-rw-r--r-- 1 root root   34 Apr 11 11:29 checkpoint.221
-rw-r--r-- 1 root root  64M Apr 11 11:29 page.222
-rw-r--r-- 1 root root   34 Apr 12 15:38 checkpoint.222
-rw-r--r-- 1 root root  64M Apr 12 15:38 page.223
-rw-r--r-- 1 root root   34 Apr 12 20:39 checkpoint.223
-rw-r--r-- 1 root root  64M Apr 12 20:39 page.224
-rw-r--r-- 1 root root   34 Apr 13 05:08 checkpoint.224
-rw-r--r-- 1 root root  64M Apr 13 05:08 page.225
-rw-r--r-- 1 root root   34 Apr 13 14:08 checkpoint.225
-rw-r--r-- 1 root root  64M Apr 13 14:08 page.226
-rw-r--r-- 1 root root   34 Apr 14 17:28 checkpoint.226
-rw-r--r-- 1 root root  64M Apr 14 17:28 page.227
-rw-r--r-- 1 root root   34 Apr 15 12:51 checkpoint.227
-rw-r--r-- 1 root root  64M Apr 15 12:51 page.228
-rw-r--r-- 1 root root   34 Apr 15 17:01 checkpoint.228
-rw-r--r-- 1 root root  64M Apr 15 17:01 page.229
-rw-r--r-- 1 root root   34 Apr 16 23:14 checkpoint.229
-rw-r--r-- 1 root root  64M Apr 16 23:14 page.230
-rw-r--r-- 1 root root   34 Apr 17 02:52 checkpoint.230
-rw-r--r-- 1 root root  64M Apr 17 02:52 page.231
-rw-r--r-- 1 root root   34 Apr 17 06:19 checkpoint.231
-rw-r--r-- 1 root root  64M Apr 17 06:19 page.232
-rw-r--r-- 1 root root   34 Apr 17 22:34 checkpoint.232
-rw-r--r-- 1 root root  64M Apr 17 22:34 page.233
-rw-r--r-- 1 root root   34 Apr 18 01:41 checkpoint.233
-rw-r--r-- 1 root root  64M Apr 18 01:41 page.234
-rw-r--r-- 1 root root   34 Apr 18 07:46 checkpoint.234
-rw-r--r-- 1 root root  64M Apr 18 07:46 page.235
-rw-r--r-- 1 root root   34 Apr 18 09:17 checkpoint.235
-rw-r--r-- 1 root root  64M Apr 18 09:17 page.236
-rw-r--r-- 1 root root   34 Apr 18 09:17 checkpoint.head
drwxrwsr-x 2 root root  20K Apr 18 09:17 .

oltausr@localhost:~$ sudo du -sh  /mnt/onl/sdpon/data/logstash/data/
15G     /mnt/onl/sdpon/data/logstash/data/

leandrojmp · June 20, 2023, 4:24am

Do you have anything in Logstash logs? Is your data being indexed?

You do not have any recent page file, that last one seems to be from April, it looks like some left over data for some reason, but it is pretty hard to know why Logstash didn't cleaned the path without seeing any logs from Logstash that could give some hint.

Also, can you share how you are running Logstash on Kubernetes? The configurations you are using to mount the path for example.

Sindhu_Bandi · June 20, 2023, 5:19am

Part of logs from logstash pod

[2023-05-29T09:57:23,851][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:24,853][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:25,856][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:26,859][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:27,861][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:28,864][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:29,866][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:30,869][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:31,871][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:32,873][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:33,876][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:34,878][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:35,880][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:36,883][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:37,885][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:38,888][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:39,890][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:40,893][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:41,895][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}
[2023-05-29T09:57:42,898][WARN ][logstash.outputs.syslog  ][main][bc9aaa7c7f9af05a5f1ec8520251eb5202be6a01a5fbc75766ccf7e1de5a4108] syslog tcp output exception: closing, reconnecting and resending event {:host=>"192.168.56.151", :port=>514, :exception=>#<Errno::ECONNREFUSED: Connection refused - connect(2) for "192.168.56.151" port 514>, :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'", "org/jruby/RubyIO.java:876:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:209:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:177:in `publish'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.1.0/lib/logstash/codecs/plain.rb:59:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-syslog-3.0.5/lib/logstash/outputs/syslog.rb:147:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299:in `block in start_workers'"], :event=>#<LogStash::Event:0x6e333a20>}

Logstash logs were with error loglevel.
Logstash is running on kubernetes 1.21.5 with these configurations
volume path on host : /mnt/onl/sdpon/data/logstash/data
Data Mount path inside Pod: /usr/share/logstash/data
logstash.yml

http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
xpack.monitoring.enabled: false
queue.type: persisted
queue.max_bytes: 2gb

logstash.conf

Data
====
logstash.conf:
----
input {
  kafka {
    bootstrap_servers => "kafka.default.svc.cluster.local:9092"
    consumer_threads => 1
    auto_offset_reset => "earliest"

        topics => ["list of topics to be relayed"]
    decorate_events => true
  }
}

filter {
  mutate {
    remove_field => [ "@version","source","offset","topics","beat","input","host","prospector","tags","[kubernetes][labels]","[kubernetes][node]","[kubernetes][namespace]","[kubernetes][container]","[kubernetes][replicaset]","stream","log" ]
  }
}

output {
  if [@metadata][kafka][topic] == "topicname1" {
  syslog {
      sourcehost => "source IP"
      appname => "%{[@metadata][kafka][topic]}"
      host => "192.168.56.151"
      port => 514
      protocol => ["tcp"]
      rfc => "rfc5424"
      enable_metric => false
      facility => "local7"
      codec => plain { format => "%{message}" }
     }
   } else if [@metadata][kafka][topic] == "topicname2" {
     syslog {
      sourcehost => "source IP"
      appname => "%{[@metadata][kafka][topic]}"
      host => "192.168.56.151"
      port => 514
      protocol => ["tcp"]
      rfc => "rfc5424"
      enable_metric => false
      facility => "local7"
      codec => plain { format => "%{message}" }
     }
   } else {
     syslog {
      sourcehost => "source IP"
      appname => "%{[@metadata][kafka][topic]}"
      host => "192.168.56.151"
      port => 514
      protocol => ["tcp"]
      rfc => "rfc5424"
      enable_metric => false
      codec => plain { format => "%{message}" }
     }
   }
}

system · July 18, 2023, 5:20am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash persistent queue max capacity Logstash	1	234	October 28, 2021
Logstash persistent queue settings Logstash	7	1982	October 12, 2021
Persistent Queue Configuration question Logstash	13	5674	March 16, 2017
Logstash overfilling Persistent Queue? Logstash docker	4	1338	August 31, 2022
Not enough free disk space available to allocate persisted queue Logstash	8	3550	July 18, 2018

Logstash filling up disk space beyond queue.max_bytes

Related topics