Unhealthy - (DEGRADED) Applied policy - Failure enabling network events; current state is disabled

Elastic Security Endpoint Security
1

Elastic-agent v8.10.2 deployed on Fedora 37, has been running for months without issue. Noticed today that it's flagged as unhealthy. The same agent version and policy have been and are working fine on CentOS, Ubuntu etc. The issue also occurred on v8.10.1.

image
image1166×1678 96.8 KB

Some log entries from elastic_agent.endpoint_security [WARNING & ERROR] :

12:30:23.515
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: prog 'fentry__tty_write': failed to load: -22
12:30:23.515
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: failed to load object 'EventProbe_bpf'
12:30:23.515
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: failed to load BPF skeleton 'EventProbe_bpf': -22
12:30:25.574
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (r:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe tcp_sendpage rv=$retval)
12:30:25.574
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe)
12:30:25.583
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] rtnetlink replied: No such file or directory
12:30:25.583
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] error talking to the kernel (rtnetlink_send)
12:30:25.620
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] Config.cpp:2335 Policy failed to apply and rollback is disabled
12:30:25.622
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] AgentContext.cpp:264 Failed to apply new policy from Agent.
12:30:25.622
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] AgentContext.cpp:516 Endpoint is setting status to DEGRADED, reason: Policy Application Status
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_CLEANUP_RBUF_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_CLOSE_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_CLOSE_RET_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_V4_CONN_REQUEST_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_V4_CONN_REQUEST_RET_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_SEC_SOCK_POST_CREATE_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_SEC_SOCK_POST_CREATE_RET_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_INET_BIND_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_INET_BIND_HASH_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_INET_LISTEN_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_INET_LISTEN_RET_probe)
12:30:29.292
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_INET_BIND_BUCKET_CREATE_probe)
12:30:29.684
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] MessageHelpers.cpp:312 CURL error: SSL peer certificate or SSH remote key was not OK [SSL certificate problem: self-signed certificate in certificate chain]
12:30:30.496
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: prog 'fentry__tty_write': BPF program load failed: Invalid argument
12:30:30.496
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: prog 'fentry__tty_write': -- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function fentry__tty_write#581
0: R1=ctx(off=0,imm=0) R10=fp0
; int BPF_PROG(fentry__tty_write, struct kiocb *iocb, struct iov_iter *from)
0: (79) r2 = *(u64 *)(r1 +8)
func 'tty_write' arg1 has btf_id 888 type STRUCT 'iov_iter'
1: R1=ctx(off=0,imm=0) R2_w=ptr_iov_iter(off=0,imm=0)
1: (79) r1 = *(u64 *)(r1 +0)
func 'tty_write' arg0 has btf_id 789 type STRUCT 'kiocb'
2: R1_w=ptr_kiocb(off=0,imm=0)
; return tty_write__enter(iocb, from);
2: (85) call pc+2
reg type unsupported for arg#0 function tty_write__enter#1236
caller:
 R10=fp0
callee:
 frame1: R1_w=ptr_kiocb(off=0,imm=0) R2_w=ptr_iov_iter(off=0,imm=0) R10=fp0
5: frame1:
; static int tty_write__enter(struct kiocb *iocb, struct iov_iter *from)
5: (bf) r6 = r2                       ; frame1: R2_w=ptr_iov_iter(off=0,imm=0) R6_w=ptr_iov_iter(off=0,imm=0)
6: (bf) r7 = r1                       ; frame1: R1_w=ptr_kiocb(off=0,imm=0) R7_w=ptr_kiocb(off=0,imm=0)
; int pid = bpf_get_current_pid_tgid() >> 32;
7: (85) call bpf_get_current_pid_tgid#14      ; frame1: R0_w=scalar()
; return consumer_pid == pid;
8: (18) r1 = 0xffffa4ee800b6000       ; frame1: R1_w=map_value(off=0,ks=4,vs=25,imm=0)
10: (61) r1 = *(u32 *)(r1 +0)         ; frame1: R1_w=551353
; int pid = bpf_get_current_pid_tgid() >> 32;
11: (77) r0 >>= 32                    ; frame1: R0=scalar(umax=4294967295,var_off=(0x0; 0xffffffff))
; if (is_consumer()) {
12: (1d) if r1 == r0 goto pc+144      ; frame1: R0=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R1=551353
13: (b7) r1 = 0                       ; frame1: R1_w=0
14: (0f) r7 += r1                     ; frame1: R1_w=0 R7_w=ptr_kiocb(off=0,imm=0)
15: (bf) r1 = r10                     ; frame1: R1_w=fp0 R10=fp0
; 
16: (07) r1 += -8                     ; frame1: R1_w=fp-8
; struct file *f               = BPF_CORE_READ(iocb, ki_filp);
17: (b7) r2 = 8                       ; frame1: R2_w=8
18: (bf) r3 = r7                      ; frame1: R3_w=ptr_kiocb(off=0,imm=0) R7_w=ptr_kiocb(off=0,imm=0)
19: (85) call bpf_probe_read_kernel#113       ; frame1: R0=scalar() fp-8=mmmmmmmm
20: (b7) r1 = 200                     ; frame1: R1_w=200
; struct file *f               = BPF_CORE_READ(iocb, ki_filp);
21: (79) r3 = *(u64 *)(r10 -8)        ; frame1: R3_w=scalar() R10=fp0 fp-8=mmmmmmmm
22: (0f) r3 += r1                     ; frame1: R1_w=200 R3_w=scalar()
23: (bf) r1 = r10                     ; frame1: R1_w=fp0 R10=fp0
; 

[...]

306: (95) exit
returning from callee:
 frame2: R0=scalar() R1_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R2_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R3_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R4_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R5_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-32 R7=scalar(id=6) R8=scalar(id=3) R10=fp0 fp-8=mmmmmmmm fp-16= fp-24_w=mmmmmmmm fp-32_w=mmmmmmmm fp-40_w=mmmmmmmm fp-48_w=mmmmmmmm fp-56_w=mmmmmmmm fp-64=????mmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm
to caller at 85:
 frame1: R0=scalar() R6=ptr_iov_iter(off=0,imm=0) R7=scalar(id=1) R8=scalar() R9=0 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=00000000 fp-48=00000000 fp-56=00000000 fp-64=mm??????
; ebpf_tty_dev__fill(&master, tty);
85: (b7) r9 = 1                       ; frame1: R9_w=1
86: (bf) r7 = r8                      ; frame1: R7_w=scalar(id=11) R8=scalar(id=11)
87: (bf) r1 = r10                     ; frame1: R1_w=fp0 R10=fp0
; 
88: (07) r1 += -56                    ; frame1: R1_w=fp-56
89: (bf) r2 = r7                      ; frame1: R2_w=scalar(id=11) R7_w=scalar(id=11)
90: (85) call pc+80
caller:
 frame1: R6=ptr_iov_iter(off=0,imm=0) R7_w=scalar(id=11) R8=scalar(id=11) R9_w=1 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=00000000 fp-48=00000000 fp-56=00000000 fp-64=mm??????
callee:
 frame2: R1_w=fp-56 R2_w=scalar(id=11) R10=fp0
171: frame2:
; static void ebpf_tty_dev__fill(struct ebpf_tty_dev *tty_dev, const struct tty_struct *tty)
171: (bf) r7 = r2                     ; frame2: R2=scalar(id=11) R7_w=scalar(id=11)
172: (bf) r6 = r1                     ; frame2: R1=fp-56 R6_w=fp-56
173: (b7) r1 = 16                     ; frame2: R1_w=16
174: (bf) r8 = r7                     ; frame2: R7_w=scalar(id=11) R8_w=scalar(id=11)
175: (0f) r8 += r1                    ; frame2: R1_w=16 R8_w=scalar()
176: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
177: (07) r1 += -56                   ; frame2: R1_w=fp-56
; tty_dev->major = BPF_CORE_READ(tty, driver, major);
178: (b7) r2 = 8                      ; frame2: R2_w=8
179: (bf) r3 = r8                     ; frame2: R3_w=scalar(id=12) R8_w=scalar(id=12)
180: (85) call bpf_probe_read_kernel#113      ; frame2: R0_w=scalar() fp-56=mmmmmmmm
181: (b7) r1 = 44                     ; frame2: R1_w=44
182: (79) r3 = *(u64 *)(r10 -56)      ; frame2: R3_w=scalar() R10=fp0 fp-56=mmmmmmmm
183: (0f) r3 += r1                    ; frame2: R1_w=44 R3_w=scalar()
184: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
185: (07) r1 += -104                  ; frame2: R1_w=fp-104
; tty_dev->major = BPF_CORE_READ(tty, driver, major);
186: (b7) r2 = 4                      ; frame2: R2_w=4
187: (85) call bpf_probe_read_kernel#113      ; frame2: R0=scalar() fp-104=????mmmm
; tty_dev->major = BPF_CORE_READ(tty, driver, major);
188: (61) r1 = *(u32 *)(r10 -104)     ; frame2: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0 fp-104=????mmmm
; tty_dev->major = BPF_CORE_READ(tty, driver, major);
189: (bf) r2 = r1                     ; frame2: R1_w=scalar(id=13,umax=4294967295,var_off=(0x0; 0xffffffff)) R2_w=scalar(id=13,umax=4294967295,var_off=(0x0; 0xffffffff))
190: (77) r2 >>= 8                    ; frame2: R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff))
191: (73) *(u8 *)(r6 +3) = r2         ; frame2: R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R6=fp-56 fp-56=mmmmmmmm
192: (73) *(u8 *)(r6 +2) = r1         ; frame2: R1_w=scalar(id=13,umax=4294967295,var_off=(0x0; 0xffffffff)) R6=fp-56 fp-56=mmmmmmmm
193: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
194: (07) r1 += -56                   ; frame2: R1_w=fp-56
; tty_dev->minor = BPF_CORE_READ(tty, driver, minor_start);
195: (b7) r2 = 8                      ; frame2: R2_w=8
196: (bf) r3 = r8                     ; frame2: R3_w=scalar(id=12) R8=scalar(id=12)
197: (85) call bpf_probe_read_kernel#113      ; frame2: R0_w=scalar() fp-56=mmmmmmmm
198: (b7) r1 = 48                     ; frame2: R1_w=48
199: (79) r3 = *(u64 *)(r10 -56)      ; frame2: R3_w=scalar() R10=fp0 fp-56=mmmmmmmm
200: (0f) r3 += r1                    ; frame2: R1_w=48 R3_w=scalar()
201: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
202: (07) r1 += -104                  ; frame2: R1_w=fp-104
; tty_dev->minor = BPF_CORE_READ(tty, driver, minor_start);
203: (b7) r2 = 4                      ; frame2: R2_w=4
204: (85) call bpf_probe_read_kernel#113      ; frame2: R0=scalar() fp-104=????mmmm
; tty_dev->minor = BPF_CORE_READ(tty, driver, minor_start);
205: (61) r1 = *(u32 *)(r10 -104)     ; frame2: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0 fp-104=????mmmm
; tty_dev->minor = BPF_CORE_READ(tty, driver, minor_start);
206: (bf) r2 = r1                     ; frame2: R1_w=scalar(id=14,umax=4294967295,var_off=(0x0; 0xffffffff)) R2_w=scalar(id=14,umax=4294967295,var_off=(0x0; 0xffffffff))
207: (77) r2 >>= 8                    ; frame2: R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff))
208: (73) *(u8 *)(r6 +1) = r2         ; frame2: R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R6=fp-56 fp-56=mmmmmmmm
209: (73) *(u8 *)(r6 +0) = r1         ; frame2: R1_w=scalar(id=14,umax=4294967295,var_off=(0x0; 0xffffffff)) R6=fp-56 fp-56=mmmmmmmm
210: (b7) r1 = 32                     ; frame2: R1_w=32
211: (bf) r3 = r7                     ; frame2: R3_w=scalar(id=11) R7=scalar(id=11)
212: (0f) r3 += r1                    ; frame2: R1_w=32 R3_w=scalar()
213: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
214: (07) r1 += -56                   ; frame2: R1_w=fp-56
; tty_dev->minor += BPF_CORE_READ(tty, index);
215: (b7) r2 = 4                      ; frame2: R2_w=4
216: (85) call bpf_probe_read_kernel#113      ; frame2: R0_w=scalar() fp-56=mmmmmmmm
; tty_dev->minor += BPF_CORE_READ(tty, index);
217: (61) r1 = *(u32 *)(r10 -56)      ; frame2: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0 fp-56=mmmmmmmm
; tty_dev->minor += BPF_CORE_READ(tty, index);
218: (71) r2 = *(u8 *)(r6 +1)         ; frame2: R2_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-56=mmmmmmmm
219: (67) r2 <<= 8                    ; frame2: R2_w=scalar(umax=65280,var_off=(0x0; 0xff00))
220: (71) r3 = *(u8 *)(r6 +0)         ; frame2: R3_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-56=mmmmmmmm
221: (4f) r2 |= r3                    ; frame2: R2_w=scalar() R3_w=scalar(umax=255,var_off=(0x0; 0xff))
222: (0f) r2 += r1                    ; frame2: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R2_w=scalar()
223: (73) *(u8 *)(r6 +0) = r2         ; frame2: R2_w=scalar() R6=fp-56 fp-56=mmmmmmmm
224: (77) r2 >>= 8                    ; frame2: R2_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff))
225: (73) *(u8 *)(r6 +1) = r2         ; frame2: R2_w=scalar(umax=72057594037927935,var_off=(0x0; 0xffffffffffffff)) R6=fp-56 fp-56=mmmmmmmm
226: (b7) r1 = 428                    ; frame2: R1_w=428
227: (bf) r3 = r7                     ; frame2: R3_w=scalar(id=11) R7=scalar(id=11)
228: (0f) r3 += r1                    ; frame2: R1_w=428 R3_w=scalar()
229: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
230: (07) r1 += -56                   ; frame2: R1_w=fp-56
; struct winsize winsize     = BPF_CORE_READ(tty, winsize);
231: (b7) r2 = 8                      ; frame2: R2_w=8
232: (85) call bpf_probe_read_kernel#113      ; frame2: R0=scalar() fp-56=mmmmmmmm
; struct winsize winsize     = BPF_CORE_READ(tty, winsize);
233: (79) r1 = *(u64 *)(r10 -56)      ; frame2: R1_w=scalar() R10=fp0 fp-56=mmmmmmmm
234: (7b) *(u64 *)(r10 -8) = r1       ; frame2: R1_w=scalar() R10=fp0 fp-8_w=mmmmmmmm
235: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
236: (07) r1 += -8                    ; frame2: R1_w=fp-8
; ws.rows                    = winsize.ws_row;
237: (69) r2 = *(u16 *)(r1 +0)        ; frame2: R1_w=fp-8 R2_w=scalar(umax=65535,var_off=(0x0; 0xffff)) fp-8_w=mmmmmmmm
; ws.cols                    = winsize.ws_col;
238: (69) r1 = *(u16 *)(r1 +2)        ; frame2: R1_w=scalar(umax=65535,var_off=(0x0; 0xffff)) fp-8_w=mmmmmmmm
; tty_dev->winsize           = ws;
239: (73) *(u8 *)(r6 +6) = r1         ; frame2: R1_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-56 fp-56=mmmmmmmm
240: (77) r1 >>= 8                    ; frame2: R1_w=scalar(umax=255,var_off=(0x0; 0xff))
241: (73) *(u8 *)(r6 +7) = r1         ; frame2: R1_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-56=mmmmmmmm
242: (73) *(u8 *)(r6 +4) = r2         ; frame2: R2_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-56 fp-56=mmmmmmmm
243: (77) r2 >>= 8                    ; frame2: R2_w=scalar(umax=255,var_off=(0x0; 0xff))
244: (73) *(u8 *)(r6 +5) = r2         ; frame2: R2_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-56=mmmmmmmm
245: (b7) r1 = 264                    ; frame2: R1_w=264
246: (0f) r7 += r1                    ; frame2: R1_w=264 R7_w=scalar()
247: (bf) r1 = r10                    ; frame2: R1_w=fp0 R10=fp0
; 
248: (07) r1 += -104                  ; frame2: R1_w=fp-104
; struct ktermios termios   = BPF_CORE_READ(tty, termios);
249: (b7) r2 = 44                     ; frame2: R2_w=44
250: (bf) r3 = r7                     ; frame2: R3_w=scalar(id=15) R7_w=scalar(id=15)
251: (85) call bpf_probe_read_kernel#113      ; frame2: R0_w=scalar() fp-64=????mmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm
; struct ktermios termios   = BPF_CORE_READ(tty, termios);
252: (61) r1 = *(u32 *)(r10 -64)      ; frame2: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0 fp-64=????mmmm
253: (63) *(u32 *)(r10 -16) = r1      ; frame2: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0 fp-16=
254: (79) r1 = *(u64 *)(r10 -72)      ; frame2: R1_w=scalar() R10=fp0 fp-72=mmmmmmmm
255: (7b) *(u64 *)(r10 -24) = r1      ; frame2: R1_w=scalar() R10=fp0 fp-24_w=mmmmmmmm
256: (79) r1 = *(u64 *)(r10 -80)      ; frame2: R1_w=scalar() R10=fp0 fp-80=mmmmmmmm
257: (7b) *(u64 *)(r10 -32) = r1      ; frame2: R1_w=scalar() R10=fp0 fp-32_w=mmmmmmmm
258: (79) r1 = *(u64 *)(r10 -88)      ; frame2: R1_w=scalar() R10=fp0 fp-88=mmmmmmmm
259: (7b) *(u64 *)(r10 -40) = r1      ; frame2: R1_w=scalar() R10=fp0 fp-40_w=mmmmmmmm
260: (79) r1 = *(u64 *)(r10 -96)      ; frame2: R1_w=scalar() R10=fp0 fp-96=mmmmmmmm
261: (7b) *(u64 *)(r10 -48) = r1      ; frame2: R1_w=scalar() R10=fp0 fp-48_w=mmmmmmmm
262: (79) r1 = *(u64 *)(r10 -104)     ; frame2: R1_w=scalar() R10=fp0 fp-104=mmmmmmmm
263: (7b) *(u64 *)(r10 -56) = r1      ; frame2: R1_w=scalar() R10=fp0 fp-56_w=mmmmmmmm
264: (bf) r3 = r10                    ; frame2: R3_w=fp0 R10=fp0
; 
265: (07) r3 += -56                   ; frame2: R3_w=fp-56
; t.c_cflag                 = termios.c_cflag;
266: (61) r1 = *(u32 *)(r3 +8)        ; frame2: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R3_w=fp-56 fp-48_w=mmmmmmmm
; t.c_iflag                 = termios.c_iflag;
267: (61) r2 = *(u32 *)(r3 +0)        ; frame2: R2_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R3_w=fp-56 fp-56_w=mmmmmmmm
; t.c_oflag                 = termios.c_oflag;
268: (61) r4 = *(u32 *)(r3 +4)        ; frame2: R3_w=fp-56 R4_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) fp-56_w=mmmmmmmm
; t.c_lflag                 = termios.c_lflag;
269: (61) r3 = *(u32 *)(r3 +12)       ; frame2: R3_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) fp-48_w=mmmmmmmm
; tty_dev->termios          = t;
270: (bf) r5 = r3                     ; frame2: R3_w=scalar(id=16,umax=4294967295,var_off=(0x0; 0xffffffff)) R5_w=scalar(id=16,umax=4294967295,var_off=(0x0; 0xffffffff))
271: (77) r5 >>= 24                   ; frame2: R5_w=scalar(umax=255,var_off=(0x0; 0xff))
272: (73) *(u8 *)(r6 +19) = r5        ; frame2: R5_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-40_w=mmmmmmmm
273: (bf) r5 = r3                     ; frame2: R3_w=scalar(id=16,umax=4294967295,var_off=(0x0; 0xffffffff)) R5_w=scalar(id=16,umax=4294967295,var_off=(0x0; 0xffffffff))
274: (77) r5 >>= 16                   ; frame2: R5_w=scalar(umax=65535,var_off=(0x0; 0xffff))
275: (73) *(u8 *)(r6 +18) = r5        ; frame2: R5_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-56 fp-40_w=mmmmmmmm
276: (73) *(u8 *)(r6 +16) = r3        ; frame2: R3_w=scalar(id=16,umax=4294967295,var_off=(0x0; 0xffffffff)) R6=fp-56 fp-40_w=mmmmmmmm
277: (77) r3 >>= 8                    ; frame2: R3_w=scalar(umax=16777215,var_off=(0x0; 0xffffff))
278: (73) *(u8 *)(r6 +17) = r3        ; frame2: R3_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R6=fp-56 fp-40_w=mmmmmmmm
279: (bf) r3 = r4                     ; frame2: R3_w=scalar(id=17,umax=4294967295,var_off=(0x0; 0xffffffff)) R4_w=scalar(id=17,umax=4294967295,var_off=(0x0; 0xffffffff))
280: (77) r3 >>= 24                   ; frame2: R3_w=scalar(umax=255,var_off=(0x0; 0xff))
281: (73) *(u8 *)(r6 +15) = r3        ; frame2: R3_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-48_w=mmmmmmmm
282: (bf) r3 = r4                     ; frame2: R3_w=scalar(id=17,umax=4294967295,var_off=(0x0; 0xffffffff)) R4_w=scalar(id=17,umax=4294967295,var_off=(0x0; 0xffffffff))
283: (77) r3 >>= 16                   ; frame2: R3_w=scalar(umax=65535,var_off=(0x0; 0xffff))
284: (73) *(u8 *)(r6 +14) = r3        ; frame2: R3_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-56 fp-48_w=mmmmmmmm
285: (73) *(u8 *)(r6 +12) = r4        ; frame2: R4_w=scalar(id=17,umax=4294967295,var_off=(0x0; 0xffffffff)) R6=fp-56 fp-48_w=mmmmmmmm
286: (77) r4 >>= 8                    ; frame2: R4_w=scalar(umax=16777215,var_off=(0x0; 0xffffff))
287: (73) *(u8 *)(r6 +13) = r4        ; frame2: R4_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R6=fp-56 fp-48_w=mmmmmmmm
288: (bf) r3 = r2                     ; frame2: R2_w=scalar(id=18,umax=4294967295,var_off=(0x0; 0xffffffff)) R3_w=scalar(id=18,umax=4294967295,var_off=(0x0; 0xffffffff))
289: (77) r3 >>= 24                   ; frame2: R3_w=scalar(umax=255,var_off=(0x0; 0xff))
290: (73) *(u8 *)(r6 +11) = r3        ; frame2: R3_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-48_w=mmmmmmmm
291: (bf) r3 = r2                     ; frame2: R2_w=scalar(id=18,umax=4294967295,var_off=(0x0; 0xffffffff)) R3_w=scalar(id=18,umax=4294967295,var_off=(0x0; 0xffffffff))
292: (77) r3 >>= 16                   ; frame2: R3_w=scalar(umax=65535,var_off=(0x0; 0xffff))
293: (73) *(u8 *)(r6 +10) = r3        ; frame2: R3_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-56 fp-48_w=mmmmmmmm
294: (73) *(u8 *)(r6 +8) = r2         ; frame2: R2_w=scalar(id=18,umax=4294967295,var_off=(0x0; 0xffffffff)) R6=fp-56 fp-48_w=mmmmmmmm
295: (77) r2 >>= 8                    ; frame2: R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff))
296: (73) *(u8 *)(r6 +9) = r2         ; frame2: R2_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R6=fp-56 fp-48_w=mmmmmmmm
297: (bf) r2 = r1                     ; frame2: R1_w=scalar(id=19,umax=4294967295,var_off=(0x0; 0xffffffff)) R2_w=scalar(id=19,umax=4294967295,var_off=(0x0; 0xffffffff))
298: (77) r2 >>= 24                   ; frame2: R2_w=scalar(umax=255,var_off=(0x0; 0xff))
299: (73) *(u8 *)(r6 +23) = r2        ; frame2: R2_w=scalar(umax=255,var_off=(0x0; 0xff)) R6=fp-56 fp-40_w=mmmmmmmm
300: (bf) r2 = r1                     ; frame2: R1_w=scalar(id=19,umax=4294967295,var_off=(0x0; 0xffffffff)) R2_w=scalar(id=19,umax=4294967295,var_off=(0x0; 0xffffffff))
301: (77) r2 >>= 16                   ; frame2: R2_w=scalar(umax=65535,var_off=(0x0; 0xffff))
302: (73) *(u8 *)(r6 +22) = r2        ; frame2: R2_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-56 fp-40_w=mmmmmmmm
303: (73) *(u8 *)(r6 +20) = r1        ; frame2: R1_w=scalar(id=19,umax=4294967295,var_off=(0x0; 0xffffffff)) R6=fp-56 fp-40_w=mmmmmmmm
304: (77) r1 >>= 8                    ; frame2: R1_w=scalar(umax=16777215,var_off=(0x0; 0xffffff))
305: (73) *(u8 *)(r6 +21) = r1        ; frame2: R1_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R6=fp-56 fp-40_w=mmmmmmmm
; }
306: (95) exit
returning from callee:
 frame2: R0_w=scalar() R1_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R2_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R3_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R4_w=scalar(umax=16777215,var_off=(0x0; 0xffffff)) R5_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R6=fp-56 R7_w=scalar(id=15) R8=scalar(id=12) R10=fp0 fp-8_w=mmmmmmmm fp-16= fp-24_w=mmmmmmmm fp-32_w=mmmmmmmm fp-40_w=mmmmmmmm fp-48_w=mmmmmmmm fp-56_w=mmmmmmmm fp-64=????mmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm
to caller at 91:
 frame1: R0_w=scalar() R6=ptr_iov_iter(off=0,imm=0) R7=scalar(id=11) R8=scalar(id=11) R9=1 R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mm??????
; if (slave.major == 0 && slave.minor == 0) {
91: (69) r1 = *(u16 *)(r10 -54)       ; frame1: R1_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R10=fp0 fp-56=mmmmmmmm
; if (slave.major == 0 && slave.minor == 0) {
92: (55) if r1 != 0x0 goto pc+2       ; frame1: R1_w=0
; if (slave.major == 0 && slave.minor == 0) {
93: (69) r1 = *(u16 *)(r10 -56)       ; frame1: R1_w=scalar(umax=65535,var_off=(0x0; 0xffff)) R10=fp0 fp-56=mmmmmmmm
; if (slave.major == 0 && slave.minor == 0) {
94: (15) if r1 == 0x0 goto pc+62      ; frame1: R1_w=scalar(umax=65535,var_off=(0x0; 0xffff))
; if ((is_master && !(master.termios.c_lflag & ECHO)) && !(slave.termios.c_lflag & ECHO)) {
95: (15) if r9 == 0x0 goto pc+6       ; frame1: R9=1
; if ((is_master && !(master.termios.c_lflag & ECHO)) && !(slave.termios.c_lflag & ECHO)) {
96: (61) r1 = *(u32 *)(r10 -16)       ; frame1: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0 fp-16=mmmmmmmm
; if ((is_master && !(master.termios.c_lflag & ECHO)) && !(slave.termios.c_lflag & ECHO)) {
97: (57) r1 &= 8                      ; frame1: R1_w=scalar(umax=8,var_off=(0x0; 0x8))
; if ((is_master && !(master.termios.c_lflag & ECHO)) && !(slave.termios.c_lflag & ECHO)) {
98: (55) if r1 != 0x0 goto pc+3       ; frame1: R1_w=0
; if ((is_master && !(master.termios.c_lflag & ECHO)) && !(slave.termios.c_lflag & ECHO)) {
99: (61) r1 = *(u32 *)(r10 -40)       ; frame1: R1_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff)) R10=fp0 fp-40=mmmmmmmm
; if ((is_master && !(master.termios.c_lflag & ECHO)) && !(slave.termios.c_lflag & ECHO)) {
100: (57) r1 &= 8                     ; frame1: R1=scalar(umax=8,var_off=(0x0; 0x8))
; if ((is_master && !(master.termios.c_lflag & ECHO)) && !(slave.termios.c_lflag & ECHO)) {
101: (15) if r1 == 0x0 goto pc+55     ; frame1: R1=scalar(umax=8,var_off=(0x0; 0x8))
102: (b7) r1 = 32                     ; frame1: R1_w=32
103: (bf) r3 = r6                     ; frame1: R3_w=ptr_iov_iter(off=0,imm=0) R6=ptr_iov_iter(off=0,imm=0)
104: (0f) r3 += r1                    ; frame1: R1_w=32 R3_w=ptr_iov_iter(off=32,imm=0)
105: (bf) r1 = r10                    ; frame1: R1_w=fp0 R10=fp0
; 
106: (07) r1 += -8                    ; frame1: R1_w=fp-8
107: (b7) r7 = 8                      ; frame1: R7_w=8
; u64 nr_segs             = BPF_CORE_READ(from, nr_segs);
108: (b7) r2 = 8                      ; frame1: R2_w=8
109: (85) call bpf_probe_read_kernel#113      ; frame1: R0=scalar() fp-8=mmmmmmmm
110: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [1224] struct iov_iter.iov (0:4:0 @ offset 24)
processed 380 insns (limit 1000000) max_states_per_insn 0 total_states 17 peak_states 17 mark_read 16
-- END PROG LOAD LOG --
12:30:30.497
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: prog 'fentry__tty_write': failed to load: -22
12:30:30.497
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: failed to load object 'EventProbe_bpf'
12:30:30.497
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] libbpf: failed to load BPF skeleton 'EventProbe_bpf': -22
12:30:33.100
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (r:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe tcp_sendpage rv=$retval)
12:30:33.100
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe)
12:30:33.119
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] rtnetlink replied: No such file or directory
12:30:33.119
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] error talking to the kernel (rtnetlink_send)
12:30:33.200
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] Config.cpp:2335 Policy failed to apply and rollback is disabled
12:30:33.203
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] AgentContext.cpp:264 Failed to apply new policy from Agent.
12:30:33.203
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][warning] AgentContext.cpp:516 Endpoint is setting status to DEGRADED, reason: Policy Application Status
[...]
13:00:07.735
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] AgentComms.cpp:511 Rejecting action because of data failure: The object is not signed
2

I should have added that all four Fedora servers are showing this error. So it's likely due to a recent update, but I'm struggling to find where to start on troubleshooting this.

3

Hi @dmgeurts,

Fedora typically runs a newer version of Kernel that has some API changes that caused the behavior.

In kernel 6.5, tcp_sendpage was removed, and that's why you see this error messages.

12:30:25.574
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (r:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe tcp_sendpage rv=$retval)
12:30:25.574
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][error] PerfWatcher.cpp:178 Failed to write: (-:kprobes/elasticendpoint_TCP_SENDPAGE_RET_probe)

If possible, please update to the newer version, 8.11 or 8.10.3 of Elastic Defend, as it has fixes for this issue.

1 Like
4

Hi @ckim,

Thank you, at least I now know what is causing the issue here. I'm on Elastic Defend v8.10.3-preview, which Fleet shows as the latest version. Is there a later v8.10.3 than the one I have?

5

Hi @dmgeurts,

I should've noted "when they become available" in my previous reply. It looks like 8.10.3 will be released soon.

Meanwhile, do you feel comfortable sharing the output of following command? If you want, you can PM me directly.

/opt/Elastic/Endpoint/elastic-endpoint version

Once I have that info, I can confirm if the version you run has the fix or not.

6

version: 8.10.2, compiled: Mon Sep 18 16:00:00 2023, branch: HEAD, commit: d7d64aea9e94c3ad492857c319bca7dabeee2d9b

7

This version does not have the necessary fix for 6.5 Kernel. Please wait for 8.10.3 release and install it once released.

8

I'm having the same issue after a linux system upgrade, but I have a question about version compatibilty.

Would I be able to install the Elastic Agent 8.10.3 on a Elasticsearch cluster on 8.10.2?

If I need to upgrade the cluster to be able to use patch version of Elastic Agent this would be a big problem as we have specific windows to update the cluster.

9

Hello @ckim

Was this fixed in 8.10.3? I didn't find anything about it in the release notes.

10

I updated my Stack and Elastic Agent to version 8.10.3, with 48 hosts and I didn't notice this problem. Before, they were all on version 8.10.1 and I waited for the release of version 8.10.3 so that I could update and not have the same problem.

11

Did your hosts were running Linux with Kernel 6.5 as well?

The issue was related to the Linux kernel version on Elastic Agent versions before 8.10.3, one of the kernel APIs usesd to get some network information was removed from Linux kernel.

12

I apologize for the mistake. In fact, all hosts are Windows. I'll try to create a Linux VM with the respective kernel to test and give you feedback.

13

This is already fixed on 8.10.3, not sure what you want to test.

It was an issue on versions before 8.10.3.

14

It was this issue, they fixed on 8.10.3

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.