I want to use the auditbeat test config command to test out new configurations on the servers they need to be deployed to (automatically using ansible template validate, to avoid overwriting configurations with something that isn't working). This works well for the other beats I am using/testing/deploying using this method (filebeat/metricbeat/packetbeat), but fails for auditbeat.
The command I need to run is similar to the following, but with a different configuration for testing each time:
/usr/share/auditbeat/bin/auditbeat test config -c /etc/auditbeat/auditbeat.yml
The error messages seem to indicate that to test the configuration auditbeat is attempting to first remove existing probes, which have been setup by the instance that is already successfully running on the previous configuration and I wouldn't expect to be removed by a test config command:
Exiting: 1 error: 1 error: system/socket dataset setup failed: unable to delete existing KProbes. Is Auditbeat already running?: 16 errors: unable to remove kprobe 'p:auditbeat/sys_execve_call SyS_execve path=+0(%di):u64 arg2=+8(%di):u64 arg3=+16(%di):u64 arg4=+24(%di):u64 arg5=+32(%di):u64 arg6=+40(%di):u64 arg7=+48(%di):u64 arg8=+56(%di):u64 arg9=+64(%di):u64 arg10=+72(%di):u64 arg11=+80(%di):u64 arg12=+88(%di):u64 arg13=+96(%di):u64 arg14=+104(%di):u64 arg15=+112(%di):u64 arg16=+120(%di):u64 argptrs=+0(%si):u64 arg18=+8(%si):u64 arg19=+16(%si):u64 arg20=+24(%si):u64 arg21=+32(%si):u64 arg22=+40(%si):u64 param0=+0(+0(%si)):u64 arg24=+8(+0(%si)):u64 arg25=+16(+0(%si)):u64 arg26=+24(+0(%si)):u64 arg27=+32(+0(%si)):u64 arg28=+40(+0(%si)):u64 arg29=+48(+0(%si)):u64 arg30=+56(+0(%si)):u64 arg31=+64(+0(%si)):u64 arg32=+72(+0(%si)):u64 arg33=+80(+0(%si)):u64 arg34=+88(+0(%si)):u64 arg35=+96(+0(%si)):u64 arg36=+104(+0(%si)):u64 arg37=+112(+0(%si)):u64 arg38=+120(+0(%si)):u64 param1=+0(+8(%si)):u64 arg40=+8(+8(%si)):u64 arg41=+16(+8(%si)):u64 arg42=+24(+8(%si)):u64 arg43=+32(+8(%si)):u64 arg44=+40(+8(%si)):u64 arg45=+48(+8(%si)):u64 arg46=+56(+8(%si)):u64 arg47=+64(+8(%si)):u64 arg48=+72(+8(%si)):u64 arg49=+80(+8(%si)):u64 arg50=+88(+8(%si)):u64 arg51=+96(+8(%si)):u64 arg52=+104(+8(%si)):u64 arg53=+112(+8(%si)):u64 arg54=+120(+8(%si)):u64 param2=+0(+16(%si)):u64 arg56=+8(+16(%si)):u64 arg57=+16(+16(%si)):u64 arg58=+24(+16(%si)):u64 arg59=+32(+16(%si)):u64 arg60=+40(+16(%si)):u64 arg61=+48(+16(%si)):u64 arg62=+56(+16(%si)):u64 arg63=+64(+16(%si)):u64 arg64=+72(+16(%si)):u64 arg65=+80(+16(%si)):u64 arg66=+88(+16(%si)):u64 arg67=+96(+16(%si)):u64 arg68=+104(+16(%si)):u64 arg69=+112(+16(%si)):u64 arg70=+120(+16(%si)):u64 param3=+0(+24(%si)):u64 arg72=+8(+24(%si)):u64 arg73=+16(+24(%si)):u64 arg74=+24(+24(%si)):u64 arg75=+32(+24(%si)):u64 arg76=+40(+24(%si)):u64 arg77=+48(+24(%si)):u64 arg78=+56(+24(%si)):u64 arg79=+64(+24(%si)):u64 arg80=+72(+24(%si)):u64 arg81=+80(+24(%si)):u64 arg82=+88(+24(%si)):u64 arg83=+96(+24(%si)):u64 arg84=+104(+24(%si)):u64 arg85=+112(+24(%si)):u64 arg86=+120(+24(%si)):u64 param4=+0(+32(%si)):u64 arg88=+8(+32(%si)):u64 arg89=+16(+32(%si)):u64 arg90=+24(+32(%si)):u64 arg91=+32(+32(%si)):u64 arg92=+40(+32(%si)):u64 arg93=+48(+32(%si)):u64 arg94=+56(+32(%si)):u64 arg95=+64(+32(%si)):u64 arg96=+72(+32(%si)):u64 arg97=+80(+32(%si)):u64 arg98=+88(+32(%si)):u64 arg99=+96(+32(%si)):u64 arg100=+104(+32(%si)):u64 arg101=+112(+32(%si)):u64 arg102=+120(+32(%si)):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'r:auditbeat/sys_execve_ret SyS_execve retval=%ax:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/do_exit do_exit ': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/commit_creds commit_creds uid=+4(%di):u32 gid=+8(%di):u32 euid=+20(%di):u32 egid=+24(%di):u32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/sock_init_data sock_init_data socket=%di sock=%si': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/inet_create inet_create proto=%dx:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/inet_release inet_release sock=+32(%di)': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp4_connect_in tcp_v4_connect sock=%di laddr=+4(%di):u32 lport=+728(%di):u16 af=+0(%si):u16 addr=+4(%si):u32 port=+2(%si):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'r:auditbeat/tcp4_connect_out tcp_v4_connect retval=%ax:s32': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/ip_local_out_call ip_local_out sock=%si size=+128(%dx):u32 af=+16(%si):u16 laddr=+4(%si):u32 lport=+728(%si):u16 raddr=+0(%si):u32 rport=+12(%si):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp_v4_do_rcv_call tcp_v4_do_rcv sock=%di size=+128(%si):u32 laddr=+4(%di):u32 lport=+728(%di):u16 raddr=+0(%di):u32 rport=+12(%di):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/udp_sendmsg_in udp_sendmsg sock=%di size=%dx laddr=+4(%di):u32 lport=+728(%di):u16 raddr=+4(+0(%si)):u32 rport=+2(+0(%si)):u16 altraddr=+0(%di):u32 altrport=+12(%di):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/udp_queue_rcv_skb udp_queue_rcv_skb sock=%di size=+128(%si):u32 laddr=+4(%di):u32 lport=+728(%di):u16 iphdr=+196(%si):u16 udphdr=+194(%si):u16 base=+208(%si) packet=+0(+208(%si)):u64 arg9=+8(+208(%si)):u64 arg10=+16(+208(%si)):u64 arg11=+24(+208(%si)):u64 arg12=+32(+208(%si)):u64 arg13=+40(+208(%si)):u64 arg14=+48(+208(%si)):u64 arg15=+56(+208(%si)):u64 arg16=+64(+208(%si)):u64 arg17=+72(+208(%si)):u64 arg18=+80(+208(%si)):u64 arg19=+88(+208(%si)):u64 arg20=+96(+208(%si)):u64 arg21=+104(+208(%si)):u64 arg22=+112(+208(%si)):u64 arg23=+120(+208(%si)):u64 arg24=+128(+208(%si)):u64 arg25=+136(+208(%si)):u64 arg26=+144(+208(%si)):u64 arg27=+152(+208(%si)):u64 arg28=+160(+208(%si)):u64 arg29=+168(+208(%si)):u64 arg30=+176(+208(%si)):u64 arg31=+184(+208(%si)):u64 arg32=+192(+208(%si)):u64 arg33=+200(+208(%si)):u64 arg34=+208(+208(%si)):u64 arg35=+216(+208(%si)):u64 arg36=+224(+208(%si)):u64 arg37=+232(+208(%si)):u64 arg38=+240(+208(%si)):u64 arg39=+248(+208(%si)):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/clock_sync_probe SyS_newuname magic=+0(%di):u64 timestamp=+8(%di):u64': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'r:auditbeat/inet_csk_accept_ret4 inet_csk_accept sock=%ax laddr=+4(%ax):u32 lport=+728(%ax):u16 raddr=+0(%ax):u32 rport=+12(%ax):u16 family=+16(%ax):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy; unable to remove kprobe 'p:auditbeat/tcp_sendmsg_in4 tcp_sendmsg sock=%di size=%dx laddr=+4(%di):u32 lport=+728(%di):u16 raddr=+0(%di):u32 rport=+12(%di):u16 family=+16(%di):u16': write /sys/kernel/debug/tracing/kprobe_events: device or resource busy
If there anyway of testing new auditbeat configurations without first shutting down the existing running instance? I am happy to open an issue with a full test case on Github if this is a known issue that isn't already tracked there.