Previously i used JSON data from a file (that were previously taken by beats) to test my logstash conf and filtering. I had no problems, but when i try to read straight from auditbeat with the Beats input plugin, Logstash prints few errors and also the output doesn't appear.
here is sample of the error form logstash while receiving auditbeat events:
[2021-08-12T19:46:31,639][WARN ][logstash.filters.json ][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"Process containerd-shim (PID: 5305) by user root is RUNNING", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Process': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"Process containerd-shim (PID: 5305) by user root is RUNNING"; line: 1, column: 9]>}
[2021-08-12T19:46:31,640][WARN ][logstash.filters.json ][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"ERROR for PID 5323: failed to hash executable /pause for PID 5323: failed to stat file /pause: stat /pause: no such file or directory", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'ERROR': was expecting ('true', 'false' or 'null')
and another sample:
[2021-08-12T19:51:56,036][WARN ][logstash.filters.json ][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"Process sleep (PID: 26212) by user root STOPPED", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Process': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"Process sleep (PID: 26212) by user root STOPPED"; line: 1, column: 9]>}
[2021-08-12T19:52:06,041][WARN ][logstash.filters.json ][main][699e35f42d506ad9de208d3e678ec089b2419d6ee7373c8b174f7f900352ed56] Error parsing json {:source=>"message", :raw=>"Process sleep (PID: 26625) by user root STARTED", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Process': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"Process sleep (PID: 26625) by user root STARTED"; line: 1, column: 9]>}
Here is the logstash conf:
input{
# file{
# path => "/usr/share/logstash/config/auditbeat.json"
# start_position => "beginning"
# ecs_compatibility => "v1"
# # codec => "json"
# }
beats {
port => 5044
ecs_compatibility => "v1"
}
}
filter{
json {
source => "message"
target => "[@metadata][message]"
remove_field => "message"
}
if [@metadata][message][destination][ip] {
mutate{
add_field => { "[dst_ip]" => "%{[@metadata][message][destination][ip]}"}
}
}
if [@metadata][message][source][ip] {
mutate{
add_field => { "[src_ip]" => "%{[@metadata][message][source][ip]}"}
}
}
if ![src_ip] and ![dst_ip]{
drop {}
}
mutate{
remove_field => [ "[@metadata][message]" ]
}
.
.
.
.
output{
# stdout {
# codec => rubydebug
# { metadata => true }
# }
file{
path => "/usr/share/logstash/config/enriched-auditbeat.json"
codec => "json"
}
}