Hi, everyone!
I have faced with such problem: several CEF strings pushed into the following pipelines configuration causing _cefparseerror in result cause to incorrect string in the input. It's break original message in random place and do not parse wit CEF codec properly as result. This error do appear either in real ELK stack and in test environment (logstash pipeline tester from epacke).
Reason for two pipelines is incorrect timestamp in some messages i want to fix. Pay attention to 'timestamps' with minuses in given input strings.
This bug appear not every launch of the tests. About 1 error for 10 launches.
pipelines.yml:
- pipeline.id: cef_correction_1st_stage
path.config: "/etc/logstash/conf.d/aix-in.conf"
- pipeline.id: cef_correction_2st_stage
path.config: "/etc/logstash/conf.d/aix-out.conf"
aix-in.conf:
input {
kafka {
codec => plain {}
bootstrap_servers => "KAFKA_SERVER"
topics => "TOPIC_NAME"
client_id => "logstash01"
decorate_events => basic
group_id => "test_group"
}
}
filter {
if "-6208" in [message] {
ruby {
code => "
event.set('current_time', Time.now().to_i);
puts Time.now().to_i
"
}
mutate {
gsub => ["message", "-6208[0-9]+", "%{current_time}"]
}
mutate {
add_tag => ["Timestamp was changed"]
}
}
mutate {
add_field => {
"cefField" => "%{message}"
}
}
}
output {
tcp {
host => localhost
port => 12888
codec => plain {
format => "%{cefField}"
}
}
}
aix-out.conf
input {
tcp {
host => localhost
port => 12888 # Listener port (tcp)
codec => cef {}
}
}
filter {
ruby {
code => "
event.set('[event][start]', Time.now());
event.set('[event][end]', Time.now());
"
}
}
output {
elasticsearch {
hosts => ["ELASTIC_HOST"]
http_compression => true
index => "index_test"
manage_template => true
template_overwrite => false
template => "/etc/logstash/conf.d/aix.template"
user => "${es_pl_login}"
password => "${es_pl_pwd}"
}
}
Input data:
CEF:0|IBM|AIX Audit PR||PROC_SetUserIDs|PROC_SetUserIDs|Low| eventId=1856831065 msg=orarootagent.bin 1642298 50072098 effect: 1000, real: 1000, saved: -1, login: -1 start=-62086925465000 end=-62086925465000 categorySignificance=/Informational categoryBehavior=/Modify/Attribute categoryDeviceGroup=/Application catdt=Operating System categoryOutcome=/Attempt categoryObject=/Host/Resource/Process art=1689839632340 deviceSeverity=OK act=OK rt=-62086925465000 suser=user_ru1 sproc=2 sourceServiceName=0 flexString1=20230720105200 cn2=3 cs1Label=ACL cs2Label=Group cs3Label=Owner cs4Label=Reason or Error Code cs5Label=PCL cs6Label=Volume Group ID cn1Label=File Descriptor cn2Label=Parent PID cn3Label=Physical Volume Index ahost=host-spb-best.sims.ru agt=10.0.0.2 agentZoneURI=/All Zones/System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 amac=00-50-56-B9-68-28 av=8.4.0.8955.0 atz=Europe/Moscow at=sdkmultifolderreader dvchost=hostname-db3 dvc=10.0.0.1 deviceZoneURI=/All Zones/System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 dtz=Europe/Moscow geid=0 _cefVer=0.1 aid=3fh2sPoEBABCUfDP6qAyT2w\=\=
CEF:0|IBM|AIX Audit PR||PROC_SetUserIDs|PROC_SetUserIDs|Low| eventId=1856831065 msg=orarootagent.bin 1642298 50072098 effect: 1000, real: 1000, saved: -1, login: -1 start=1689856215000 end=1689856215000 categorySignificance=/Informational categoryBehavior=/Modify/Attribute categoryDeviceGroup=/Application catdt=Operating System categoryOutcome=/Attempt categoryObject=/Host/Resource/Process art=1689839632340 deviceSeverity=OK act=OK rt=1689856215000 suser=user_ru1 sproc=2 sourceServiceName=0 flexString1=20230720105200 cn2=3 cs1Label=ACL cs2Label=Group cs3Label=Owner cs4Label=Reason or Error Code cs5Label=PCL cs6Label=Volume Group ID cn1Label=File Descriptor cn2Label=Parent PID cn3Label=Physical Volume Index ahost=host-spb-best.sims.ru agt=10.0.0.2 agentZoneURI=/All Zones/System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 amac=00-50-56-B9-68-28 av=8.4.0.8955.0 atz=Europe/Moscow at=sdkmultifolderreader dvchost=hostname-db3 dvc=10.0.0.1 deviceZoneURI=/All Zones/System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 dtz=Europe/Moscow geid=0 _cefVer=0.1 aid=3fh2sPoEBABCUfDP6qAyT2w\=\=
Bug screenshot:
