Hello
i have problem with my filter.
my log sent by NXLOG format JSON from windows Server to two shipper logstash, the shipper send the log to kafka.
my problem kafka add some information and this information alter the format log it's not see by logstah like JSON.
this is my filter
input {
kafka {
group_id => ['windows']
topics => ['windows']
consumer_threads => 3
decorate_events => true
codec => json_lines {charset => "CP1251"}
type => 'windows'
add_field => {"kafka.type" => "windows_kafka"}
}
}
filter {
if [type] == 'windows' {
json {
source => "message"
}
}
}
and her my log
2017-11-27T12:52:57.641Z "ip" {"EventTime":"2017-11-27 13:52:55","Hostname":"jorf98.ma.abazaimkorn.org","Keywords":-9218868437227405312,"EventType":"AUDIT_FAILURE","SeverityValue":4,"Severity":"ERROR","EventID":4656,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54678548424614-5478-4994-A5BA-3E3B0jkjk5870D}","Version":1,"Task":12800,"OpcodeValue":0,"RecordNumber":148527,"ProcessID":628,"ThreadID":640,"Channel":"Security","Message":"A handle to an object was requested.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-854521854658874356854-162879516-2487165456178655800-109272\r\n\tAccount Name:\t\tjorf2\r\n\tAccount Domain:\t\tabazaimkorne\r\n\tLogon ID:\t\t0x8499EB95D\r\n\r\nObject:\r\n\tObject Server:\t\tSecurity\r\n\tObject Type:\t\tFile\r\n\tObject Name:\t\tC:\Windows\System32\dsa.msc\r\n\tHandle ID:\t\t0x0\r\n\tResource Attributes:\t-\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x525c\r\n\tProcess Name:\t\tC:\Windows\System32\mmc.exe\r\n\r\nAccess Request Information:\r\n\tTransaction ID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\tAccesses:\t\tREAD_CONTROL\r\n\t\t\t\tSYNCHRONIZE\r\n\t\t\t\tWriteData (or AddFile)\r\n\t\t\t\tAppendData (or AddSubdirectory or CreatePipeInstance)\r\n\t\t\t\tWriteEA\r\n\t\t\t\tReadAttributes\r\n\t\t\t\tWriteAttributes\r\n\t\t\t\t\r\n\tAccess Reasons:\t\tREAD_CONTROL:\tGranted by\tD:(A;;0x1200a9;;;BU)\r\n\t\t\t\tSYNCHRONIZE:\tGranted by\tD:(A;;0x1200a9;;;BU)\r\n\t\t\t\tWriteData (or AddFile):\tNot granted\r\n\t\t\t\tAppendData (or AddSubdirectory or CreatePipeInstance):\tNot granted\r\n\t\t\t\tWriteEA:\tNot granted\r\n\t\t\t\tReadAttributes:\tGranted by ACE on parent folder\tD:(A;;0x1200a9;;;BU)\r\n\t\t\t\tWriteAttributes:\tNot granted\r\n\t\t\t\t\r\n\tAccess Mask:\t\t0x120196\r\n\tPrivileges Used for Access Check:\t-\r\n\tRestricted SID Count:\t0","Category":"File System","Opcode":"Info","SubjectUserSid":"S-1-5-21-815996685-162879516-2736015800-109272","SubjectUserName":"jorf","SubjectDomainName":"abazaimkorn","SubjectLogonId":"0x277eb95d","ObjectServer":"Security","ObjectType":"File","ObjectName":"C:\Windows\System32\dsa.msc","HandleId":"0x0","TransactionId":"{00000000-0000-0000-0000-000000000000}","AccessList":"%%1538\r\n\t\t\t\t%%1541\r\n\t\t\t\t%%4417\r\n\t\t\t\t%%4418\r\n\t\t\t\t%%4420\r\n\t\t\t\t%%4423\r\n\t\t\t\t%%4424\r\n\t\t\t\t","AccessReason":"%%1538:\t%%1801\tD:(A;;0x1200a9;;;BU)\r\n\t\t\t\t%%1541:\t%%1801\tD:(A;;0x1200a9;;;BU)\r\n\t\t\t\t%%4417:\t%%1805\r\n\t\t\t\t%%4418:\t%%1805\r\n\t\t\t\t%%4420:\t%%1805\r\n\t\t\t\t%%4423:\t%%1811\tD:(A;;0x1200a9;;;BU)\r\n\t\t\t\t%%4424:\t%%1805\r\n\t\t\t\t","AccessMask":"0x120196","PrivilegeList":"-","RestrictedSidCount":"0","ProcessName":"C:\Windows\System32\mmc.exe","ResourceAttributes":"-","EventReceivedTime":"2017-11-27 13:52:57","SourceModuleName":"in","SourceModuleType":"im_msvistalog"}
how can delete the timestamp send it by kafka (2017-11-27T12:52:57.641Z "ip")