Hello,
I'm using filebeat to send syslog input to a kafka server (it works wonderfully, thank you).
But I'm wondering: how can I add the IP from the machine that is sending its syslog input in my logs?
(I'm aware of processors like add_host_metada but I need the IP from the machine filebeat is receiving from)
Hi @emilie, welcome to the Elastic community forums!
Events produced by the syslog Filebeat input should contain a log.source.address field — is this not what you are looking for?
Shaunak
hello @shaunak,
I haven't got the source. log.address field in my output. here is an example of the result (syslog from Trend Micro server is being received in my filebeat):
{
"@timestamp": "2020-01-14T08:06:18.020Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.5.1",
"truncated": false
},
"message": "Pattern Update Status",
"source": {
"domain": "Apex One (Mac) as a Service"
},
"input": {
"type": "syslog"
},
"agent": {
"type": "filebeat",
"ephemeral_id": "9112a2fd-1161-4c02-8ca4-dd36dfff1c4f",
"hostname": "*****",
"id": "1111f525-77e2-35b4-9441-7788db304dd7",
"version": "7.5.1"
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "*****"
},
"cef": {
"device": {
"event_class_id": "800101",
"vendor": "Trend Micro",
"product": "Apex Central",
"version": "2019"
},
"name": "Pattern Update Status",
"severity": "3",
"extensions": {
"deviceCustomNumber1Label": "EntityPatternUpdate_ConnectionStatus",
"deviceCustomString5Label": "EntityPatternUpdate_PatternVersion",
"deviceCustomString1": "Windows server 2016",
"deviceCustomString1Label": "EntityPatternUpdate_OSName",
"deviceCustomString6Label": "AUComponent_Type",
"deviceReceiptTime": "Jan 14 2020 06:00:16 GMT+00:00",
"deviceCustomString6": "2",
"DeviceCustomNumber2": "1208090624",
"deviceCustomNumber3": "1",
"deviceCustomString2": "fe80::181:e295:2a19:956%3,fe80::1810:51e:e89b:fb9d%5,10.0.0.5,2001:0:2851:782c:1810:51e:e89b:fb9d",
"deviceCustomNumber1": "100",
"deviceCustomString5": "15.619.00",
"deviceCustomString3": "0",
"deviceFacility": "Apex One (Mac) ",
"deviceCustomNumber2Label": "EntityPatternUpdate_PatternType",
"deviceCustomNumber3Label": "EntityPatternUpdate_PatternStatus",
"sourceHostName": "Apex One (Mac) as a Service",
"deviceCustomString2Label": "EntityPatternUpdate_IPAddressList",
"deviceCustomString3Label": "EntityPatternUpdate_UpdateAgent"
},
"version": "0"
},
"event": {
"original": "<133>Jan 14 2020 09:06:17 xeu4kc.manage.trendmicro.com CEF:0|Trend Micro|Apex Central|2019|800101|Pattern Update Status|3|rt=Jan 14 2020 06:00:16 GMT+00:00 shost=Apex One (Mac) as a Service cs1Label=EntityPatternUpdate_OSName cs1=Windows server 2016 cs2Label=EntityPatternUpdate_IPAddressList cs2=fe80::181:e295:2a19:956%3,fe80::1810:51e:e89b:fb9d%5,10.0.0.5,2001:0:2851:782c:1810:51e:e89b:fb9d cs3Label=EntityPatternUpdate_UpdateAgent cs3=0 cn1Label=EntityPatternUpdate_ConnectionStatus cn1=100 cn2Label=EntityPatternUpdate_PatternType cn2=1208090624 cs5Label=EntityPatternUpdate_PatternVersion cs5=15.619.00 cn3Label=EntityPatternUpdate_PatternStatus cn3=1 cs6Label=AUComponent_Type cs6=2 deviceFacility=Apex One (Mac) ",
"code": "800101",
"severity": 3
},
"observer": {
"vendor": "Trend Micro",
"product": "Apex Central",
"version": "2019"
},
"error": {
"message": "deviceReceiptTime: value is not a valid timestamp"
}
}
But I hadn't noticed there was a parsing problem with a timestamp, maybe I just have this problem:
Yeah, I'm also wondering if you are running into the same issue. The fix for that issue should be released in 7.5.2 and 7.6.0, if you want to wait for a bit to try either of those out.
Shaunak
Ok, I will wait and check out if it is better with these versions, thank you!
I wonder if there might be another problem though. If you look at the rt field in the CEF (event.original) you see
rt=Jan 14 2020 06:00:16 GMT+00:00
which seems OK considering this documentation
cef.extensions.deviceReceiptTime
The time at which the event related to the activity was received. The format is MMM dd yyyy HH:mm:ss or milliseconds since epoch (Jan 1st 1970)
type: date
or maybe not because of the trailing GMT part?
I also have other parsing issues on the "." character in filename and filePath:
{
"@timestamp":"2020-01-14T10:33:29.788Z",
"@metadata":{
"beat":"filebeat",
"type":"_doc",
"version":"7.5.1",
"truncated":false
},
"ecs":{
"version":"1.1.0"
},
"event":{
"original":"<133>Jan 14 2020 11:33:29 xeu4kc.manage.trendmicro.com CEF:0|Trend Micro|Apex Central|2019|Spyware Detected|Spyware Detected|3|deviceExternalId=11 rt=Jan 14 2020 10:31:14 GMT+00:00 cnt=1 dhost=DESKTOP-158G7DH cn1Label=PatternType cn1=1073741840 cs1Label=VirusName cs1=HKTL_MIMIKATZ64 cs2Label=EngineVersion cs2=6.2.4035 cs5Label=ActionResult cs5=Fichier nettoyé cs6Label=PatternVersion cs6=2249 cat=1727 dvchost=WE-PRO1-3806-2 fname=C:\\Users\\user\\AppData\\Local\\Temp\\VirtualBox Dropped Files\\2020-01-14T10_31_04.086895800Z\\mimilib.dll filePath=C:\\Users\\user\\AppData\\Local\\Temp\\VirtualBox Dropped Files\\2020-01-14T10_31_04.086895800Z\\mimilib.dll dst=10.0.2.15 deviceFacility=Apex One fileHash=6A315ED2B7458262AF917E58D0C6E85DB450E589 duser=user cn2Label=SpywareHostDetail_ScanType cn2=11 cn3Label=SpywareHostDetail_RiskType cn3=6 ",
"code":"Spyware Detected",
"severity":3
},
"input":{
"type":"syslog"
},
"agent":{
"id":"6165f525-77e2-42b4-9441-6c88db304dd7",
"version":"7.5.1",
"type":"filebeat",
"ephemeral_id":"43d6ee55-8bc3-4bc8-902a-213f1092cb9a",
"hostname":"****"
},
"host":{
"name":"****"
},
"cef":{
"version":"0",
"device":{
"event_class_id":"Spyware Detected",
"vendor":"Trend Micro",
"product":"Apex Central",
"version":"2019"
},
"name":"Spyware Detected",
"severity":"3",
"extensions":{
"deviceCustomNumber3":"6 ",
"deviceExternalId":"11",
"deviceCustomString1":"HKTL_MIMIKATZ64",
"deviceFacility":"Apex One",
"destinationAddress":"10.0.2.15",
"deviceCustomString2Label":"EngineVersion",
"fileHash":"6A315ED2B7458262AF917E58D0C6E85DB450E589",
"deviceHostName":"WE-PRO1-3806-2",
"deviceCustomNumber3Label":"SpywareHostDetail_RiskType",
"deviceCustomNumber1Label":"PatternType",
"destinationHostName":"DESKTOP-158G7DH",
"deviceCustomString2":"6.2.4035",
"deviceCustomString5Label":"ActionResult",
"deviceCustomString6":"2249",
"baseEventCount":"1",
"deviceCustomString1Label":"VirusName",
"destinationUserName":"user",
"deviceCustomString6Label":"PatternVersion",
"deviceCustomNumber1":"1073741840",
"deviceEventCategory":"1727",
"DeviceCustomNumber2":"11",
"deviceReceiptTime":"Jan 14 2020 10:31:14 GMT+00:00",
"deviceCustomString5":"Fichier nettoyé",
"deviceCustomNumber2Label":"SpywareHostDetail_ScanType"
}
},
"observer":{
"version":"2019",
"hostname":"WE-PRO1-3806-2",
"vendor":"Trend Micro",
"product":"Apex Central"
},
"message":"Spyware Detected",
"destination":{
"ip":"10.0.2.15",
"domain":"DESKTOP-158G7DH",
"user":{
"name":"user"
}
},
"error":{
"message":[
"malformed value for fname at pos 383",
"malformed value for filePath at pos 493",
"deviceReceiptTime: value is not a valid timestamp"
]
}
}
If I understand it right, reading this spec of CEF, which makes reference to SimpleDateFormat
CEF supports several variations on time/date formats to identify the time an event occurred accurately. These formats are detailed below.
1.Milliseconds since January 1, 1970 (integer)
—This time format supplies an integer with the count in milliseconds from January 1, 1970 to the time the event occurred.
2.MMM dd HH:mm:ss
3.MMM dd HH:mm:ss.SSS zzz
4.MMM dd HH:mm:ss.SSS
5.MMM dd HH:mm:ss zzz
6.MMM dd yyyy HH:mm:ss
7.MMM dd yyyy HH:mm:ss.SSS zzz
8.MMM dd yyyy HH:mm:ss.SSS
9.MMM dd yyyy HH:mm:ss zzz
For a key to the date formats shown above, visit the SimpleDateFormat page at: java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html.
which seems to be handled there in beats
there should be more format strings in timeLayouts.
In my case "Jan 2 2006 15:04:05 GMT-07:00" is missing, RFC 822 time zone is also missing
If I'm not wrong, General time zone can be specified as Pacific Standard Time or GMT-08:00 not only the PST string (like it is handled in beats).
hello @andrewkroh, do you agree with me on this date thing?
about the fname/filePath parsing issue I'm afraid the parser.go is quite a piece for me, sorry I can't help more
thank you for your work, cheers
@shaunak actually I am not sure it is the same problem. I have a filebeat listening for syslog on my local network on tcp port 514 with this config file:
filebeat.inputs:
- type: syslog
protocol.tcp:
host: "192.168.2.190:514"
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#filebeat.autodiscover:
# providers:
# - type: docker
# hints.enabled: true
processors:
- add_cloud_metadata: ~
- rename:
fields:
- {from: "message", to: "event.original"}
- decode_cef:
field: event.original
using linux logger command I send:
logger -n 192.168.2.190 -P 514 "CEF:0|Trend Micro|Apex Central|2019|700211|Attack Discovery Detections|3|deviceExternalId=5 rt=Jan 17 2019 03:38:06 EST dhost=VCAC-Window-331 dst=10.201.86.150 customerExternalID=8c1e2d8f-a03b-47ea-aef8-5aeab99ea697 cn1Label=SLF_RiskLevel cn1=0 cn2Label=SLF_PatternNumber cn2=30.1012.00 cs1Label=SLF_RuleID cs1=powershell invoke expression cat=point of entry cs2Label=SLF_ADEObjectGroup_Info_1 cs2=process - powershell.exe - {#012 "META_FILE_MD5" : "7353f60b1739074eb17c5f4dddefe239",#012 "META_FILE_NAME" : "powershell.exe",#012 "META_FILE_SHA1" : "6cbce4a295c163791b60fc23d285e6d84f28ee4c",#012 "META_FILE_SHA2" : "de96a6e69944335375dc1ac238336066889d9ffc7d73628ef4fe1b1b160ab32c",#012 "META_PATH" : "c:\\windows\\system32\\windowspowershell\\v1.0\\",#012 "META_PROCESS_CMD" : [ "powershell iex test2" ],#012 "META_PROCESS_PID" : 10924,#012 "META_SIGNER" : "microsoft windows",#012 "META_SIGNER_VALIDATION" : true,#012 "META_USER_USER_NAME" : "Administrator",#012 "META_USER_USER_SERVERNAME" : "VCAC-WINDOW-331",#012 "OID" : 1#012}#012" --tcp
I took this CEF example but I edited the rt date for Jan 17 2019 03:38:06 EST (since Jan 17 2019 03:38:06 GMT+
00:00 is causing parsing issue "deviceReceiptTime: value is not a valid timestamp")
result is:
{
"@timestamp": "2019-01-17T03:38:06.000Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.5.1",
"truncated": false
},
"input": {
"type": "syslog"
},
"agent": {
"type": "filebeat",
"ephemeral_id": "440a3d53-f595-4c02-bb7f-5b519dfed9ae",
"hostname": "****",
"id": "aa50291e-4fc8-4080-8fa9-330e20544e43",
"version": "7.5.1"
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "****"
},
"observer": {
"vendor": "Trend Micro",
"product": "Apex Central",
"version": "2019"
},
"message": "Attack Discovery Detections",
"organization": {
"id": "8c1e2d8f-a03b-47ea-aef8-5aeab99ea697"
},
"event": {
"code": "700211",
"severity": 3,
"original": "<13>1 2020-01-20T17:35:24.242599+01:00 **** emilie - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"67500\"] CEF:0|Trend Micro|Apex Central|2019|700211|Attack Discovery Detections|3|deviceExternalId=5 rt=Jan 17 2019 03:38:06 EST dhost=VCAC-Window-331 dst=10.201.86.150 customerExternalID=8c1e2d8f-a03b-47ea-aef8-5aeab99ea697 cn1Label=SLF_RiskLevel cn1=0 cn2Label=SLF_PatternNumber cn2=30.1012.00 cs1Label=SLF_RuleID cs1=powershell invoke expression cat=point of entry cs2Label=SLF_ADEObjectGroup_Info_1 cs2=process - powershell.exe - {#012 META_FILE_MD5 : 7353f60b1739074eb17c5f4dddefe239,#012 META_FILE_NAME : powershell.exe,#012 META_FILE_SHA1 : 6cbce4a295c163791b60fc23d285e6d84f28ee4c,#012 META_FILE_SHA2 : de96a6e69944335375dc1ac238336066889d9ffc7d73628ef4fe1b1b160ab32c,#012 META_PATH : c:\\\\windows\\\\system32\\\\windowspowershell\\\\v1.0\\\\,#012 META_PROCESS_CMD : [ powershell iex test2 ],#012 META_PROCESS_PID : 10924,#012 META_SIGNER : microsoft windows,#012 META_SIGNER_VALIDATION : true,#012 META_USER_USER_NAME : Administrator,#012 META_USER_USER_SERVERNAME : VCAC-WINDOW-331,#012 OID : 1#012}#012"
},
"cef": {
"version": "0",
"device": {
"vendor": "Trend Micro",
"product": "Apex Central",
"version": "2019",
"event_class_id": "700211"
},
"name": "Attack Discovery Detections",
"severity": "3",
"extensions": {
"DeviceCustomNumber2": "30.1012.00",
"destinationAddress": "10.201.86.150",
"destinationHostName": "VCAC-Window-331",
"deviceCustomNumber2Label": "SLF_PatternNumber",
"deviceCustomNumber1": "0",
"deviceCustomString2": "process - powershell.exe - {#012 META_FILE_MD5 : 7353f60b1739074eb17c5f4dddefe239,#012 META_FILE_NAME : powershell.exe,#012 META_FILE_SHA1 : 6cbce4a295c163791b60fc23d285e6d84f28ee4c,#012 META_FILE_SHA2 : de96a6e69944335375dc1ac238336066889d9ffc7d73628ef4fe1b1b160ab32c,#012 META_PATH : c:\\windows\\system32\\windowspowershell\\v1.0\\,#012 META_PROCESS_CMD : [ powershell iex test2 ],#012 META_PROCESS_PID : 10924,#012 META_SIGNER : microsoft windows,#012 META_SIGNER_VALIDATION : true,#012 META_USER_USER_NAME : Administrator,#012 META_USER_USER_SERVERNAME : VCAC-WINDOW-331,#012 OID : 1#012}#012",
"deviceCustomString1Label": "SLF_RuleID",
"deviceCustomString2Label": "SLF_ADEObjectGroup_Info_1",
"deviceExternalId": "5",
"deviceEventCategory": "point of entry",
"deviceReceiptTime": "Jan 17 2019 03:38:06 EST",
"deviceCustomString1": "powershell invoke expression",
"deviceCustomNumber1Label": "SLF_RiskLevel",
"customerExternalID": "8c1e2d8f-a03b-47ea-aef8-5aeab99ea697"
}
},
"destination": {
"ip": "10.201.86.150",
"domain": "VCAC-Window-331"
}
}
as you can see I don't have a parsing error this time but I haven't got a event.source.ip neither. What am I missing there?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.