XML encoded logs from Mcafee EPO logs to logstash

Elastic Stack Logstash
1

Hi my scenario is we need to collect logs from Mcafee EPO and send to our third party cloud logging platform.We have logstash server in between and its receiving logs from EPO and forwarding to logging platform.However mcafee is sending syslogs encoded with xml and xml is not supported by cloud logging platform.
I am prettty new to logstash can we do some filtering or xml to syslog conversion on logstash to achieve this.?

My Config:-elk-5.5.0-0

input {
tcp {
port => 514
ssl_cert => 'C:/Bitnami/elk-5.5.0-0/logstash/ssl/syslogselfsigned.crt'
ssl_key => 'C:/Bitnami/elk-5.5.0-0/logstash/ssl/syslogselfsigned.key'
ssl_enable => true
ssl_verify => false
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
}

syslog {
host => "10.x.x.x"
port => 514
}
}

Sample message:-1:
<29>1 2017-10-13T06:45:34.0Z SERVERMCAFEE01 EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] <?xml version="1.0" encoding="utf-8"?>{3ecdcaaa-45f5-11e7-0a6c-0050569e0a71}SERVER10050569E500110.x.x.x5.0.6.220Linux420root242742017-10-13T06:45:34ENDP_GS_1020LYNX040983Property CollectionN/AEPOAGENT3000N/AN/A

2

Even if we can covert xml to json using logstash and forward it to cloud logging platform, it will be helpful.

3

I was about to say that you could use an xml filter to parse the XML, but the example you posted isn't valid XML. I suggest you just use a grok filter to parse the message.

4

Hi Magnus,

Thanks for your quick reply, how about below message ( I am more interested in this message, so that I can create some alerting for virus detection).

<29>1 2017-10-06T11:21:16.0Z SERVERMCAFEE01 EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] <?xml version="1.0" encoding="UTF-8"?><EPOevent><MachineInfo><MachineName>TS01</MachineName><AgentGUID>{768ab1ba-31aa-11e7-2f62-00505696c547}</AgentGUID><IPAddress>10.x.x.x</IPAddress><OSName>Windows 2008 R2</OSName><UserName>SYSTEM</UserName><TimeZoneBias>420</TimeZoneBias><RawMACAddress></RawMACAddress></MachineInfo><SoftwareInfo ProductName="McAfee Endpoint Security" ProductVersion="10.5.0" ProductFamily="TVD"><CommonFields><Analyzer>ENDP_AM_1050</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.5.0</AnalyzerVersion><AnalyzerHostName>TS01</AnalyzerHostName><AnalyzerEngineVersion>5900.7806</AnalyzerEngineVersion><AnalyzerDetectionMethod>On-Demand Scan</AnalyzerDetectionMethod><AnalyzerDATVersion>3125.0</AnalyzerDATVersion></CommonFields><Event><EventID>1278</EventID><Severity>3</Severity><GMTTime>2017-10-06T11:17:35</GMTTime><CommonFields><ThreatCategory>av.detect</ThreatCategory><ThreatEventID>1278</ThreatEventID><ThreatSeverity>2</ThreatSeverity><ThreatName>EICAR test file</ThreatName><ThreatType>test</ThreatType><DetectedUTC>2017-10-06T11:17:35Z</DetectedUTC><ThreatActionTaken>IDS_ALERT_ACT_TAK_DEL</ThreatActionTaken><ThreatHandled>True</ThreatHandled><SourceHostName>TS01</SourceHostName><SourceProcessName>On-Demand Scan</SourceProcessName><TargetHostName>TS01</TargetHostName><TargetUserName>CASAA\mohab</TargetUserName><TargetFileName>C:\Users\mohab\Desktop\dell.com</TargetFileName></CommonFields><CustomFields target="EPExtendedEventMT"><BladeName>IDS_BLADE_NAME_SPB</BladeName><AnalyzerContentCreationDate>2017-10-04T15:46:00Z</AnalyzerContentCreationDate><AnalyzerGTIQuery>False</AnalyzerGTIQuery><ThreatDetectedOnCreation>False</ThreatDetectedOnCreation><TargetName>dell.com</TargetName><TargetPath>C:\Users\mohab\Desktop</TargetPath><TargetHash>01c65da52c8f3e5959ce9bb405ba6372</TargetHash><TargetFileSize>70</TargetFileSize><TargetModifyTime>2017-10-06T11:16:03Z</TargetModifyTime><TargetAccessTime>2017-10-06T11:16:03Z</TargetAccessTime><TargetCreateTime>2017-10-06T11:16:03Z</TargetCreateTime><Cleanable>False</Cleanable><TaskName>IDS_ODS_TASK_NAME_RIGHT_CLICK</TaskName><FirstAttemptedAction>IDS_ALERT_THACT_ATT_CLE</FirstAttemptedAction><FirstActionStatus>False</FirstActionStatus><SecondAttemptedAction>IDS_ALERT_THACT_ATT_DEL</SecondAttemptedAction><SecondActionStatus>True</SecondActionStatus><AttackVectorType>4</AttackVectorType><DurationBeforeDetection>92</DurationBeforeDetection><NaturalLangDescription>IDS_NATURAL_LANG_ODS_DETECTION_DELETED|TargetName=dell.com|TargetPath=C:\Users\mohab\Desktop|ThreatName=EICAR test file|ThreatType=test|TaskName=IDS_ODS_TASK_NAME_RIGHT_CLICK|TargetUserName=CASAA\mohab</NaturalLangDescription><AccessRequested></AccessRequested><AMCoreContentVersion>3125.0</AMCoreContentVersion></CustomFields></Event></SoftwareInfo></EPOevent>

I earlier modified logstash conf but that did not help.

input {
tcp {
port => 514
ssl_cert => 'C:/Bitnami/elk-5.5.0-0/logstash/ssl/syslogselfsigned.crt'
ssl_key => 'C:/Bitnami/elk-5.5.0-0/logstash/ssl/syslogselfsigned.key'
ssl_enable => true
ssl_verify => false
}
}

filter {
xml {
source => "message"
store_xml => false
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
}

syslog {
host => "10.x.x.x"
port => 514
}
}

5

Use a grok filter to extract timestamp and other parts of the message to separate fields. One of the fields should contain the XML payload and that's the field you then reference in your xml filter.

6

Hi Magnus,

I tried grok and there is a pattern ready for this sort of messages, %{SYSLOG5424LINE} this pattern extracted all the fields and "syslog5424_msg" is the field which is mapped to the valid xml payload.

But during the grok process its adding backlash infront of double quotes turning xml content in to an invalid xml, at this point I think that is the reason why xml filtering is not working.I tried to replace the backslash but that is not working?

filter {
  grok {
    match => { "message" => '%{SYSLOG5424LINE}' }
  }
mutate {
       gsub => ["syslog5424_msg",'\"','"']
    }
xml {
source => "syslog5424_msg"
store_xml => false
}
}
7

But during the grok process its adding backlash infront of double quotes turning xml content in to an invalid xml

No, I don't think that's what's happening. Show an example event, e.g. by copy/pasting from Kibana's JSON tab or using a stdout { codec => rubydebug } output.

8

Uploaded json in a file as I cant post it in comment, it exceeded 7000 characters :roll_eyes: .

Scratch that I am not authorized to upload JSON.

9 
{
  "_index": "logstash-2017.10.26",
  "_type": "logs",
  "_id": "AV9YXpGL9BeeCRxNMOe1",
  "_version": 1,
  "_score": 9.158171,
  "_source": {
    "syslog5424_sd": "[agentInfo@3401 tenantId=\"1\"]",
    "syslog5424_ver": "1",
    "message": "<29>1 2017-10-26T11:08:31.0Z SERVERMCAFEE01 EPOEvents - EventFwd [agentInfo@3401 tenantId=\"1\"] <?xml version=\"1.0\" encoding=\"UTF-8\"?><EPOevent><MachineInfo><MachineName>SERVERTS01</MachineName><AgentGUID>{768ab1ba-31aa-11e7-2f62-00505696c547}</AgentGUID><IPAddress>10.0.0.2</IPAddress><OSName>Windows 2008 R2</OSName><UserName>SYSTEM</UserName><TimeZoneBias>420</TimeZoneBias><RawMACAddress></RawMACAddress></MachineInfo><SoftwareInfo ProductName=\"McAfee Endpoint Security\" ProductVersion=\"10.5.0\" ProductFamily=\"TVD\"><CommonFields><Analyzer>ENDP_AM_1050</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.5.0</AnalyzerVersion><AnalyzerHostName>SERVERTS01</AnalyzerHostName><AnalyzerEngineVersion>5900.7806</AnalyzerEngineVersion><AnalyzerDetectionMethod>On-Access Scan</AnalyzerDetectionMethod><AnalyzerDATVersion>3146.0</AnalyzerDATVersion></CommonFields><Event><EventID>1278</EventID><Severity>3</Severity><GMTTime>2017-10-26T11:04:59</GMTTime><CommonFields><ThreatCategory>av.detect</ThreatCategory><ThreatEventID>1278</ThreatEventID><ThreatSeverity>2</ThreatSeverity><ThreatName>EICAR test file</ThreatName><ThreatType>test</ThreatType><DetectedUTC>2017-10-26T11:04:59Z</DetectedUTC><ThreatActionTaken>IDS_ALERT_ACT_TAK_DEL</ThreatActionTaken><ThreatHandled>True</ThreatHandled><SourceHostName>SERVERTS01</SourceHostName><SourceProcessName>C:\\Windows\\System32\\notepad.exe</SourceProcessName><TargetHostName>SERVERTS01</TargetHostName><TargetUserName>CASAASNP\\testuser</TargetUserName><TargetFileName>C:\\Users\\testuser\\Desktop\\bahu.com</TargetFileName></CommonFields><CustomFields target=\"EPExtendedEventMT\"><BladeName>IDS_BLADE_NAME_SPB</BladeName><AnalyzerContentCreationDate>2017-10-25T16:32:00Z</AnalyzerContentCreationDate><AnalyzerGTIQuery>False</AnalyzerGTIQuery><ThreatDetectedOnCreation>True</ThreatDetectedOnCreation><TargetName>bahu.com</TargetName><TargetPath>C:\\Users\\testuser\\Desktop</TargetPath><TargetHash>44d88612fea8a8f36de82e1278abb02f</TargetHash><TargetFileSize>68</TargetFileSize><TargetModifyTime>2017-10-26T11:04:59Z</TargetModifyTime><TargetAccessTime>2017-10-26T11:04:59Z</TargetAccessTime><TargetCreateTime>2017-10-26T11:04:59Z</TargetCreateTime><Cleanable>False</Cleanable><TaskName>IDS_OAS_TASK_NAME</TaskName><FirstAttemptedAction>IDS_ALERT_THACT_ATT_CLE</FirstAttemptedAction><FirstActionStatus>False</FirstActionStatus><SecondAttemptedAction>IDS_ALERT_THACT_ATT_DEL</SecondAttemptedAction><SecondActionStatus>True</SecondActionStatus><AttackVectorType>4</AttackVectorType><DurationBeforeDetection>0</DurationBeforeDetection><NaturalLangDescription>IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=bahu.com|TargetPath=C:\\Users\\testuser\\Desktop|ThreatName=EICAR test file|SourceProcessName=C:\\Windows\\System32\\notepad.exe|ThreatType=test|TargetUserName=CASAASNP\\testuser</NaturalLangDescription><AccessRequested></AccessRequested><DetectionMessage>IDS_OAS_DEFAULT_THREAT_MESSAGE</DetectionMessage><AMCoreContentVersion>3146.0</AMCoreContentVersion></CustomFields></Event></SoftwareInfo></EPOevent>\r",
    "syslog5424_app": "EPOEvents",
    "syslog5424_msg": "<?xml version=\"1.0\" encoding=\"UTF-8\"?><EPOevent><MachineInfo><MachineName>SERVERTS01</MachineName><AgentGUID>{768ab1ba-31aa-11e7-2f62-00505696c547}</AgentGUID><IPAddress>10.0.0.2</IPAddress><OSName>Windows 2008 R2</OSName><UserName>SYSTEM</UserName><TimeZoneBias>420</TimeZoneBias><RawMACAddress></RawMACAddress></MachineInfo><SoftwareInfo ProductName=\"McAfee Endpoint Security\" ProductVersion=\"10.5.0\" ProductFamily=\"TVD\"><CommonFields><Analyzer>ENDP_AM_1050</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.5.0</AnalyzerVersion><AnalyzerHostName>SERVERTS01</AnalyzerHostName><AnalyzerEngineVersion>5900.7806</AnalyzerEngineVersion><AnalyzerDetectionMethod>On-Access Scan</AnalyzerDetectionMethod><AnalyzerDATVersion>3146.0</AnalyzerDATVersion></CommonFields><Event><EventID>1278</EventID><Severity>3</Severity><GMTTime>2017-10-26T11:04:59</GMTTime><CommonFields><ThreatCategory>av.detect</ThreatCategory><ThreatEventID>1278</ThreatEventID><ThreatSeverity>2</ThreatSeverity><ThreatName>EICAR test file</ThreatName><ThreatType>test</ThreatType><DetectedUTC>2017-10-26T11:04:59Z</DetectedUTC><ThreatActionTaken>IDS_ALERT_ACT_TAK_DEL</ThreatActionTaken><ThreatHandled>True</ThreatHandled><SourceHostName>SERVERTS01</SourceHostName><SourceProcessName>C:\\Windows\\System32\\notepad.exe</SourceProcessName><TargetHostName>SERVERTS01</TargetHostName><TargetUserName>CASAASNP\\testuser</TargetUserName><TargetFileName>C:\\Users\\testuser\\Desktop\\bahu.com</TargetFileName></CommonFields><CustomFields target=\"EPExtendedEventMT\"><BladeName>IDS_BLADE_NAME_SPB</BladeName><AnalyzerContentCreationDate>2017-10-25T16:32:00Z</AnalyzerContentCreationDate><AnalyzerGTIQuery>False</AnalyzerGTIQuery><ThreatDetectedOnCreation>True</ThreatDetectedOnCreation><TargetName>bahu.com</TargetName><TargetPath>C:\\Users\\testuser\\Desktop</TargetPath><TargetHash>44d88612fea8a8f36de82e1278abb02f</TargetHash><TargetFileSize>68</TargetFileSize><TargetModifyTime>2017-10-26T11:04:59Z</TargetModifyTime><TargetAccessTime>2017-10-26T11:04:59Z</TargetAccessTime><TargetCreateTime>2017-10-26T11:04:59Z</TargetCreateTime><Cleanable>False</Cleanable><TaskName>IDS_OAS_TASK_NAME</TaskName><FirstAttemptedAction>IDS_ALERT_THACT_ATT_CLE</FirstAttemptedAction><FirstActionStatus>False</FirstActionStatus><SecondAttemptedAction>IDS_ALERT_THACT_ATT_DEL</SecondAttemptedAction><SecondActionStatus>True</SecondActionStatus><AttackVectorType>4</AttackVectorType><DurationBeforeDetection>0</DurationBeforeDetection><NaturalLangDescription>IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=bahu.com|TargetPath=C:\\Users\\testuser\\Desktop|ThreatName=EICAR test file|SourceProcessName=C:\\Windows\\System32\\notepad.exe|ThreatType=test|TargetUserName=CASAASNP\\testuser</NaturalLangDescription><AccessRequested></AccessRequested><DetectionMessage>IDS_OAS_DEFAULT_THREAT_MESSAGE</DetectionMessage><AMCoreContentVersion>3146.0</AMCoreContentVersion></CustomFields></Event></SoftwareInfo></EPOevent>\r",
    "syslog5424_msgid": "EventFwd",
    "@timestamp": "2017-10-26T11:08:33.141Z",
    "port": 63866,
    "syslog5424_ts": "2017-10-26T11:08:31.0Z",
    "syslog5424_pri": "29",
    "@version": "1",
    "host": "10.0.0.1",
    "syslog5424_host": "SERVERMCAFEE01"
  },
10

That looks perfectly normal. The double quotes in the XML are of course escaped when they're inside a double-quoted string.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.