Event fields missing if renderinginfo is empty

[ssiws]

Hello,

I'm using Winlogbeat to push Forwarded Event to logstash but I have the following problem:

It seems that Windows 7 clients can't reliably fill the "RenderingInfo" section of the events: some of them contain only the "Message" part, or no renderinginfo at all, etc...

I noticed that when there is a "RenderingInfo" section on the event, Winlogbeat seems to ignore the original "Level" (for example) and replaces it with the Level in the "RenderingInfo" section, but, when the RendringInfo Level is empty, I think that Winlogbeat should fallback on the "raw" level which is always populated.

The problem here is that a lot of event in Elasticsearch are recorded without the "Level" information probably because of the almost empty RenderingInfo section

One of these events here, as you can see, the renderinginfo is "empty", and the event published in Elasticsearch doesn't contains the event level (4, in the Event.System part of the XML)

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
	<System>
		<Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" /> 
		<EventID Qualifiers="16384">7036</EventID> 
		<Version>0</Version> 
		<Level>4</Level> 
		<Task>0</Task> 
		<Opcode>0</Opcode> 
		<Keywords>0x8080000000000000</Keywords> 
		<TimeCreated SystemTime="2017-05-05T10:42:16.133065800Z" /> 
		<EventRecordID>159915</EventRecordID> 
		<Correlation /> 
		<Execution ProcessID="820" ThreadID="8820" /> 
		<Channel>System</Channel> 
		<Computer>computername</Computer> 
		<Security /> 
	</System>
	<EventData>
		<Data Name="param1">test service</Data> 
		<Data Name="param2">Running</Data> 
		<Binary>570069006E00480074007400700041007500740062341455500000005300760063002F0034000000</Binary> 
	</EventData>
	<RenderingInfo Culture="en-US">
		<Message /> 
		<Level /> 
		<Task /> 
		<Opcode /> 
		<Channel /> 
		<Provider /> 
		<Keywords /> 
	</RenderingInfo>
  </Event>

[andrewkroh]

This sounds like a reasonable enhancement to me.

The Level field always comes directly from RenderingInfo when the XML is unmarshaled. https://github.com/elastic/beats/blob/6b7df0c1b8b820da7826aa5871de40052ecaaf80/winlogbeat/sys/event.go#L38

This is where Level is added to the output. Some logic could be added to fallback to LevelRaw when Level is empty. And logic already exists to convert the number to a string. Just need to do a eventlogging.EventType(LevelRaw).String() to convert it to string.

Any interest in contributing to the project?

[ssiws]

OK, I can contribute to the project, but I've found another problem by looking at the code: The levels in the logic to convert the LevelRaw number to a string are only valid for Windows XP and below.

Now (since Vista), we have:
LogAlways (default level if nothing is specified, the event viewer assumes it's Informational), Critical, Error, Warning, Informational, Verbose

source: https://msdn.microsoft.com/en-us/library/windows/desktop/dd996917(v=vs.85).aspx

and https://msdn.microsoft.com/en-us/library/aa363652.aspx ([...]In Windows Vista, the event logging infrastructure was redesigned[...])

What do you think? Do we still want to keep compatibility with Windows XP and below?

[maddin2016]

@ssiws, for updating the Log Levels .

[andrewkroh]

The EventType I referenced is in the eventlogging package which is specifically for the Event Logging API (pre-Windows Vista). We must maintain capability with pre-Vista versions, so it seems that we cannot change or reuse EventType.

The wineventlog package is for the Windows Event Log API (vista and newer). So that package is probably the best place to put any new types specifically for vista and newer.

And I previously mentioned that the eventlog.go file would be the place to add the fallback logic, but this is common to both the old and new event log APIs. So it probably needs to perform the fallback to LevelRaw in code specific to the newer API. This file would be a good place.

[ssiws]

Ok, thank you for these informations. I just sent a PR on 5.4, here: https://github.com/elastic/beats/pull/4257