"Task" field from event viewer is not parsed into elasticsearch

Elastic Stack Beats
1

Hi,

We are trying to use "Winlogbeat" with output into Elasticsearch. We are almost successful in getting most of the fields from event viewer to elasticsearch, but except "Task".

Even though 'Task' has value in Windows event viewer, but in elasticsearch it doesn't. Please refer the below screenshot.

Environment : Windows 10 / ELK 6.5.0 / Winlogbeat 6.5.0 / Default Index Template

Winlogbeat-Error
Winlogbeat-Error.PNG610×806 15.8 KB

P.S. we've enabled debug logging and found that 'Task' field picked up as empty value.

Debug Log : "Level:Information Task: Opcode:Info Keywords:[Classic] RenderErrorCode:0 RenderErrorDataItemName: RenderErr:} 2018-12-27T20:33:01.974+0530 DEBUG"

Please provide some insights on how to get the value of the field 'Task'

Thanks,
Saravanan

2

Could you please share your configuration formatted using </> and the debug logs you have captured?

3

Please find the winlogbeat.yml configuration below.

###################### Winlogbeat Configuration Example ##########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html

#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
  - name: Application
provider: 
 - KitBag
  
#==================== Elasticsearch template setting ==========================

setup.template.overwrite: false
setup.template.settings:
  index.number_of_shards: 3
  index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "elkserver.com:80"
  
  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  space.id: "winlogbeat"

#============================= Elastic Cloud ==================================

# These settings simplify using winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elkserver.com:9200"]
  compression_level: 9
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Procesors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.to_files: true
logging.files:
path: D:/Winlogbeat/logs
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# winlogbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
xpack.monitoring.enabled: true

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
xpack.monitoring.elasticsearch:
4

Please find the debug logs below : Part - 1

2019-01-04T11:31:50.579Z	INFO	instance/beat.go:424	winlogbeat start running.
2019-01-04T11:31:50.600Z	DEBUG	[eventlog]	eventlog/wineventlog.go:130	WinEventLog[Application] using subscription query=<QueryList>
  <Query Id="0">
    <Select Path="Application">*[System[Provider[@Name='KitBag']]]</Select>
  </Query>
</QueryList>
2019-01-04T11:31:50.630Z	DEBUG	[winlogbeat]	beater/eventlogger.go:117	EventLog[Application] opened successfully
2019-01-04T11:31:50.631Z	DEBUG	[eventlog_detail]	eventlog/wineventlog.go:156	WinEventLog[Application] EventHandles returned 3 handles
2019-01-04T11:31:50.632Z	DEBUG	[eventlog]	eventlog/cache.go:86	messageFilesCache[Application] size=1
2019-01-04T11:31:50.636Z	DEBUG	[eventlog_detail]	eventlog/wineventlog.go:257	WinEventLog[Application] XML=<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='KitBag'/><EventID Qualifiers='0'>0</EventID><Level>4</Level><Task>0</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime='2019-01-04T11:31:02.000000000Z'/><EventRecordID>18452</EventRecordID><Channel>Application</Channel><Computer>appserver.com</Computer><Security/></System><EventData><Data>Transkits_In folder configuration set to: 'D:\DENT_Files\Transkits_In'</Data></EventData><RenderingInfo Culture='en-US'><Message>Transkits_In folder configuration set to: 'D:\DENT_Files\Transkits_In'</Message><Level>Information</Level><Task></Task><Opcode>Info</Opcode><Channel></Channel><Provider></Provider><Keywords><Keyword>Classic</Keyword></Keywords></RenderingInfo></Event> Event={Provider:{Name:KitBag GUID: EventSourceName:} EventIdentifier:{Qualifiers:0 ID:0} Version:0 LevelRaw:4 TaskRaw:0 OpcodeRaw:0 TimeCreated:{SystemTime:2019-01-04 11:31:02 +0000 UTC} RecordID:18452 Correlation:{ActivityID: RelatedActivityID:} Execution:{ProcessID:0 ThreadID:0 ProcessorID:0 SessionID:0 KernelTime:0 UserTime:0 ProcessorTime:0} Channel:Application Computer:appserver.com User:SID Identifier[] Name[] Domain[] Type[] EventData:{Pairs:[{Key:Data Value:Transkits_In folder configuration set to: 'D:\DENT_Files\Transkits_In'}]} UserData:{Name:{Space: Local:} Pairs:[]} Message:Transkits_In folder configuration set to: 'D:\DENT_Files\Transkits_In' Level:Information Task: Opcode:Info Keywords:[Classic] RenderErrorCode:0 RenderErrorDataItemName: RenderErr:}
2019-01-04T11:31:50.637Z	DEBUG	[eventlog_detail]	eventlog/wineventlog.go:257	WinEventLog[Application] XML=<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='KitBag'/><EventID Qualifiers='0'>0</EventID><Level>4</Level><Task>0</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime='2019-01-04T11:31:02.000000000Z'/><EventRecordID>18453</EventRecordID><Channel>Application</Channel><Computer>appserver.com</Computer><Security/></System><EventData><Data>Service started successfully.</Data></EventData><RenderingInfo Culture='en-US'><Message>Service started successfully.</Message><Level>Information</Level><Task></Task><Opcode>Info</Opcode><Channel></Channel><Provider></Provider><Keywords><Keyword>Classic</Keyword></Keywords></RenderingInfo></Event> Event={Provider:{Name:KitBag GUID: EventSourceName:} EventIdentifier:{Qualifiers:0 ID:0} Version:0 LevelRaw:4 TaskRaw:0 OpcodeRaw:0 TimeCreated:{SystemTime:2019-01-04 11:31:02 +0000 UTC} RecordID:18453 Correlation:{ActivityID: RelatedActivityID:} Execution:{ProcessID:0 ThreadID:0 ProcessorID:0 SessionID:0 KernelTime:0 UserTime:0 ProcessorTime:0} Channel:Application Computer:appserver.com User:SID Identifier[] Name[] Domain[] Type[] EventData:{Pairs:[{Key:Data Value:Service started successfully.}]} UserData:{Name:{Space: Local:} Pairs:[]} Message:Service started successfully. Level:Information Task: Opcode:Info Keywords:[Classic] RenderErrorCode:0 RenderErrorDataItemName: RenderErr:}
2019-01-04T11:31:50.637Z	DEBUG	[eventlog_detail]	eventlog/wineventlog.go:257	WinEventLog[Application] XML=<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='KitBag'/><EventID Qualifiers='0'>0</EventID><Level>4</Level><Task>0</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime='2019-01-04T11:31:04.000000000Z'/><EventRecordID>18454</EventRecordID><Channel>Application</Channel><Computer>appserver.com</Computer><Security/></System><EventData><Data>Service stopped successfully.</Data></EventData><RenderingInfo Culture='en-US'><Message>Service stopped successfully.</Message><Level>Information</Level><Task></Task><Opcode>Info</Opcode><Channel></Channel><Provider></Provider><Keywords><Keyword>Classic</Keyword></Keywords></RenderingInfo></Event> Event={Provider:{Name:KitBag GUID: EventSourceName:} EventIdentifier:{Qualifiers:0 ID:0} Version:0 LevelRaw:4 TaskRaw:0 OpcodeRaw:0 TimeCreated:{SystemTime:2019-01-04 11:31:04 +0000 UTC} RecordID:18454 Correlation:{ActivityID: RelatedActivityID:} Execution:{ProcessID:0 ThreadID:0 ProcessorID:0 SessionID:0 KernelTime:0 UserTime:0 ProcessorTime:0} Channel:Application Computer:appserver.com User:SID Identifier[] Name[] Domain[] Type[] EventData:{Pairs:[{Key:Data Value:Service stopped successfully.}]} UserData:{Name:{Space: Local:} Pairs:[]} Message:Service stopped successfully. Level:Information Task: Opcode:Info Keywords:[Classic] RenderErrorCode:0 RenderErrorDataItemName: RenderErr:}
2019-01-04T11:31:50.638Z	DEBUG	[eventlog]	eventlog/wineventlog.go:194	WinEventLog[Application] Read() is returning 3 records
2019-01-04T11:31:50.638Z	DEBUG	[winlogbeat]	beater/eventlogger.go:133	EventLog[Application] Read() returned 3 records
5

Please find the debug logs below : Part - 2

2019-01-04T11:31:50.638Z	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-01-04T11:31:02.000Z",
  "@metadata": {
    "beat": "winlogbeat",
    "type": "doc",
    "version": "6.5.0"
  },
  "type": "wineventlog",
  "record_number": "18452",
  "level": "Information",
  "opcode": "Info",
  "message": "Transkits_In folder configuration set to: 'D:\\DENT_Files\\Transkits_In'",
  "event_data": {
    "param1": "Transkits_In folder configuration set to: 'D:\\DENT_Files\\Transkits_In'"
  },
  "beat": {
    "hostname": "appserver",
    "version": "6.5.0",
    "name": "appserver"
  },
  "event_id": 0,
  "host": {
    "name": "appserver",
    "architecture": "x86_64",
    "os": {
      "platform": "windows",
      "version": "6.3",
      "family": "windows",
      "build": "9600.19204"
    },
    "id": "514fc96d-c9cd-43a0-b46d-81523c117565"
  },
  "computer_name": "appserver.com",
  "keywords": [
    "Classic"
  ],
  "log_name": "Application",
  "source_name": "KitBag"
}
2019-01-04T11:31:50.638Z	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-01-04T11:31:02.000Z",
  "@metadata": {
    "beat": "winlogbeat",
    "type": "doc",
    "version": "6.5.0"
  },
  "record_number": "18453",
  "keywords": [
    "Classic"
  ],
  "event_id": 0,
  "computer_name": "appserver.com",
  "beat": {
    "version": "6.5.0",
    "name": "appserver",
    "hostname": "appserver"
  },
  "source_name": "KitBag",
  "event_data": {
    "param1": "Service started successfully."
  },
  "level": "Information",
  "opcode": "Info",
  "type": "wineventlog",
  "log_name": "Application",
  "host": {
    "name": "appserver",
    "id": "514fc96d-c9cd-43a0-b46d-81523c117565",
    "architecture": "x86_64",
    "os": {
      "version": "6.3",
      "family": "windows",
      "build": "9600.19204",
      "platform": "windows"
    }
  },
  "message": "Service started successfully."
}
2019-01-04T11:31:50.638Z	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2019-01-04T11:31:04.000Z",
  "@metadata": {
    "beat": "winlogbeat",
    "type": "doc",
    "version": "6.5.0"
  },
  "keywords": [
    "Classic"
  ],
  "type": "wineventlog",
  "event_data": {
    "param1": "Service stopped successfully."
  },
  "log_name": "Application",
  "event_id": 0,
  "record_number": "18454",
  "opcode": "Info",
  "beat": {
    "name": "appserver",
    "hostname": "appserver",
    "version": "6.5.0"
  },
  "host": {
    "id": "514fc96d-c9cd-43a0-b46d-81523c117565",
    "name": "appserver",
    "architecture": "x86_64",
    "os": {
      "platform": "windows",
      "version": "6.3",
      "family": "windows",
      "build": "9600.19204"
    }
  },
  "level": "Information",
  "message": "Service stopped successfully.",
  "source_name": "KitBag",
  "computer_name": "appserver.com"
}
2019-01-04T11:31:50.639Z	DEBUG	[eventlog_detail]	eventlog/wineventlog.go:213	WinEventLog[Application] No more events
6

Any luck with this request ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.