Elastic-agent to logstash mapper_parsing_exception with windows system integration

[irivas95]

Hi,
I am trying to send windows events via an elastic-agent (8.6.0) (with fleet in 8.6.0) to logstash (8.6.0).
the eleastic-agent is configured with an agent policy that has a system integration configured as follows:

and the pipeline:

input {
  elastic_agent {
    port => "5047"
    ssl => true
    ssl_key => '/etc/pki/logstash.pkcs8.key'
    ssl_certificate => '/etc/pki/logstash.crt'
  }
}

filter {
  if ([input][type] =~ "winlog") {
    mutate {
      remove_field => ["[winlog][logon][id]",
                       "[winlog][provider_guid]",
                       "[winlog][record_id]",
                       "[winlog][event_data][LogonType ]",
                       "[winlog][event_data][AccessList]",
                       "[winlog][event_data][AccessMask]",
                       "[winlog][event_data][HandleId]",
                       "[winlog][event_data][ElevatedToken]",
                       "[winlog][event_data][ResourceAttributes]",
                       "[winlog][event_data][ImpersonationLevel]",
                       "[winlog][event_data][KeyType]",
                       "[winlog][event_data][SamAccountName]",
                       "[winlog][event_data][KeyLength]",
                       "[winlog][event_data][Operation]",
                       "[winlog][event_data][Binary ]",
                       "[winlog][event_data][NewState]",
                       "[winlog][event_data][SubjectLogonId]",
                       "[winlog][event_data][CountNew]",
                       "[winlog][event_data][CountOld]",
                       "[winlog][event_data][DwordVal]",
                       "[winlog][event_data][DriverNameLength]",
                       "[winlog][event_data][FailureNameLength]",
                       "[winlog][event_data][ExtraStringLength]",
                       "[winlog][event_data][AddressLength]",
                       "[winlog][event_data][ProcessNameLength]",
                       "[winlog][keywords]",
                       "[winlog][process][thread][id]",
                       "[host][os][family]",
                       "[host][os][type]",
                       "[host][os][platform]"
                      ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://host1:9200","https://host2:9200","https://host3:9200"]
    user => 'user'
    password => "${PASSWORD}"
    cacert => '/etc/pki/ca.crt'
    ilm_rollover_alias  => "win-logs"
    ilm_pattern         => "{now/d}-000001"
    ilm_policy          => "20-days-default"
  }
}

when i enable the system and security options, i get the following error in logstash:

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [data_stream.dataset] of type [constant_keyword] in document. Preview of field's value: 'system.system'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"[constant_keyword] field [data_stream.dataset] only accepts values that are equal to the value defined in the mappings [system.security], but got [system.system]"}}}}

Could someone help me to understand what could cause this error or/and how to solve it?

Thanks in advance.

[Julia_Bardi (Júlia Bárdi)]

Hello,

From the error message it seems like the system.security data stream only accepts "system.security" as dataset name, and it received "system.system".
Do you see these settings in the agent policy Advanced options of the data streams?

The System integration docs are here: System | Elastic docs