Getting lots of "failed to load publisher metadata for" into winlogbeat

yquirion · February 20, 2024, 2:54pm

Dear all,

I have configures my Windows servers to send their logs to a Windows Event Collector (WEC). On this server, I created a "subscription" to send System and Security event logs to that WEC server.

On the WEC server, I installed a winlogbeat agent and enabled the "ForwardedEvents" to be sent to my ELK cluster:

winlogbeat.event_logs:
  - name: ForwardedEvents
    ignore_older: 96h
    event_id: -5152,-5156,-5157,-5158
    tags: [forwarded]

So far so good. But later I modify that subscription to add more logs: I added Application, Windows Powershell, Microsoft-Windows-PowerShell/Admin*, Microsoft-Windows-PowerShell/Operational.

When the WEC started receiving those event, the winlogbeat started to display lots of warning into its log file:

{"log.level":"warn","@timestamp":"2024-02-20T09:11:20.810-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for Passwordstate Service (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:22.744-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for AD FS (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:23.214-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for COM+ SOAP Services (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:25.195-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for Directory Synchronization (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:25.197-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for ADSync (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:27.397-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for PasswordResetService (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:27.538-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for nssm (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:32.335-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for gupdate (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:36.417-0500","log.origin":{"file.name":"wineventlog/metadata_store.go","file.line":182},"message":"Failed to read event metadata from publisher. Continuing to next event.","service.name":"winlogbeat","publisher":"Microsoft-Windows-Perflib","error":{"message":"XML syntax error on line 5: unexpected EOF"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:36.417-0500","log.origin":{"file.name":"wineventlog/metadata_store.go","file.line":182},"message":"Failed to read event metadata from publisher. Continuing to next event.","service.name":"winlogbeat","publisher":"Microsoft-Windows-Perflib","error":{"message":"XML syntax error on line 5: unexpected EOF"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:36.417-0500","log.origin":{"file.name":"wineventlog/metadata_store.go","file.line":182},"message":"Failed to read event metadata from publisher. Continuing to next event.","service.name":"winlogbeat","publisher":"Microsoft-Windows-Perflib","error":{"message":"XML syntax error on line 5: unexpected EOF"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:41.512-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSExchangeIS (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:48.791-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSSQL$MICROSOFT##WID (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:50.744-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for ASP.NET 4.0.30319.0 (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:51.181-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":517},"message":"WinEventLog[WEC-STI/ADFS] EventHandles returned error The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:11:51.181-0500","log.logger":"winlogbeat","log.origin":{"file.name":"beater/eventlogger.go","file.line":179},"message":"Read() encountered recoverable error. Reopening handle...","service.name":"winlogbeat","id":"WEC-STI/ADFS","error":{"message":"The query result is stale or invalid and must be recreated. This may be due to the log being cleared or rolling over after the query result was created."},"channel":"WEC-STI/ADFS","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:22.608-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for sshd (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:30.453-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for Xerox CentreWare Web (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:46.668-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSExchangeTransport (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:51.536-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSExchangeFrontEndTransport (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:51.727-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for Microsoft-Filtering-FIPFS (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:51.738-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSExchangeTransportDelivery (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:51.755-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSExchangeTransportSubmission (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:12:54.208-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSExchange Mid-Tier Storage (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:13:10.945-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for Fusion Audit (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T09:13:17.648-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for MSExchange RBAC (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}

Searching the web I found this, but there are no really conclusion: https://github.com/elastic/beats/issues/36111 and its related to version 7.x of the beat. I'm running version 8.11.1

I'm wondering why the winlogbeat log file is complaining about thise things: Service, AD FS, COM+ SOAP Services, Directory Synchronization, PasswordResetService, nssm, gupdate, and so on...

Since I was getting too many of those warnings, I just updated the WEC Subscription to only sends System and Secutity like I did at the beginning. Those errors slow down a lot to almost nothing. Just get 3 of those warning in the past 15 minutes.

If you need more information, please feel free to ask me!

Thank you and best regards,
Yanick

andrewkroh · February 20, 2024, 3:41pm

Are your WEC subscriptions configured to send RenderedText?

Add forwarded: true (docs) to all of the readers that are consuming forwarded logs. This should stop Winlogbeat from trying to source the message contents from the collector host.

winlogbeat.event_logs:
  - name: MyForwardedChannel
    forwarded: true
    ...

yquirion · February 20, 2024, 4:34pm

Hi @andrewkroh,

Thank you very much for your reply.

The documentation says:

The value defaults to true for the ForwardedEvents log and false for any other log

So I tought I didn't need to specify that parameter, but to make sure, I added it. Unfortunately, after restarting the winlogbeat, I got some of those warnings again:

{"log.level":"warn","@timestamp":"2024-02-20T11:22:41.833-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for AD FS Auditing (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T11:22:41.833-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for Microsoft-Windows-IIS-Logging (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-20T11:22:41.862-0500","log.origin":{"file.name":"eventlog/wineventlog.go","file.line":637},"message":"failed to load publisher metadata for AD FS (returning an empty metadata store): failed in EvtOpenPublisherMetadata: The system cannot find the file specified.","service.name":"winlogbeat","ecs.version":"1.6.0"}

Here is how is configured my WEC subscription:

C:\>wecutil gs "Logon"
Subscription Id: Logon
SubscriptionType: SourceInitiated
Description: Inventorier l'ouverture de session sur les serveurs windows
Enabled: true
Uri: http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog
ConfigurationMode: Normal
DeliveryMode: Push
DeliveryMaxLatencyTime: 900000
HeartbeatInterval: 900000
Query: <QueryList><Query Id="0"><Select Path="Security">*</Select><Select Path="System">*</Select><Suppress Path="Security">*[System[(EventID=5152 or EventID=5157)]]</Suppress><Suppress Path="System">*[System[(EventID=5152 or EventID=5157)]]</Suppress></Query></QueryList>
ReadExistingEvents: false
TransportName: HTTPS
ContentFormat: RenderedText
Locale: en-CA
LogFile: ForwardedEvents
PublisherName: Microsoft-Windows-EventCollector

As we can see, they are configured in RenderedText. Is this may be a bug in my version of winlogbeat?

The event who is generating lots those warning is the Application one. The WEC server doesn't have all the different Event Sources that are available on other servers:

Thanks again for your time, I appreciate it!

Regards,
Yanick

yquirion · February 20, 2024, 4:59pm

Hi again!

I just created another subsciption only for Application and selected some of my computers to use that new subscription.

<?xml version="1.0" encoding="UTF-8"?>
<Subscription xmlns="http://schemas.microsoft.com/2006/03/windows/events/subscription">
        <SubscriptionId>Application Logs</SubscriptionId>
        <SubscriptionType>SourceInitiated</SubscriptionType>
        <Description></Description>
        <Enabled>true</Enabled>
        <Uri>http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri>
        <ConfigurationMode>MinLatency</ConfigurationMode>
        <Delivery Mode="Push">
                <Batching>
                        <MaxLatencyTime>30000</MaxLatencyTime>
                </Batching>
                <PushSettings>
                        <Heartbeat Interval="3600000"/>
                </PushSettings>
        </Delivery>
        <Query>
                <![CDATA[
<QueryList><Query Id="0"><Select Path="Application">*</Select></Query></QueryList>
                ]]>
        </Query>
        <ReadExistingEvents>false</ReadExistingEvents>
        <TransportName>HTTPS</TransportName>
        <ContentFormat>RenderedText</ContentFormat>
        <Locale Language="en-CA"/>
        <LogFile>WEC-FMSS</LogFile>
        <PublisherName></PublisherName>
        <AllowedSourceNonDomainComputers>
                <AllowedIssuerCAList>
                        <IssuerCA>ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ</IssuerCA>
                </AllowedIssuerCAList>
        </AllowedSourceNonDomainComputers>
        <AllowedSourceDomainComputers>ZZZZZZZZZZZZZZZZZZZZZZ:</AllowedSourceDomainComputers>
</Subscription>

I configure it to send the logs into a custom destination for those events. Then I enable this new source into winlogbeat:

name: WEC-FMSS
ignore_older: 96h
forwarded: true
tags: [wec-fmss]

Looking at the winlogbeat log file while looking at this destination log folder, I saw:

But when looking at Kibana, all data seems to be there:

Thanks!
Yanick

system · March 19, 2024, 6:59pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.