Office 365 module: data is not showing in dashboard and beat fails after 1 hour

After installing and configuring the Office 365 Module according to instructions here, I'm seeing a couple of issues.

  1. After running successfully for (exactly) 1 hour, the o365beat process on the Windows 10 machine fails with an error like the following:
ERROR	instance/beat.go:916	Exiting: error listing all available content between 2021-08-27 20:32:33 +0000 UTC and 2021-08-27 15:43:44.6768062 -0500 CDT m=+3600.127845301: 
non-200 status during api request. 
confirm audit log searching is enabled for the target tenancy
(https://docs.microsoft.com/en-us/microsoft-365/compliance/turn-audit-log-search-on-or-off#turn-on-audit-log-search).
Message: "Authorization has been denied for this request."

There's more to the error log but that's the essential part. However, the Audit Log feature is definitely enabled, which can be seen by running "Get-AdminAuditLogConfig | FL UnifiedAuditLogIngestionEnabled" in the Exchange Powershell. I disabled and re-enabled it, I've restarted the computer, the service, etc., numerous times across several days and still get the same issue.

  1. No data shows up in the dashboard which comes with this module. I can see in the logs that data is being transferred, but there is no sign of it in the Kibana dashboard.

Please advise

It looks like you're getting an error related to credentials/permissions.

Yes, but why after 1 hour? I finally resolved this auth issue by using the latest Filebeat download (Download Filebeat • Lightweight Log Analysis | Elastic), still not sure why it happened, but it's not a problem any more.

However, I'm still curious about the other issue, which is that absolutely no data shows up in the dashboard. "Module Status" shows that I'm correctly receiving data from the agent. Agent logs show that I'm connected just fine, but nothing appears in the dashboard, after days.

Everything is properly authorized on the client side, audit log is enabled, etc. What am I missing?

Do you see data in the discover tab? If so then I suspect the dashboard is filtering on something that's preventing the data from showing.

No data appears in the Discover tab.

The filebeat 0365 agent log shows hundreds of entries like the following:

2021-09-07T09:34:45.757-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {monitoring: {metrics: {beat:{cpu:{system:{ticks:454828,time:{ms:47}},total:{ticks:1506640,time:{ms:47},value:1506640},user:{ticks:1051812}},handles:{open:341},info:{ephemeral_id:4e14a072-e267-438e-b7a1-09c48c2ecff5,uptime:{ms:592350191},version:7.14.0},memstats:{gc_next:116944976,memory_alloc:58793600,memory_total:54508858752,rss:105738240},runtime:{goroutines:32}},filebeat:{harvester:{open_files:0,running:0}},libbeat:{config:{module:{running:1}},output:{events:{active:2531},read:{bytes:210}},pipeline:{clients:5,events:{active:4120}}},registrar:{states:{current:0}}}}}
2021-09-07T09:35:15.759-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {monitoring: {metrics: {beat:{cpu:{system:{ticks:454843,time:{ms:47}},total:{ticks:1506655,time:{ms:47},value:1506655},user:{ticks:1051812}},handles:{open:343},info:{ephemeral_id:4e14a072-e267-438e-b7a1-09c48c2ecff5,uptime:{ms:592380188},version:7.14.0},memstats:{gc_next:116944976,memory_alloc:58952160,memory_total:54509017312,rss:105738240},runtime:{goroutines:32}},filebeat:{harvester:{open_files:0,running:0}},libbeat:{config:{module:{running:1}},output:{events:{active:2531},read:{bytes:210}},pipeline:{clients:5,events:{active:4120}}},registrar:{states:{current:0}}}}}
2021-09-07T09:35:45.752-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {monitoring: {metrics: {beat:{cpu:{system:{ticks:454984,time:{ms:109}},total:{ticks:1506968,time:{ms:281},value:1506968},user:{ticks:1051984,time:{ms:172}}},handles:{open:344},info:{ephemeral_id:4e14a072-e267-438e-b7a1-09c48c2ecff5,uptime:{ms:592410215},version:7.14.0},memstats:{gc_next:116943616,memory_alloc:58471616,memory_total:54512770440,rss:105766912},runtime:{goroutines:34}},filebeat:{harvester:{open_files:0,running:0}},libbeat:{config:{module:{running:1}},output:{events:{active:2531},read:{bytes:210}},pipeline:{clients:5,events:{active:4120}}},registrar:{states:{current:0}}}}}

In the Exchange Powershell, the audit log is enabled:

Are you sure the credentials that you're using for Filebeat are good and have the right permissions. Its the only thing that I can presume based off the initial log message that you posted.

I finally re-installed everything on the server side from scratch, and it's now working. While installing, I discovered that I had not done "filebeat setup" (on the server side) properly. At the least, logstash/filebeat were missing pipelines and probably other similar issues. I wish I could say exactly what was wrong, but at least it's working now. Thank you for your help, I really appreciate you being there.

By using VM snapshots, rolling back, and duplicating my efforts, I have now confirmed that this was a server side pipeline issue.

I also confirmed that, in addition to installing/enabling the o365 filebeat module on the client side (Windows machine), the o365 filebeat module must be enabled on the server side. This is in spite of the fact that it looks like custom config in server side o365.yml is not relevant, since the remote Windows filebeat client does the actual connection to the 365 audit log servers).

Here are the key commands that I ran on the server side to get this working:

sudo filebeat modules enable o365
sudo filebeat setup -E output.elasticsearch.enabled=true -E output.logstash.enabled=false -E setup.kibana.host="http://<myhost>.local:5601" --pipelines --modules o365

(Some details of that second command may be modified; in my case, elasticsearch connectivity including cert/username/password is configured in filebeat.yml, but in that file enabled is set to false, so by setting it to true here, and toggling off logstash, the filebeat setup command works cleanly while adding the o365 pipeline. I guess the key takeaway here is this part: --pipelines --modules o365)

I still don't fully understand what's going on behind the scenes, but for future purposes, this was the fix that worked for me.