The first production release of o365beat is now available as v1.2.0 on github.
O365beat is an open source log shipper used to fetch Office 365 audit logs from the Office 365 Management Activity API and forward them with all the flexibility and capability provided by the beats platform (specifically, libbeat).
The latest release includes updated documentation and a new ECS field mapping processor in the default config file to map the raw API-provided events to Elastic Common Schema (ECS) fields. This allows o365beat to work with standard Kibana dashboards, including capabilities in Elastic SIEM.
There is still a lot on the to-do list and probably a few bugs. Please open an issue or submit a pull request if you notice any problems in testing or production.
Please contact us if we can help in any way. Thanks!
Andrew - I'll update the docs to explain that a bit more. The o365beat-registry.json file is just the registry keeping track of the state of the downloads from the API to prevent repeatedly downloading the same events. If you look at the code you'll see it's just a date right now, the latest content blob retrieved.
That error message could probably be changed as well, to make it clearer. When you kick off the beat the first time, the registry file won't exist, so you'll see that message and then the beat will create it for the first time. If it doesn't find the registry file, it just pulls the most recent 7 days (the max available) from the API.
Thanks for the feedback! I'll make those updates before the next release, let me know if there's anything else I can do to help.
Happy to help! Those 400 errors can come from a number of steps in the process, and some of the debug output does have sensitive-ish info in it. I'll post an example of what I see, with that stuff redacted, perhaps that'll help find the exact spot things are failing for you:
When I run it (on Windows in this case), using the following command in powershell:
# /snip/
2019-09-26T14:32:29.524-0500 INFO beater/o365beat.go:414 o365beat is running! Hit CTRL-C to stop it.
2019-09-26T14:32:29.524-0500 DEBUG [service] service/service_windows.go:72 Windows is interactive: true
2019-09-26T14:32:29.524-0500 INFO beater/o365beat.go:156 getting content subscriptions
2019-09-26T14:32:29.524-0500 DEBUG [api] beater/o365beat.go:157 getting content subscriptions from https://manage.office.com/api/v1.0/$O365_DIRECTORY_ID/activity/feed/subscriptions/list
2019-09-26T14:32:29.524-0500 INFO beater/o365beat.go:102 auth nil or expired, re-authenticating
2019-09-26T14:32:29.524-0500 INFO beater/o365beat.go:127 authenticating via https://login.microsoftonline.com/[$O365_TENANT_DOMAIN]/oauth2/token?api-version=1.0
2019-09-26T14:32:29.525-0500 DEBUG [auth] beater/o365beat.go:135 sending auth req: &{POST https://login.microsoftonline.com/[$O365_TENANT_DOMAIN]/oauth2/token?api-version=1.0 HTTP/1.1 1 1 map[Content-Type:[application/x-www-form-urlencoded]] {0x$HEX_NUMBER} 0x$HEX_NUMBER $DECIMAL_NUMBER [] false login.microsoftonline.com map[] map[] <nil> map[] <nil> <nil> <nil> <nil>}
2019-09-26T14:32:29.988-0500 DEBUG [auth] beater/o365beat.go:148 got auth info: {Bearer $NUMBER $NUMBER $NUMBER https://manage.office.com $MANY_LINE_AUTH_TOKEN}
# *** after first auth, it checks your content subscriptions, and subscribes you if they're disabled ***
# *** this can take time, and it's an issue that i don't check for it here: ***
2019-09-26T14:32:29.988-0500 DEBUG [api] beater/o365beat.go:111 issuing api request: https://manage.office.com/api/v1.0/$O365_DIRECTORY_ID/activity/feed/subscriptions/list?PublisherIdentifier=$O365_DIRECTORY_ID
2019-09-26T14:32:30.535-0500 DEBUG [api] beater/o365beat.go:168 got these subscriptions: [map[contentType:Audit.AzureActiveDirectory status:enabled webhook:] map[contentType:Audit.Exchange status:enabled webhook:] map[contentType:Audit.General status:enabled webhook:] map[contentType:Audit.SharePoint status:enabled webhook:]]
# *** and here's the registry check ... ***
# *** with that misleading error message you pointed out; we expect it when running the first time: ***
2019-09-26T14:32:30.536-0500 DEBUG [beat] beater/o365beat.go:387 getting registry info from ./o365beat-registry.json
2019-09-26T14:32:30.536-0500 WARN beater/o365beat.go:390 error parsing registry file, may not exist.
#*** starts checking at the beginning of time, since it found no registry file: ***
2019-09-26T14:32:30.536-0500 DEBUG [beat] beater/o365beat.go:342 polling since 0001-01-01 00:00:00 +0000 UTC
# *** then starts getting content locations and we're off to the races: ***
2019-09-26T14:32:30.536-0500 INFO beater/o365beat.go:260 getting all available content between 2019-09-19 14:32:30.5361932 -0500 CDT m=-604795.719550099 and 2019-09-26 14:32:30.5361932 -0500 CDT m=+4.280449901
2019-09-26T14:32:30.536-0500 DEBUG [api] beater/o365beat.go:261 getting all available content from https://manage.office.com/api/v1.0/$O365_DIRECTORY_ID/activity/feed/subscriptions/content between 2019-09-19 14:32:30.5361932 -0500 CDT m=-604795.719550099 and 2019-09-26 14:32:30.5361932 -0500 CDT m=+4.280449901
2019-09-26T14:32:30.536-0500 INFO beater/o365beat.go:198 getting available content of type Audit.AzureActiveDirectory between 2019-09-19 14:32:30.5361932 -0500 CDT m=-604795.719550099 and 2019-09-20 14:32:30.5361932 -0500 CDT m=-518395.719550099
2019-09-26T14:32:30.536-0500 DEBUG [api] beater/o365beat.go:199 getting available content from https://manage.office.com/api/v1.0/$O365_DIRECTORY_ID/activity/feed/subscriptions/content of type Audit.AzureActiveDirectory between 2019-09-19 14:32:30.5361932 -0500 CDT m=-604795.719550099 and 2019-09-20 14:32:30.5361932 -0500 CDT m=-518395.719550099
2019-09-26T14:32:30.537-0500 DEBUG [api] beater/o365beat.go:111 issuing api request: https://manage.office.com/api/v1.0/$O365_DIRECTORY_ID/activity/feed/subscriptions/content?PublisherIdentifier=$O365_DIRECTORY_ID&contentType=Audit.AzureActiveDirectory&endTime=2019-09-20T19%3A32%3A30&startTime=2019-09-19T19%3A32%3A30
2019-09-26T14:32:30.943-0500 INFO beater/o365beat.go:249 got 5 available content locations of type Audit.AzureActiveDirectory between 2019-09-19 14:32:30.5361932 -0500 CDT m=-604795.719550099 and 2019-09-20 14:32:30.5361932 -0500 CDT m=-518395.719550099
2019-09-26T14:32:30.944-0500 DEBUG [api] beater/o365beat.go:253 got this available content:
[map[contentCreated:2019-09-20T00:10:31.642Z contentExpiration:2019-09-26T19:46:03.784Z contentId:[$CONTENT_ID] contentType:Audit.AzureActiveDirectory contentUri:https://manage.office.com/api/v1.0/$O365_DIRECTORY_ID/activity/feed/audit/[$CONTENT_BLOB_ADDRESS]]
# /snip/
]
# *** with /snips/ for brevity ***
# /snip/
It could be that your subscriptions to the various event types are taking a while to "kick in" (see the first note in this section of the documentation) ... it can take up to 12 hours for the subscriptions to start creating content. If this turns out to be the problem (that is, if you try to run it today and it works), I'll make an issue and implement some logic to show a more informative message.
I wanted to post an update summarizing what we discovered when debugging @awataszko's issue: in short, the auto-subscription logic was broken, requiring manual subscriptions to the API content types before running the beat.
This is fixed in version 1.3.0, released this afternoon (27 Sep 19). From the release notes: "This release fixes a bug in the auto-subscription logic (see the issue on github) that left some users unable to launch the beat without manually subscribing to content types using curl or Invoke-WebRequest (or similar).
Documentation is also updated based on some user feedback, otherwise the functionality is the same as v1.2.0.
Please open an issue or pull request if you notice any bugs or deficiencies, and contact us if you need assistance with o365beat, logging, security, IR, or any other services we offer."
Hi,
Do you have any common visualizations or dashboard with this?
the /usr/share/o365beat/kibana dir is empty and when i run ./o365beat setup it errors and says the kibana/7 directory is empty... in fact the /7 directory is totally missing.
What i also managed to get around is my kibana config uses the machines ip instead of localhost and when running o365beats i have to change from the ip tolocalhost in the config the first time, then after beats is running, change it back to ip and restart kibana service. which is kinda ok for now as im just testing it, but im pulling data from my 365 subscription, and trying to get some vis or dashboards up.
Thanks for the question! The current version does not have any visualizations or dashboards built in, but it's a commonly requested feature and I will be rolling out a new version this weekend that includes a few. Is there anything in particular you'd like to see? The goal is for this to be as complete as possible, including a module implementation.
As far as the configuration (IP vs. localhost), I'm not sure I understand the question. If there's an error with a certain setup, could you send me the config that works and the config that fails? You can send them (without your keys) to chris@counteractive.net, I'd be happy to help sort it out.
Thanks again, looking forward to getting everything up and running for you.
That error message could probably be changed as well, to make it clearer. When you kick off the beat the first time, the registry file won't exist, so you'll see that message and then the beat will create it for the first time. If it doesn't find the registry file, it just pulls the most recent 7 days (the max available) from the API.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
