Filebeat (google_workspace module) retrieving partial events on initial startup

YuWatanabe · March 31, 2022, 10:24am

Hello I would like to ask question related to below topic.

I want to retrieve events backtracking up to 96 h for drive events .
However, I was only able to retrieve partial of them .

Looking at DEBUG log , the startTime was replaced by different value in the follow-on request .

Initial polling (startTime=2022-03-26T14%3A33%3A42Z)

2022-03-30T14:33:42.912Z        INFO    [input.httpjson-cursor] compat/compat.go:111    Input httpjson-cursor starting  {"id": "BBFA557C8D443451"}
2022-03-30T14:33:42.914Z        DEBUG   [input.httpjson-cursor] v2/cursor.go:27 new cursor: nothing loaded      {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:42.914Z        INFO    [input.httpjson-cursor] v2/input.go:112 Process another repeated request.       {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:42.914Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:84      template execution: falling back to default value       {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:42.914Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:97      template execution: evaluated template "2022-03-26T14:33:42Z"   {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:42.914Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:97      template execution: evaluated template "2022-03-26T14:33:42Z"   {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:42.914Z        DEBUG   [input.httpjson-cursor] v2/request.go:77        new request: v2.transformable{"header":http.Header{"Accept":[]string{"application/json"}, "User-Agent":[]string{"Elastic-Filebeat/7.17.1 (linux; amd64; 1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff; 2022-02-23 23:38:04 +0000 UTC)"}}, "url":(*url.URL)(0xc00080d3b0)}    {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:42.916Z        DEBUG   [input.httpjson-cursor.retryablehttp]   go-retryablehttp@v0.6.6/client.go:504   performing request%!(EXTRA string=method, string=POST, string=url, *url.URL=https://oauth2.googleapis.com/token)   {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:43.088Z        DEBUG   [input.httpjson-cursor.retryablehttp]   go-retryablehttp@v0.6.6/client.go:504   performing request%!(EXTRA string=method, string=GET, string=url, *url.URL=https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive?startTime=2022-03-26T14%3A33%3A42Z)        {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}```

Polling for page2 (startTime=2022-03-30T14%3A33%3A46Z)

2022-03-30T14:33:46.758Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:97      template execution: evaluated template "2022-03-30T14:33:46Z"   {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.758Z        DEBUG   [input.httpjson-cursor] v2/cursor.go:56 cursor.last_execution_datetime stored with 2022-03-30T14:33:46Z {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.759Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:97      template execution: evaluated template "2022-03-30T14:33:46Z"   {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.759Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:97      template execution: evaluated template "A:1648627960313000:-5113882487620209611:777491262838:C04ha7adf" {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.759Z        DEBUG   [input.httpjson-cursor] v2/request.go:77        new request: v2.transformable{"header":http.Header{"Accept":[]string{"application/json"}, "User-Agent":[]string{"Elastic-Filebeat/7.17.1 (linux; amd64; 1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff; 2022-02-23 23:38:04 +0000 UTC)"}}, "url":(*url.URL)(0xc004fd82d0)}    {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.759Z        DEBUG   [input.httpjson-cursor.retryablehttp]   go-retryablehttp@v0.6.6/client.go:504   performing request%!(EXTRA string=method, string=GET, string=url, *url.URL=https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive?pageToken=A%3A1648627960313000%3A-5113882487620209611%3A777491262838%3AC04ha7adf&startTime=2022-03-30T14%3A33%3A46Z)       {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}

As described in the doc , I believe the parameters except the pageToken should be the same as initial request when paginating.

In your follow-on request getting the next page of the report, enter the nextPageToken value in the pageToken query string.

Would there be anyway to set same values for the startTIme on all pagination requests ?

Filebeat I used is 7.16.1 and configuration used for my testing is on below gist.

gist.github.com

https://gist.github.com/yuwtennis/cb9117cfcb2eb5e6e69a83dab05383a5

filebeat.debug.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

This file has been truncated. show original

filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

This file has been truncated. show original

google_workspace.yml

# Module: google_workspace
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-google_workspace.html

- module: google_workspace
  saml:
    enabled: false
    # var.jwt_file: credentials.json
    # var.delegated_account: admin@example.com
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s

This file has been truncated. show original

There are more than three files. show original

Any advice is appreciated !

Thanks,
Yu watanabe

legoguy1000 · March 31, 2022, 12:43pm

@Marius_Iversen seems like the same issue I was experiencing with the Atlassian integrations.

legoguy1000 · March 31, 2022, 12:47pm

@YuWatanabe when looking at the debug logs, the events that are returned from the API, are the timestamps after the new start time or so they look like they're picking up where page 1 ended? I'm trying to gauge if the Google API ignores the start time parameter if the next page token is present.

YuWatanabe · March 31, 2022, 2:43pm

@legoguy1000

Looks like it's this. Published event in page2 is before startTime .

so they look like they're picking up where page 1 ended?

published event in page 2 , @timestamp": "2022-03-30T08:12:40.313Z"
request param for page 2 , &startTime=2022-03-30T14%3A33%3A46Z

I see the log like this.

gist.github.com

https://gist.github.com/yuwtennis/cb9117cfcb2eb5e6e69a83dab05383a5#file-log-snippet

filebeat.debug.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

This file has been truncated. show original

filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

This file has been truncated. show original

google_workspace.yml

# Module: google_workspace
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-google_workspace.html

- module: google_workspace
  saml:
    enabled: false
    # var.jwt_file: credentials.json
    # var.delegated_account: admin@example.com
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s

This file has been truncated. show original

There are more than three files. show original

These are the first 10 @timestamp s in the debug log.

[ssm-user@ip-10-0-10-12 filebeat]$ grep @timestamp filebeat | head -10
  "@timestamp": "2022-03-30T14:31:33.368Z",
  "@timestamp": "2022-03-30T14:27:08.492Z",
  "@timestamp": "2022-03-30T14:27:07.676Z",
  "@timestamp": "2022-03-30T14:23:32.720Z",
  "@timestamp": "2022-03-30T14:23:28.352Z",
  "@timestamp": "2022-03-30T14:23:18.592Z",
  "@timestamp": "2022-03-30T14:19:40.606Z",
  "@timestamp": "2022-03-30T14:15:09.675Z",
  "@timestamp": "2022-03-30T14:12:08.090Z",
  "@timestamp": "2022-03-30T14:02:48.930Z",
[ssm-user@ip-10-0-10-12 filebeat]$

These are the last 10 @timestamp s in the debug log.

[ssm-user@ip-10-0-10-12 ~]$ grep @timestamp filebeat/filebeat | tail -10
  "@timestamp": "2022-03-30T08:13:30.641Z",
  "@timestamp": "2022-03-30T08:13:26.285Z",
  "@timestamp": "2022-03-30T08:13:11.725Z",
  "@timestamp": "2022-03-30T08:13:10.832Z",
  "@timestamp": "2022-03-30T08:13:10.065Z",
  "@timestamp": "2022-03-30T08:13:01.041Z",
  "@timestamp": "2022-03-30T08:12:57.216Z",
  "@timestamp": "2022-03-30T08:12:56.669Z",
  "@timestamp": "2022-03-30T08:12:56.669Z",  <- last published event in page 1
  "@timestamp": "2022-03-30T08:12:40.313Z",  <- single event  published in page 2

legoguy1000 · March 31, 2022, 2:58pm

Sorry, can u actually look at the timestamp field from the Google event data, not filebeat's @timestamp? That field won't be properly updated until it's in Elasticsearch.

YuWatanabe · April 1, 2022, 2:31am

@legoguy1000
Yeah . Understood. I believe your talking about the id.time

[ssm-user@ip-10-0-10-12 filebeat]$ grep -oP '\\"id\\":{.*?}' filebeat | sed -e "s/CUSTOMERID/MASKED/g" | head -10
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:31:33.368Z\",\"uniqueQualifier\":\"-8678004808945565637\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:27:08.492Z\",\"uniqueQualifier\":\"7801333792512993495\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:27:07.676Z\",\"uniqueQualifier\":\"3073723659828988113\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:23:32.720Z\",\"uniqueQualifier\":\"7398511666600379244\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:23:28.352Z\",\"uniqueQualifier\":\"-4403104155775101877\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:23:18.592Z\",\"uniqueQualifier\":\"5667666403298765403\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:19:40.606Z\",\"uniqueQualifier\":\"-1473095349496659424\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:15:09.675Z\",\"uniqueQualifier\":\"4920740105111875611\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:12:08.090Z\",\"uniqueQualifier\":\"-1230078526531462997\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T14:02:48.930Z\",\"uniqueQualifier\":\"-6261765094560388394\"}
[ssm-user@ip-10-0-10-12 filebeat]$
[ssm-user@ip-10-0-10-12 filebeat]$
[ssm-user@ip-10-0-10-12 filebeat]$ grep -oP '\\"id\\":{.*?}' filebeat | sed -e "s/CUSTOMERID/MASKED/g" | tail -10
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:13:30.641Z\",\"uniqueQualifier\":\"-6682738832970861995\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:13:26.285Z\",\"uniqueQualifier\":\"-154616765354559012\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:13:11.725Z\",\"uniqueQualifier\":\"4496001752976791343\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:13:10.832Z\",\"uniqueQualifier\":\"-6706812910553500356\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:13:10.065Z\",\"uniqueQualifier\":\"681135857486310931\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:13:01.041Z\",\"uniqueQualifier\":\"-8521522634179346711\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:12:57.216Z\",\"uniqueQualifier\":\"1891074331260643701\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:12:56.669Z\",\"uniqueQualifier\":\"-2307473763742687566\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:12:56.669Z\",\"uniqueQualifier\":\"-2307473763742687566\"}
\"id\":{\"applicationName\":\"drive\",\"customerId\":\"MASKED\",\"time\":\"2022-03-30T08:12:40.313Z\",\"uniqueQualifier\":\"-5113882487620209611\"}
[ssm-user@ip-10-0-10-12 filebeat]$

I've updated the snippet as well for your reference.

gist.github.com

https://gist.github.com/yuwtennis/cb9117cfcb2eb5e6e69a83dab05383a5

filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

This file has been truncated. show original

google_workspace.yml

# Module: google_workspace
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-google_workspace.html

- module: google_workspace
  saml:
    enabled: false
    # var.jwt_file: credentials.json
    # var.delegated_account: admin@example.com
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s

This file has been truncated. show original

log-snippet

...
2022-03-30T14:33:46.755Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:97      template execution: evaluated template "2022-03-30T14:33:46Z"   {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.755Z        DEBUG   [input.httpjson-cursor] v2/cursor.go:56 cursor.last_execution_datetime stored with 2022-03-30T14:33:46Z {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.756Z        DEBUG   [processors]    processing/processors.go:203    Publish event: {
...
"id\":{\"applicationName\":\"drive\",\"customerId\":\"xxx\",\"time\":\"2022-03-30T08:12:57.216Z\",\"uniqueQualifier\":\"1891074331260643701\"}
...
}
2022-03-30T14:33:46.756Z        DEBUG   [input.httpjson-cursor] v2/value_tpl.go:97      template execution: evaluated template "2022-03-30T14:33:46Z"   {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}
2022-03-30T14:33:46.756Z        DEBUG   [input.httpjson-cursor] v2/cursor.go:56 cursor.last_execution_datetime stored with 2022-03-30T14:33:46Z {"id": "BBFA557C8D443451", "input_source": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive", "input_url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/drive"}

This file has been truncated. show original

legoguy1000 · April 1, 2022, 3:02pm

Just looking at what u posted, it seems the API is ignoring the start time parameter when the page token is present. So I think ur ok and there is no issue but can't say for certain with the truncated logs.

YuWatanabe · April 8, 2022, 10:23am

Thank you for the reply.

This is a problem for us because filebeat does not return expected number of events in initial startup.

gist.github.com

https://gist.github.com/yuwtennis/a66de5a236af38a512e612dd8bc7ce16#file-result-md

RESULT.md

## For go

[Source](https://github.com/yuwtennis/google-api-tutorials/blob/main/sdks/go/examples/quickstarts/admin/report/main.go)

```
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ go run examples/quickstarts/admin/report/main.go &> go.result
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ grep Page go.result
2022/04/08 18:35:22 Page: 1, Received 1000 items
2022/04/08 18:35:22 NextPageToken: A:1649401926164000:486361902784510281:777491262838:C04ha7adf
2022/04/08 18:35:24 Page: 2, Received 1000 items

This file has been truncated. show original

go.result

2022/04/08 18:54:35 Initializing client. duration: -24h0m0s, startTime: 2022-04-07T09:54:35Z
2022/04/08 18:54:37 Page: 1, Received 1000 items
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:13.630Z , Id.ApplicationName: drive, Id.UniqueQualifier: 2036629703250211577, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:12.613Z , Id.ApplicationName: drive, Id.UniqueQualifier: 3787910670504857267, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:11.741Z , Id.ApplicationName: drive, Id.UniqueQualifier: 7705104968783469441, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:04.447Z , Id.ApplicationName: drive, Id.UniqueQualifier: 182212403344434375, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:04.426Z , Id.ApplicationName: drive, Id.UniqueQualifier: 3745771873306198992, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:53:30.592Z , Id.ApplicationName: drive, Id.UniqueQualifier: -5462296767638075870, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:53:27.574Z , Id.ApplicationName: drive, Id.UniqueQualifier: 3922446045594138869, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:53:18.574Z , Id.ApplicationName: drive, Id.UniqueQualifier: 7932317726553000222, Num of Events: 1

This file has been truncated. show original

python.result

2022-04-

If you use google sdk , it crawls all the way to last 24 hours. I have attached the code in above gist.

go sdk returns 6,231 events, total 7 pages for last 24 hrs
python sdk returns 6,225 events, total 7 pages for last 24 hrs
filebeat returns 1,143 events, total 2 pages

So I thought I am missing some parameter in configuration to work this out but not sure how.

I also attached example debug output including full original event.

gist.github.com

https://gist.github.com/yuwtennis/a66de5a236af38a512e612dd8bc7ce16?permalink_comment_id=4125910#gistcomment-4125910

RESULT.md

## For go

[Source](https://github.com/yuwtennis/google-api-tutorials/blob/main/sdks/go/examples/quickstarts/admin/report/main.go)

See *go.result* for full log.

```
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ go run examples/quickstarts/admin/report/main.go &> go.result
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ grep Page go.result
2022/04/08 18:35:22 Page: 1, Received 1000 items

This file has been truncated. show original

filebeat

2022-04-08T19:06:16.932+0900	INFO	instance/beat.go:686	Home path: [/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64] Config path: [/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64] Data path: [/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64/data] Logs path: [/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64/logs] Hostfs Path: [/]
2022-04-08T19:06:16.940+0900	INFO	instance/beat.go:694	Beat ID: 8ec5cf5a-c5b3-42c3-8210-951f2d2c4d75
2022-04-08T19:06:16.943+0900	WARN	[add_cloud_metadata]	add_cloud_metadata/provider_aws_ec2.go:79	read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: connection refused. No token in the metadata request will be used.
2022-04-08T19:06:16.944+0900	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:101	add_cloud_metadata: hosting provider type not detected.
2022-04-08T19:06:16.945+0900	INFO	[seccomp]	seccomp/seccomp.go:101	Syscall filter could not be installed because the kernel does not support seccomp
2022-04-08T19:06:16.945+0900	INFO	[beat]	instance/beat.go:1040	Beat info	{"system_info": {"beat": {"path": {"config": "/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64", "data": "/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64/data", "home": "/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64", "logs": "/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64/logs"}, "type": "filebeat", "uuid": "8ec5cf5a-c5b3-42c3-8210-951f2d2c4d75"}}}
2022-04-08T19:06:16.945+0900	INFO	[beat]	instance/beat.go:1049	Build info	{"system_info": {"build": {"commit": "1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff", "libbeat": "7.17.1", "time": "2022-02-23T23:38:04.000Z", "version": "7.17.1"}}}
2022-04-08T19:06:16.945+0900	INFO	[beat]	instance/beat.go:1052	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.6"}}}
2022-04-08T19:06:16.946+0900	INFO	[beat]	instance/beat.go:1056	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-04-07T10:11:31+09:00","containerized":false,"name":"LAPTOP-IG41EBJ5","ip":["169.254.103.246/16","fe80::21dd:e1ba:24da:67f6/64","192.168.11.200/24","fe80::6d38:2fbd:ab18:88a9/64","172.18.192.1/20","fe80::64bd:3d67:ceca:9d99/64","127.0.0.1/8","::1/128","169.254.190.183/16","fe80::6450:ea67:2715:beb7/64","169.254.197.200/16","fe80::d9e5:d15d:b73d:c5c8/64","192.168.127.111/32","fe80::e87d:3f42:dd34:9b09/64"],"kernel_version":"4.4.0-19041-Microsoft","mac":["10:5b:ad:6e:12:e8","cc:30:80:36:4a:e4","00:15:5d:94:11:3c","10:5b:ad:6e:12:e7","22:5b:ad:6e:12:e7"],"os":{"type":"linux","family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"10 (buster)","major":10,"minor":0,"patch":0,"codename":"buster"},"timezone":"JST","timezone_offset_sec":32400,"id":"42ccda32846b42e3ba2b86267338236b"}}}
2022-04-08T19:06:16.947+0900	INFO	[beat]	instance/beat.go:1085	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64", "exe": "/home/y-watanabe/src/filebeat-7.17.1-linux-x86_64/filebeat", "name": "filebeat", "pid": 3755, "ppid": 8, "seccomp": {"mode":""}, "start_time": "2022-04-08T19:06:16.520+0900"}}}

This file has been truncated. show original

go.result

2022/04/08 18:54:35 Initializing client. duration: -24h0m0s, startTime: 2022-04-07T09:54:35Z
2022/04/08 18:54:37 Page: 1, Received 1000 items
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:13.630Z , Id.ApplicationName: drive, Id.UniqueQualifier: 2036629703250211577, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:12.613Z , Id.ApplicationName: drive, Id.UniqueQualifier: 3787910670504857267, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:11.741Z , Id.ApplicationName: drive, Id.UniqueQualifier: 7705104968783469441, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:04.447Z , Id.ApplicationName: drive, Id.UniqueQualifier: 182212403344434375, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:54:04.426Z , Id.ApplicationName: drive, Id.UniqueQualifier: 3745771873306198992, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:53:30.592Z , Id.ApplicationName: drive, Id.UniqueQualifier: -5462296767638075870, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:53:27.574Z , Id.ApplicationName: drive, Id.UniqueQualifier: 3922446045594138869, Num of Events: 1
2022/04/08 18:54:37 Activity Record: Id.Time: 2022-04-08T09:53:18.574Z , Id.ApplicationName: drive, Id.UniqueQualifier: 7932317726553000222, Num of Events: 1

This file has been truncated. show original

There are more than three files. show original

system · May 6, 2022, 12:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Does initial interval work on start up for google workspace module? Beats docker , filebeat	3	403	April 27, 2022
Google_workspace poll loads of data Beats filebeat	11	1091	May 29, 2021
Google Workplace no new events Beats filebeat	1	371	July 5, 2021
Filebeat missing log lines Beats filebeat	16	8352	July 5, 2017
Filebeat missing files Beats filebeat	19	5692	October 11, 2016

Filebeat (google_workspace module) retrieving partial events on initial startup

Related topics