Gsuite module in Filebeat not generating logs

Elastic Stack Beats
1

Gsuite module in Filebeat not pushing the logs to Elasticsearch. I have configured the service account and domain wide delegation and setup the proper scope required. Still it throws back a error code 401 which says cannot process the request.

2

Can you post the configuration files and some logs?

3 
    - module: gsuite
  saml:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxxxx@xxxx.gserviceaccount.com"

user_accounts:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxxxx@xxxx.gserviceaccount.com"
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s
    # var.user_key: all
    # var.interval: 2h
  login:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: ""xxxxx@xxxx.gserviceaccount.com"

This is the gsuite.yml file in /etc/filebeat/modules.d.

Logs :

Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.113Z INFO [httpjson] httpjson/input.go:130 Initialized httpjson input. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/login"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.116Z INFO [httpjson] httpjson/input.go:130 Initialized httpjson input. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/saml"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.118Z INFO [httpjson] httpjson/input.go:130 Initialized httpjson input. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/user_accounts"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.143Z INFO [httpjson] httpjson/input.go:140 httpjson input worker has started. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/login"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.150Z INFO [httpjson] httpjson/input.go:140 httpjson input worker has started. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/saml"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.164Z INFO [httpjson] httpjson/input.go:140 httpjson input worker has started. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/user_accounts"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.361Z ERROR [httpjson] httpjson/input.go:145 http request was unsuccessful with a status code 401 {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/login"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.362Z INFO [httpjson] httpjson/input.go:146 httpjson input worker has stopped. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/login"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.370Z ERROR [httpjson] httpjson/input.go:145 http request was unsuccessful with a status code 401 {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/saml"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.371Z INFO [httpjson] httpjson/input.go:146 httpjson input worker has stopped. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/saml"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.377Z ERROR [httpjson] httpjson/input.go:145 http request was unsuccessful with a status code 401 {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/user_accounts"}
Nov 04 12:58:43 ip-10-0-0-143 filebeat[4731]: 2020-11-04T12:58:43.378Z INFO [httpjson] httpjson/input.go:146 httpjson input worker has stopped. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/user_accounts"}
Nov 04 13:41:24 ip-10-0-0-143 filebeat[5740]: 2020-11-04T13:41:24.626Z INFO [httpjson] httpjson/input.go:130 Initialized httpjson input. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/saml"}
Nov 04 13:41:24 ip-10-0-0-143 filebeat[5740]: 2020-11-04T13:41:24.629Z INFO [httpjson] httpjson/input.go:140 httpjson input worker has started. {"url": "https://www.googleapis.com/admin/reports/v1/activity/users/all/applications/saml"}

This is the logs which shows 401 status code. I have confirmed scope is set correctly and other steps as per the documentation for the module.

4

Is that correct? Double quotes "" at the beginning of the email?

5

This I put the service account email which was created and provide access to scope. I did not want to reveal it.

And Yes there are double quotes . Is that an issue ?

6

I mean that you write double quotes twice at the beginning of the email

7

No. That was accidental. In my real code I didn't do it. Thank you for pointing it out. I wanted to know if there was any way I could manually check my access to Admin API is set correctly using command line. Error 401 shows invalid request.

8

Hello @Edwin.v! Is the delegated account an admin account?

In 7.9 docs this was not specified clearly, which is fixed in 7.10 ones already https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-gsuite.html

9

No. I am using the service account. I tried my admin account instead and it's working. I see the logs coming now. I was about to post this.

Only thing I am concerned now about is getting a dashboard access. As I don't have any specific option for this. And I am having some difficulty creating an alert for the login alerts

10

Awesome!

I am afraid right now there is no default dashboard provided for Gsuite module, but I opened an issue to keep track of this [Filebeat][Gsuite] Create default dashboard · Issue #22441 · elastic/beats · GitHub

11

Great. Thanks

12

@marc.guasch. Inside the gsuite module : saml, user_accounts, login works fine without any issues. But when I enable admin or drive or groups. I am getting the following error.

2020-11-06T04:42:56.962Z#011ERROR#011cfgfile/reload.go:273#011Error loading config from file '/etc/filebeat/modules.d/gsuite.yml', error invalid config: yaml: line 38: did not find expected key.

I tried every option individually still the same issue. Any idea what the issue can be ?

13

Can you paste the config to see what can be the issue?

14

Yes. This is the config:

# Module: gsuite
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.9/filebeat-module-gsuite.html

- module: gsuite
  saml:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxx@g.com"
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s
    # var.user_key: all
    # var.interval: 2h
  user_accounts:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxx@g.com"
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s
    # var.user_key: all
    # var.interval: 2h
  login:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxx@g.com"
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s
    # var.user_key: all
    # var.interval: 2h
  admin:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxx@g.com"
    # var.initial_interval: 4h
    # var.http_client_timeout: 60s
    # var.user_key: all
    # var.interval: 2h
  drive:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxx@g.com"
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s
    # var.user_key: all
    # var.interval: 2h
  groups:
    enabled: true
    var.jwt_file: "/etc/filebeat/credentials.json"
    var.delegated_account: "xxx@g.com"
    # var.initial_interval: 24h
    # var.http_client_timeout: 60s
    # var.user_key: all
    # var.interval: 2h
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.