Hi im using Elastic Agent version 8.6 installed on one host, applying a policy with Google Workspace module enabled retrieving all type of events from our Google Workspace tenant.
It works perfectly as alert rules give me insights of our tenant, but every day our cluster keeps receiving repeated events, so I get repeated alerts for a event that ocurred a lot of days ago.
As you can see in the image i received the event with an specific "event.id" 7 days in a row
Thanks!
Thanks for letting us know @German_Bravo, that does not look like something that is intended. We are planning on doing some updates on this integration, and I will look into this first.
If we manage to find the issue, I will update it here. Once the change has been merged you should automatically get access to it, and a "update" icon should appear for your google workspace integration if everything goes as it should.
@German_Bravo Would you be able to provide a JSON copy of two of these duplicated events? Apparently we already handle duplication of this kind, and this would usually only happen if there are changes.
We would be interested in seeing what actually changed between them.
Of course @Marius_Iversen there you have 2 events that are repeated.
This event keeps repeating two times a day for example, but it is not the unique one.
JSON 1:
{
"_index": ".ds-logs-google_workspace.admin-default-2023.01.19-000008",
"_id": "qcgCPpYbYXRN9LyanGrVYS32nVA=",
"_version": 1,
"_score": 0,
"_source": {
"agent": {
"name": "SRV-AWS-Eventcolector",
"id": "a594459d-0d54-4c0d-879e-f19d22c70a8b",
"type": "filebeat",
"ephemeral_id": "a1a87700-fbbe-41db-b3e3-99340c3a91e0",
"version": "8.5.2"
},
"elastic_agent": {
"id": "a594459d-0d54-4c0d-879e-f19d22c70a8b",
"version": "8.5.2",
"snapshot": false
},
"source": {
"geo": {
"continent_name": "South America",
"region_iso_code": "UY-MO",
"city_name": "Montevideo",
"country_iso_code": "UY",
"country_name": "Uruguay",
"region_name": "Departamento de Montevideo",
"location": {
"lon": -98.7654,
"lat": -98.7654
}
},
"as": {
"number": 6057,
"organization": {
"name": "Administracion Nacional de Telecomunicaciones"
}
},
"ip": "192.0.92.5",
"user": {
"domain": "altered-domain.com",
"name": "name.lastname",
"id": "104861331778014733922",
"email": "name.lastname@altered-domain.com"
}
},
"tags": [
"forwarded",
"google-workspace-admin"
],
"cloud": {
"availability_zone": "us-east-1b",
"image": {
"id": "ami-03445544beb6afeda"
},
"instance": {
"id": "i-01f017c6bf13edcb3"
},
"provider": "aws",
"service": {
"name": "EC2"
},
"machine": {
"type": "t3a.xlarge"
},
"region": "us-east-1",
"account": {
"id": "956980917244"
}
},
"input": {
"type": "httpjson"
},
"@timestamp": "2023-01-19T14:42:22.933Z",
"ecs": {
"version": "8.5.0"
},
"related": {
"ip": [
"192.0.92.5"
],
"user": [
"name.lastname",
"fernandinho.almada"
]
},
"google_workspace": {
"actor": {
"type": "USER"
},
"kind": "admin#reports#activity",
"admin": {
"user": {
"email": "fernandinho.almada@altered-domain.com"
}
},
"event": {
"type": "USER_SETTINGS"
}
},
"data_stream": {
"namespace": "default",
"type": "logs",
"dataset": "google_workspace.admin"
},
"organization": {
"id": "C025zio51"
},
"event": {
"agent_id_status": "verified",
"ingested": "2023-01-19T15:53:20Z",
"provider": "admin",
"created": "2023-01-19T15:53:19.285Z",
"kind": "event",
"action": "CHANGE_PASSWORD",
"id": "-6264539690912704257",
"category": [
"iam"
],
"type": [
"change",
"user"
],
"dataset": "google_workspace.admin"
},
"user": {
"domain": "altered-domain.com",
"name": "name.lastname",
"id": "104861331778014733922",
"email": "name.lastname@altered-domain.com",
"target": {
"domain": "altered-domain.com",
"name": "fernandinho.almada",
"email": "fernandinho.almada@altered-domain.com"
}
}
},
"fields": {
"elastic_agent.version": [
"8.5.2"
],
"event.category": [
"iam"
],
"source.user.email": [
"name.lastname@altered-domain.com"
],
"cloud.availability_zone": [
"us-east-1b"
],
"source.user.name.text": [
"name.lastname"
],
"user.target.email": [
"fernandinho.almada@altered-domain.com"
],
"source.geo.region_name": [
"Departamento de Montevideo"
],
"google_workspace.actor.type": [
"USER"
],
"source.ip": [
"192.0.92.5"
],
"agent.name": [
"SRV-AWS-Eventcolector"
],
"source.geo.region_iso_code": [
"UY-MO"
],
"user.target.name.text": [
"fernandinho.almada"
],
"event.agent_id_status": [
"verified"
],
"event.kind": [
"event"
],
"google_workspace.event.type": [
"USER_SETTINGS"
],
"source.geo.city_name": [
"Montevideo"
],
"cloud.region": [
"us-east-1"
],
"user.id": [
"104861331778014733922"
],
"input.type": [
"httpjson"
],
"data_stream.type": [
"logs"
],
"user.target.name": [
"fernandinho.almada"
],
"tags": [
"forwarded",
"google-workspace-admin"
],
"related.user": [
"name.lastname",
"fernandinho.almada"
],
"cloud.machine.type": [
"t3a.xlarge"
],
"cloud.provider": [
"aws"
],
"event.provider": [
"admin"
],
"cloud.service.name": [
"EC2"
],
"agent.id": [
"a594459d-0d54-4c0d-879e-f19d22c70a8b"
],
"ecs.version": [
"8.5.0"
],
"event.created": [
"2023-01-19T15:53:19.285Z"
],
"google_workspace.admin.user.email": [
"fernandinho.almada@altered-domain.com"
],
"organization.id": [
"C025zio51"
],
"agent.version": [
"8.5.2"
],
"source.user.name": [
"name.lastname"
],
"source.as.number": [
6057
],
"user.name": [
"name.lastname"
],
"source.geo.location": [
{
"coordinates": [
-98.7654,
-98.7654
],
"type": "Point"
}
],
"cloud.instance.id": [
"i-01f017c6bf13edcb3"
],
"agent.type": [
"filebeat"
],
"event.module": [
"google_workspace"
],
"user.email": [
"name.lastname@altered-domain.com"
],
"related.ip": [
"192.0.92.5"
],
"source.geo.country_iso_code": [
"UY"
],
"user.target.domain": [
"altered-domain.com"
],
"source.user.id": [
"104861331778014733922"
],
"elastic_agent.snapshot": [
false
],
"user.domain": [
"altered-domain.com"
],
"source.as.organization.name.text": [
"Administracion Nacional de Telecomunicaciones"
],
"elastic_agent.id": [
"a594459d-0d54-4c0d-879e-f19d22c70a8b"
],
"data_stream.namespace": [
"default"
],
"source.as.organization.name": [
"Administracion Nacional de Telecomunicaciones"
],
"source.geo.continent_name": [
"South America"
],
"google_workspace.kind": [
"admin#reports#activity"
],
"cloud.image.id": [
"ami-03445544beb6afeda"
],
"event.action": [
"CHANGE_PASSWORD"
],
"event.ingested": [
"2023-01-19T15:53:20.000Z"
],
"@timestamp": [
"2023-01-19T14:42:22.933Z"
],
"cloud.account.id": [
"956980917244"
],
"data_stream.dataset": [
"google_workspace.admin"
],
"event.type": [
"change",
"user"
],
"agent.ephemeral_id": [
"a1a87700-fbbe-41db-b3e3-99340c3a91e0"
],
"source.user.domain": [
"altered-domain.com"
],
"source.geo.country_name": [
"Uruguay"
],
"event.id": [
"-6264539690912704257"
],
"event.dataset": [
"google_workspace.admin"
],
"user.name.text": [
"name.lastname"
]
}
}
JSON 2:
{
"_index": ".ds-logs-google_workspace.admin-default-2023.01.19-000008",
"_id": "Bq3vDX+YdBPKMO45DIrHAu7KQiE=",
"_version": 1,
"_score": 0,
"_source": {
"agent": {
"name": "SRV-AWS-Eventcolector",
"id": "a594459d-0d54-4c0d-879e-f19d22c70a8b",
"type": "filebeat",
"ephemeral_id": "a1a87700-fbbe-41db-b3e3-99340c3a91e0",
"version": "8.5.2"
},
"elastic_agent": {
"id": "a594459d-0d54-4c0d-879e-f19d22c70a8b",
"version": "8.5.2",
"snapshot": false
},
"source": {
"geo": {
"continent_name": "South America",
"region_iso_code": "UY-MO",
"city_name": "Montevideo",
"country_iso_code": "UY",
"country_name": "Uruguay",
"region_name": "Departamento de Montevideo",
"location": {
"lon": -98.7654,
"lat": -98.7654
}
},
"as": {
"number": 6057,
"organization": {
"name": "Administracion Nacional de Telecomunicaciones"
}
},
"ip": "192.0.92.5",
"user": {
"domain": "altered-domain.com",
"name": "name.lastname",
"id": "104861331778014733922",
"email": "name.lastname@altered-domain.com"
}
},
"tags": [
"forwarded",
"google-workspace-admin"
],
"cloud": {
"image": {
"id": "ami-03445544beb6afeda"
},
"availability_zone": "us-east-1b",
"instance": {
"id": "i-01f017c6bf13edcb3"
},
"provider": "aws",
"machine": {
"type": "t3a.xlarge"
},
"service": {
"name": "EC2"
},
"region": "us-east-1",
"account": {
"id": "956980917244"
}
},
"input": {
"type": "httpjson"
},
"@timestamp": "2023-01-19T14:42:22.933Z",
"ecs": {
"version": "8.5.0"
},
"related": {
"ip": [
"192.0.92.5"
],
"user": [
"name.lastname",
"fernandinho.almada"
]
},
"google_workspace": {
"actor": {
"type": "USER"
},
"kind": "admin#reports#activity",
"admin": {
"old_value": "false",
"user": {
"email": "fernandinho.almada@altered-domain.com"
},
"new_value": "true"
},
"event": {
"type": "USER_SETTINGS"
}
},
"data_stream": {
"namespace": "default",
"type": "logs",
"dataset": "google_workspace.admin"
},
"organization": {
"id": "C025zio51"
},
"event": {
"agent_id_status": "verified",
"ingested": "2023-01-19T15:53:20Z",
"provider": "admin",
"created": "2023-01-19T15:53:19.285Z",
"kind": "event",
"action": "CHANGE_PASSWORD_ON_NEXT_LOGIN",
"id": "-6264539690912704257",
"category": [
"iam"
],
"type": [
"change",
"user"
],
"dataset": "google_workspace.admin"
},
"user": {
"domain": "altered-domain.com",
"name": "name.lastname",
"id": "104861331778014733922",
"email": "name.lastname@altered-domain.com",
"target": {
"domain": "altered-domain.com",
"name": "fernandinho.almada",
"email": "fernandinho.almada@altered-domain.com"
}
}
},
"fields": {
"elastic_agent.version": [
"8.5.2"
],
"event.category": [
"iam"
],
"source.user.email": [
"name.lastname@altered-domain.com"
],
"cloud.availability_zone": [
"us-east-1b"
],
"source.user.name.text": [
"name.lastname"
],
"user.target.email": [
"fernandinho.almada@altered-domain.com"
],
"source.geo.region_name": [
"Departamento de Montevideo"
],
"google_workspace.actor.type": [
"USER"
],
"source.ip": [
"192.0.92.5"
],
"agent.name": [
"SRV-AWS-Eventcolector"
],
"source.geo.region_iso_code": [
"UY-MO"
],
"user.target.name.text": [
"fernandinho.almada"
],
"event.agent_id_status": [
"verified"
],
"event.kind": [
"event"
],
"google_workspace.event.type": [
"USER_SETTINGS"
],
"source.geo.city_name": [
"Montevideo"
],
"cloud.region": [
"us-east-1"
],
"user.id": [
"104861331778014733922"
],
"google_workspace.admin.new_value": [
"true"
],
"input.type": [
"httpjson"
],
"data_stream.type": [
"logs"
],
"user.target.name": [
"fernandinho.almada"
],
"tags": [
"forwarded",
"google-workspace-admin"
],
"related.user": [
"name.lastname",
"fernandinho.almada"
],
"cloud.machine.type": [
"t3a.xlarge"
],
"cloud.provider": [
"aws"
],
"event.provider": [
"admin"
],
"cloud.service.name": [
"EC2"
],
"agent.id": [
"a594459d-0d54-4c0d-879e-f19d22c70a8b"
],
"ecs.version": [
"8.5.0"
],
"event.created": [
"2023-01-19T15:53:19.285Z"
],
"google_workspace.admin.user.email": [
"fernandinho.almada@altered-domain.com"
],
"organization.id": [
"C025zio51"
],
"agent.version": [
"8.5.2"
],
"source.user.name": [
"name.lastname"
],
"source.as.number": [
6057
],
"user.name": [
"name.lastname"
],
"source.geo.location": [
{
"coordinates": [
-98.7654,
-98.7654
],
"type": "Point"
}
],
"cloud.instance.id": [
"i-01f017c6bf13edcb3"
],
"agent.type": [
"filebeat"
],
"event.module": [
"google_workspace"
],
"user.email": [
"name.lastname@altered-domain.com"
],
"related.ip": [
"192.0.92.5"
],
"source.geo.country_iso_code": [
"UY"
],
"user.target.domain": [
"altered-domain.com"
],
"source.user.id": [
"104861331778014733922"
],
"elastic_agent.snapshot": [
false
],
"user.domain": [
"altered-domain.com"
],
"google_workspace.admin.old_value": [
"false"
],
"source.as.organization.name.text": [
"Administracion Nacional de Telecomunicaciones"
],
"elastic_agent.id": [
"a594459d-0d54-4c0d-879e-f19d22c70a8b"
],
"data_stream.namespace": [
"default"
],
"source.as.organization.name": [
"Administracion Nacional de Telecomunicaciones"
],
"source.geo.continent_name": [
"South America"
],
"google_workspace.kind": [
"admin#reports#activity"
],
"cloud.image.id": [
"ami-03445544beb6afeda"
],
"event.action": [
"CHANGE_PASSWORD_ON_NEXT_LOGIN"
],
"event.ingested": [
"2023-01-19T15:53:20.000Z"
],
"@timestamp": [
"2023-01-19T14:42:22.933Z"
],
"cloud.account.id": [
"956980917244"
],
"data_stream.dataset": [
"google_workspace.admin"
],
"event.type": [
"change",
"user"
],
"agent.ephemeral_id": [
"a1a87700-fbbe-41db-b3e3-99340c3a91e0"
],
"source.user.domain": [
"altered-domain.com"
],
"source.geo.country_name": [
"Uruguay"
],
"event.id": [
"-6264539690912704257"
],
"event.dataset": [
"google_workspace.admin"
],
"user.name.text": [
"name.lastname"
]
}
}
I altered some values for security reasons.
Okay so the same event is ingested 2 times a day, but then the next day its the same event ID, but a different time?
Are we sure this is not some automated procedure to update passwords? Looking at the event the action performed is a user password change.
If google workspace is actually creating these events each day, then its not much we can do to remove them, as we can only show what we receive from the Google Workspace API.
Exactly @Marius_Iversen , but it happens with a lot of events from google_workspace.admin data stream. As you can see the event id repeats, but a different time in timestamp is shown.
I am sure because I have other events like route creations in gmail that triggers some alerts in Elastic Security, or a transfer in google drive from one user to another that only happened 1 week ago, but it keeps repeating once a day.
Hi
Any update on this case? I keep receiveing repeated events, and yesterday, as i received new alerts on custom Gmail routes creations, they repeated today and im getting more than 15 alerts everyday for the same events that ocurred 1 week to 1 day ago.
Thanks!
I looked at the two provided JSON events and the @timestamp (which is Google Workspace's id.time) and the event.id ( id.uniqueQualifier) appear to be the same in both events. Based on this I would definitely say that these two documents were ingested at the same time and came from the same source Google Workspace activity.
jq -c '[._id, ._source.agent.ephemeral_id, ._source."@timestamp", ._source.event.id, ._source.event.ingested, ._source.event.created]' /tmp/events
["qcgCPpYbYXRN9LyanGrVYS32nVA=","a1a87700-fbbe-41db-b3e3-99340c3a91e0","2023-01-19T14:42:22.933Z","-6264539690912704257","2023-01-19T15:53:20Z","2023-01-19T15:53:19.285Z"]
["Bq3vDX+YdBPKMO45DIrHAu7KQiE=","a1a87700-fbbe-41db-b3e3-99340c3a91e0","2023-01-19T14:42:22.933Z","-6264539690912704257","2023-01-19T15:53:20Z","2023-01-19T15:53:19.285Z"]
It is normal for multiple events to share the same @timestamp and event.id because when a single google workspace activity report contains multiple events the integration splits the report into multiple Elasticsearch documents (one for each item in the events array).
It is not normal for the same activity report to be ingested repeatedly (like we see in the first screenshot where the event.ingested dates are different). @German_Bravo can you confirm the version of the Google Workspace integration that you are running is the latest available version (2.2.1 at time of posting)? See the changelog at https://docs.elastic.co/en/integrations/google_workspace#changelog.
Sorry I had an old version of Google Workspace module, last week I upgraded the module and now i dont see repeated alerts.
Fixed working version: 2.2.1
Thanks a lot!
Greetings!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.