# Duplicated Kubernetes Events

**URL:** <https://discuss.elastic.co/t/duplicated-kubernetes-events/174166>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [March 27, 2019, 3:34pm UTC](https://discuss.elastic.co/t/duplicated-kubernetes-events/174166 "2019-03-27T15:34:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\
**Post date:** [March 27, 2019, 3:34pm UTC](https://discuss.elastic.co/t/duplicated-kubernetes-events/174166/1 "2019-03-27T15:34:00Z")

</div>

Hi,

We're using the Kubernetes `event` metricset to ingest Kubernetes events into Elasticsearch, however, we've observed occurrences where the exact same event is being emitted to Elasticsearch multiple times, which can cause confusion when looking through events.

I understand Kubernetes does aggregate events so I could imagine the same event appearing multiple times if the count of that event is increasing, however in these instances it's the exact same vent:

 ![37](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e28b1f675db902fee8885c92da05b123f02193eb.png) ![52](https://us1.discourse-cdn.com/elastic/original/3X/f/7/f73e0302ca17040e6ab6c70602c4726c9c056c80.png)

In the example above you can see the same event was emitted about 30 minutes apart. I checked and Metricbeat had been running for a number of days (it hadn't restarted for example)

My question is if there's any sort of registry similar to Filebeat it can use to track which events have been processed?

Cheers

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 19, 2019, 8:37pm UTC](https://discuss.elastic.co/t/duplicated-kubernetes-events/174166/2 "2019-04-19T20:37:40Z")

</div>

Hi @Evesy,

This is actually unexpected. This could be caused by some retry, but it is strange to happen after half an hour, this could be a bug.

Did you see any error in metricbeat logs during this time?

What is the output configured in metricbeat?

How often does it happen?

---

<div class="post-metadata">

**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\
**Post date:** [April 23, 2019, 3:53pm UTC](https://discuss.elastic.co/t/duplicated-kubernetes-events/174166/3 "2019-04-23T15:53:51Z")

</div>

Hi @jsoriano,

Thanks for reaching out!

Our log profile is pretty consistent with the below entries being pretty common:

```auto
2019-04-23T14:20:42.127Z	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure

```

```auto
2019-04-23T15:42:05.372Z	ERROR	pipeline/output.go:121	Failed to publish events: 500 Internal Server Error: {"took":7,"ignored":false,"errors":true,"error":{"type":"export_exception","reason":"Exception when closing export bulk","caused_by":{"type":"export_exception","reason":"failed to flush export bulks","caused_by":{"type":"export_exception","reason":"bulk [default_local] reports failures when exporting documents"

2019-04-23T15:42:14.273Z	INFO	pipeline/output.go:95	Connecting to backoff(publish(elasticsearch(https://<HOST>:443)))

```

^^ We have a pretty busy cluster and occasionally the bulk queue fills up and sending applications backoff & retry

```auto
|2019-04-23T15:46:17.217Z|ERROR|kubernetes/watcher.go:254|kubernetes: Watching API error EOF|
|---|---|---|---|
|2019-04-23T15:46:17.217Z|INFO|kubernetes/watcher.go:238|kubernetes: Watching API for resource events|

```

^^ Not entirely sure about these, but we are running on GKE so occasionally the master will be unavailable for resizing, upgrades etc.

Below is our metricbeat config:

```auto
metricbeat.config.modules:
  # Mounted `metricbeat-daemonset-modules` configmap:
  path: ${path.config}/modules.d/*.yml
  # Reload module configs as they change:
  reload.enabled: false

processors:
  - add_cloud_metadata:

output.elasticsearch:
  hosts: ["<REDACTED>:443"]
  protocol: 'https'
  username: '<REDACTED>'
  password: "${ELASTICSEARCH_PASSWORD}"
  index: "kubernetes-%{+yyyy.MM.dd}"

setup.template.enabled: false

xpack.monitoring.enabled: true

```

And the only file in `modules.d`:

```auto
- module: kubernetes
  metricsets:
    - event

```

It's happening pretty frequently, I can fairly easily manually find examples of where it's happening at any given time -- I haven't been able to create a Kibana query though for identical documents (except timestamp), which would give me an exact figure

Cheers,  
Mike

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [April 24, 2019, 9:51am UTC](https://discuss.elastic.co/t/duplicated-kubernetes-events/174166/4 "2019-04-24T09:51:07Z")

</div>

We are investigating an issue on reconnections when watching for Kubernetes events, this could cause your duplicated events. I have opened an issue to keep track of this: [https://github.com/elastic/beats/issues/11917](https://github.com/elastic/beats/issues/11917)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2019, 9:51am UTC](https://discuss.elastic.co/t/duplicated-kubernetes-events/174166/5 "2019-05-22T09:51:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.