# Remove duplications in Kibana dashboard
**URL:** https://discuss.elastic.co/t/remove-duplications-in-kibana-dashboard/148491
**Category:** Kibana
**Created:** [September 13, 2018, 5:49pm UTC](https://discuss.elastic.co/t/remove-duplications-in-kibana-dashboard/148491 "2018-09-13T17:49:15Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@elkNewbie1](https://discuss.elastic.co/u/elkNewbie1)
#### Post date: [September 13, 2018, 5:49pm UTC](https://discuss.elastic.co/t/remove-duplications-in-kibana-dashboard/148491/1 "2018-09-13T17:49:15Z")
Currently using two separate index patterns (logstash-\* and filebeat-\*), with separate dashboards. The live logs go to a local machine that is then read through filebeat and sent to elasticsearch which sends to kibana. Every night those same logs get copied to a different location which logstash sees and ingests. By default filebeat puts everything into a filebeat-yyyy-mm index while logstash does logstash-yyyy-mm index. So the data is the same but because of the labeling they get into kibana and are not considered duplicate. Any ideas on how to remove the duplicate data? Ideally I would like to get rid of the filebeat data after logstash does its ingest.
---
### Author:  [@jbudz](https://discuss.elastic.co/u/jbudz)
#### Post date: [September 13, 2018, 6:06pm UTC](https://discuss.elastic.co/t/remove-duplications-in-kibana-dashboard/148491/2 "2018-09-13T18:06:20Z")
Is the goal to remove duplicate data from elasticsearch or not have it visible in Kibana?
If it's from elasticsearch, the easiest way I can think of is to use the same logstash pipeline for all ingestion. Filebeat logs can send to logstash, and logstash can use the document\_id field and an id field or something like [https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) to get a consistent id.
Can you add a little more detail about the nightly logstash job vs filebeat? Maybe we can use one or the other and keep the log files in storage if we ever need to repopulate.
---
### Author:  [@elkNewbie1](https://discuss.elastic.co/u/elkNewbie1)
#### Post date: [September 13, 2018, 6:51pm UTC](https://discuss.elastic.co/t/remove-duplications-in-kibana-dashboard/148491/3 "2018-09-13T18:51:14Z")
Both if we can, right now Kibana shows same data on logstash and filebeat index.
Nightly, the live log files get stored in a directory that logstash checks for apache log files.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 11, 2018, 6:51pm UTC](https://discuss.elastic.co/t/remove-duplications-in-kibana-dashboard/148491/4 "2018-10-11T18:51:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.