# Is it possible to archive and analyze 1 year metricbeat in kibana **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [November 27, 2018, 6:03am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273 "2018-11-27T06:03:29Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [November 27, 2018, 6:03am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/1 "2018-11-27T06:03:29Z")
Hi, Can anyone help me? How to do archive and analyze the metricbeat data in kibana. How to setup the infrastructure as well ?? ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [November 29, 2018, 9:51am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/2 "2018-11-29T09:51:49Z")
Hi @sarava 🙂 You can follow [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html) easy getting started guide of Elasticsearch to understand how to setup the infrastructure. As soon as you have available space on your disk you could store 1 year or 10. Once you have elasticsearch runnning, you just need to connect [Kibana](https://www.elastic.co/guide/en/kibana/current/introduction.html) to Elasticsearch to analize your data For your reference, [this](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-getting-started.html) is the getting started guide of Metricbeat. Best regards! ---
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [November 30, 2018, 10:05am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/3 "2018-11-30T10:05:59Z")
We are going to build new setup for analyse the one year data of linux and windows, so can i know the resource details for CPU, Memory and Disk space of the server. single node and 3-node both if give means it helps lot... Metricbeats count - 1500 Winlogbeats count - 1500 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2018, 10:13am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/4 "2018-11-30T10:13:58Z")
Both Beats can generate varying amount of data depending on how they are configured and how much the host they are deployed on logs, so it is hard for anyone without more detailed knowledge about your deployment to estimate how much data that would be. I recommend setting it up on a few host and collect data over a few days to see how much it generates. That will give you a better idea of how much data the cluster will need to hold and will make it easier to get help estimating what the required number of nodes and hardware specification would be. ---
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [November 30, 2018, 11:06am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/5 "2018-11-30T11:06:03Z")
i have mentioned 1500 host count only already we installed few host and receiving logs as well. 175 servers - 50GB data receiving. May i know the resource details of Elasticsearch and Kibana for 1500 servers logs need to analyze. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2018, 11:10am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/6 "2018-11-30T11:10:30Z")
Over what time period did the 175 hosts generate this 50GB of data? How many indices and shards was this indexed into? Was this on a single node or a cluster with replicas configured? Did this include representative data from both types of beats? ---
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [November 30, 2018, 11:14am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/7 "2018-11-30T11:14:42Z")
Only one day data of metricbeat is 50GB only one indices and shards in indexed 3-node cluster with replicas configured ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2018, 11:19am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/8 "2018-11-30T11:19:32Z")
How much data does Winlogbeat generate for the same period? ---
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [November 30, 2018, 11:54am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/9 "2018-11-30T11:54:37Z")
Hi Christian, Winlogbeat - 8GB - 25 servers ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2018, 12:32pm UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/10 "2018-11-30T12:32:22Z")
OK. Based on that it seems like 1 host generates 50GB/175 + 8GB/25 = 605MB/day (this includes one replica shard). This is split almost even between Metricbeat and Winlog beat data. If we simply extrapolate on this we get 1500 hosts generating 909GB/day. Over a year that is close to 324TB of data stored. If you have that amount of data coming in and want to keep it online for a full year, that will require a quite substantial cluster. For Metricbeat data it may be possible to utilise rollups and reduce the volume significantly, but that feature is not yet GA. ---
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [December 3, 2018, 6:07am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/11 "2018-12-03T06:07:00Z")
Thanks for your update.. I will discuss with my management and will get back to you.. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 3, 2018, 6:14am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/12 "2018-12-03T06:14:45Z")
If you do not need the full year online and it is acceptable to restore data before querying older data you look into using snapshot and restore. ---
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [December 3, 2018, 6:19am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/13 "2018-12-03T06:19:42Z")
Already i tried earlier the snapshot and restore. once done the data restoration then how make it to index status from red to green. Is there any command to refresh the cluster index or how. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 3, 2018, 6:25am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/14 "2018-12-03T06:25:38Z")
Did you monitor the restore progress to make sure it had completed? Once done it will go from red to yellow or green depending whether any replicas need to be restored or not. ---
**Author:** ![sarava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sarava/32/35899_2.png) [@sarava](https://discuss.elastic.co/u/sarava)\ **Post date:** [December 3, 2018, 6:27am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/15 "2018-12-03T06:27:15Z")
it was completed once again i will do it then let you know the status. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 31, 2018, 6:27am UTC](https://discuss.elastic.co/t/is-it-possible-to-archive-and-analyze-1-year-metricbeat-in-kibana/158273/16 "2018-12-31T06:27:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.