# Index data size is too big

**URL:** <https://discuss.elastic.co/t/index-data-size-is-too-big/101140>\
**Category:** Elasticsearch\
**Created:** [September 20, 2017, 9:43am UTC](https://discuss.elastic.co/t/index-data-size-is-too-big/101140 "2017-09-20T09:43:01Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![w11th](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/w11th/32/22260_2.png) [@w11th](https://discuss.elastic.co/u/w11th)\
**Post date:** [September 20, 2017, 9:43am UTC](https://discuss.elastic.co/t/index-data-size-is-too-big/101140/1 "2017-09-20T09:43:01Z")

</div>

![55](https://us1.discourse-cdn.com/elastic/original/3X/0/9/094af62e769e8f06f2184d26252018814f92e760.png)

I deploy an ELK system on Ubuntu, use Filebeat to collect logs. But the index size is too huge. I can't figure out why...

This is my Logstash setting:

```auto
input {
  beats {
    port => 8903
  }
}

output {
    elasticsearch {
        hosts => localhost
        manage_template => false
        index => "huopu_tool-%{+YYYY.MM.dd}"
    }
}

```

This is my Filebeat setting:

```auto
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/access.log*
  exclude_files: [".gz$"]
  document_type: nginx_access

- input_type: log
  paths:
    - /var/log/nginx/error.log*
  exclude_files: [".gz$"]
  document_type: nginx_error

- input_type: log
  paths:
    - /home/deploy/projects/site/shared/log/production.log
  document_type: rails_production

- input_type: log
  paths:
    - /home/deploy/projects/site/shared/log/puma_access.log
  document_type: puma_access

- input_type: log
  paths:
    - /home/deploy/projects/site/shared/log/puma_error.log
  document_type: puma_error
- input_type: log
  paths:
    - /home/deploy/projects/site/shared/log/sidekiq.log
  document_type: sidekiq

output.logstash:
  hosts: ["localhost:8903"]

```

And this is my Elasticsearch index setting, mostly is the default:

```auto
"settings" : {
      "index" : {
        "creation_date" : "1505887670966",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "h5EuSxuJTOaMU9MRFxMvOg",
        "version" : {
          "created" : "5060099"
        },
        "provided_name" : "huopu_tool-2017.09.20"
      }
    }

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 20, 2017, 9:54am UTC](https://discuss.elastic.co/t/index-data-size-is-too-big/101140/2 "2017-09-20T09:54:33Z")

</div>

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out 😉

Why do you say it's too big?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 20, 2017, 9:54am UTC](https://discuss.elastic.co/t/index-data-size-is-too-big/101140/3 "2017-09-20T09:54:58Z")

</div>

How much space your data will take up on disk will depend on a a number of things, e.g. amount of data added through enrichment and the mappings you are using. I wrote a [blog post](https://www.elastic.co/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements) discussing this that may be useful and give you some ideas about how you can go about optimizing your mappings.

---

<div class="post-metadata">

**Author:** ![w11th](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/w11th/32/22260_2.png) [@w11th](https://discuss.elastic.co/u/w11th)\
**Post date:** [September 20, 2017, 11:22am UTC](https://discuss.elastic.co/t/index-data-size-is-too-big/101140/4 "2017-09-20T11:22:26Z")

</div>

My bad 😛 , I'm a newbie to the Elastic Stack world.

It's embarrassing that I thought the `Document Count: 41.1m` means the size of received log files.

You reminded me to check my log file. I found I collect a wrong log, which is a 7G-sized log...

---

<div class="post-metadata">

**Author:** ![w11th](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/w11th/32/22260_2.png) [@w11th](https://discuss.elastic.co/u/w11th)\
**Post date:** [September 20, 2017, 11:28am UTC](https://discuss.elastic.co/t/index-data-size-is-too-big/101140/5 "2017-09-20T11:28:28Z")

</div>

Thank you very much! Your article help me a log. I searched many articles, but still have no idea how to save my disk space. I will follow your article and have a try.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 18, 2017, 11:28am UTC](https://discuss.elastic.co/t/index-data-size-is-too-big/101140/6 "2017-10-18T11:28:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.