# Question about elasticsearch index and logstash ingestion

**URL:** <https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057>\
**Category:** Elasticsearch\
**Created:** [June 2, 2023, 8:32am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057 "2023-06-02T08:32:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 2, 2023, 8:32am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/1 "2023-06-02T08:32:51Z")

</div>

Hello everyone  
I'd like to ask you 2 questions.  
I receive approximately 270 csv per month, 9 of them per day. Each csv is between 1kB and 3MG in size. All these csv are sent to the same index on elasticsearch with logstash. I would like to know what the maximum storage capacity is for an index in elasticsearch.  
I would also like to know how to automate the import into elasticsearch via logstash so that each time a file comes into the directory it retrieves it without retrieving those that were already there.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 2, 2023, 8:51am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/2 "2023-06-02T08:51:35Z")

</div>

The sky is the limit... 😛

More seriously, the limitation will come from your hardware most likely.  
As Elasticsearch is elastic, it means that you can scale out (add more machines) to add even more space...

But I'd recommend not going above 50gb per shard.

Depending on your use case, you should may be consider using [datastreams](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html).

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 2, 2023, 9:00am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/3 "2023-06-02T09:00:29Z")

</div>

How can I shard my index with logstash?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 2, 2023, 10:49am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/4 "2023-06-02T10:49:25Z")

</div>

You can define the number of shards when you create the index in Elasticsearch or using an index template.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 5, 2023, 7:55am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/5 "2023-06-05T07:55:21Z")

</div>

But I am not creating my index with elasticsearch. I create it with logstash

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 5, 2023, 8:20am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/6 "2023-06-05T08:20:51Z")

</div>

You need to create an index template in Elasticsearch first.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 5, 2023, 9:48am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/7 "2023-06-05T09:48:39Z")

</div>

Do I need to create the index and specify the column names?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 5, 2023, 10:06am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/8 "2023-06-05T10:06:14Z")

</div>

You need to create an index template.

You can leave elasticsearch guess what you want to do with the "columns" (we are saying "fields" in Elasticsearch vocabulary), but I'd definitely recommend to define this before the first document is indexed.

Something like:

```auto
PUT _index_template/csv-template
{
  "index_patterns": ["csv*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "properties": {
        "my_field1": {
          "type": "text"
        },
        "my_field2": {
          "type": "date"
        }
      }
    }
  },
  "priority": 100
}

```

Source:

> **[Index templates | Elasticsearch Guide \[8.8\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html)**

This example supposes that you configure logstash to send the data to an index name starting with `csv`...

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 5, 2023, 1:00pm UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/9 "2023-06-05T13:00:07Z")

</div>

Thank you . But I don't understand this line : "index\_patterns": ["csv\*"]

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 5, 2023, 1:05pm UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/10 "2023-06-05T13:05:30Z")

</div>

> [@Hanni](#):
>
> But I don't understand this line : "index\_patterns": ["csv\*"]

This means that the template will be applied to every indice that starts with `csv`, so in your logstash output you would configure your output index to `csv-something` for example.

It is explained in the already shared documentation.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 5, 2023, 2:39pm UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/11 "2023-06-05T14:39:37Z")

</div>

got it. Thank you

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 6, 2023, 3:47am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/12 "2023-06-06T03:47:56Z")

</div>

What sort of data are you ingesting? Is it time based?

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 6, 2023, 8:08am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/13 "2023-06-06T08:08:04Z")

</div>

Yes!

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 6, 2023, 8:10am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/14 "2023-06-06T08:10:53Z")

</div>

Hello, I have a question.  
After creating my index model I'd like to send my data to an elasticsearch index. In my context I receive data every day. How can I automate the logstash task so that it can send new files arriving on elasticsearch every day?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 6, 2023, 3:31pm UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/15 "2023-06-06T15:31:11Z")

</div>

> [@Hanni](#):
>
> How can I automate the logstash task so that it can send new files arriving on elasticsearch every day?

The best way is to run Logstash as a service and configure your input to read files on an specific folder, when a new file appears on this folder logstash will automatically read and process it.

How to run logstash as a service depends on your operating system, but just follow the [documentation](https://www.elastic.co/guide/en/logstash/current/setup-logstash.html).

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 6, 2023, 11:20pm UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/16 "2023-06-06T23:20:06Z")

</div>

Make sure you are using [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) then.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 7, 2023, 8:51am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/17 "2023-06-07T08:51:36Z")

</div>

When I create my index model, if I set the number of shards to 2, what happens when I send my data to the indexes?

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 7, 2023, 1:52pm UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/18 "2023-06-07T13:52:10Z")

</div>

Hello,  
I have one more question.  
Let me put it in context:  
I've created an index template in elasticsearch. Then, using logstash, I created an index that follows this template. Before I can visualize my data in kibana, I need to create a data view from this index. Every day, a script is launched and logstash sends new data to my index. My question is: I'd like to know if every time I receive data in my index, my data view created from this index updates automatically, as well as the data in my dashboard.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 7, 2023, 2:55pm UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/19 "2023-06-07T14:55:20Z")

</div>

The data in elasticsearch will be seen by Kibana.  
As long as the data view corresponds to the same index, alias...

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [June 14, 2023, 9:08am UTC](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057/20 "2023-06-14T09:08:37Z")

</div>

Thank you

[Next page](https://discuss.elastic.co/t/question-about-elasticsearch-index-and-logstash-ingestion/335057.md?page=2)