# How to change log files indexing between Logstash and Elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-change-log-files-indexing-between-logstash-and-elasticsearch/83096>\
**Category:** Logstash\
**Created:** [April 20, 2017, 5:18pm UTC](https://discuss.elastic.co/t/how-to-change-log-files-indexing-between-logstash-and-elasticsearch/83096 "2017-04-20T17:18:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 20, 2017, 5:18pm UTC](https://discuss.elastic.co/t/how-to-change-log-files-indexing-between-logstash-and-elasticsearch/83096/1 "2017-04-20T17:18:13Z")

</div>

Hi there, I have many log files for scraped web-pages for online shopping stores, I ingest and process them using logstash, but every log file takes an elasticsearch index alone, I think this is not very efficient and not helping, as I want every store log files to be grouped together.

How can I tweak the indexing process?

I am thinking about creating a single index, a type for each store name and an ID for each log file (daily generated).  
Do you think this is the best approach?

Here's how my logstash configuration looks like ([http://codepad.org/aWFkdnfE](http://codepad.org/aWFkdnfE)).

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 21, 2017, 5:30am UTC](https://discuss.elastic.co/t/how-to-change-log-files-indexing-between-logstash-and-elasticsearch/83096/2 "2017-04-21T05:30:34Z")

</div>

A separate index for each store probably doesn't make sense, but it depends on how many stores you're indexing. Start with a single index.

Why use a separate type? I'd probably use a single type (especially if the documents have the same schema) and a separate field to indicate the name of the store.

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 21, 2017, 1:08pm UTC](https://discuss.elastic.co/t/how-to-change-log-files-indexing-between-logstash-and-elasticsearch/83096/3 "2017-04-21T13:08:00Z")

</div>

Yes, that's what I meant, single index for all stores, but how would I be able to separate log files for each store on Kibana? (e.g: for every store, I want to create a line chart where the X-axis is date, Y-axis is number of scraped products from i-th store in this date).

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 21, 2017, 1:09pm UTC](https://discuss.elastic.co/t/how-to-change-log-files-indexing-between-logstash-and-elasticsearch/83096/4 "2017-04-21T13:09:50Z")

</div>

As I said, use a separate field to indicate the name of the store. Use that field for filtering and/or aggregation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 19, 2017, 1:19pm UTC](https://discuss.elastic.co/t/how-to-change-log-files-indexing-between-logstash-and-elasticsearch/83096/5 "2017-05-19T13:19:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.