# Elasticsearch-Logstash Storage and events per second
**URL:** https://discuss.elastic.co/t/elasticsearch-logstash-storage-and-events-per-second/67367
**Category:** Elasticsearch
**Created:** [November 28, 2016, 2:55pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-storage-and-events-per-second/67367 "2016-11-28T14:55:43Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@rpoveda](https://discuss.elastic.co/u/rpoveda)
#### Post date: [November 28, 2016, 2:55pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-storage-and-events-per-second/67367/1 "2016-11-28T14:55:43Z")
> [@EPS Kafka-Logstash; Dimensionamiento Indices Elasticsearch](https://discuss.elastic.co/t/eps-kafka-logstash-dimensionamiento-indices-elasticsearch/67346):
>
> Buenos días, pruebo a postear en Castellano porque las dudas que tengo ahora son mucho más abstractas y desconfío de mi capacidad de expresión en inglés. Necesito contestarme a unas ciertas dudas que tengo acerca del comportamiento de un par cosas. ¿Podríais ayudarme por favor? Logstash-Kafka-Logstash. Logstash 1 Para las maquinas que no permitan la comunicación con Kafka tengo un Logstash que realiza esta comunicación, esto funciona perfectamente. Envía a una buena velocidad, y se traga l…
I need to answer some questions I have about the behavior of a couple things. Could you please help me?
**Logstash-Kafka-Logstash.**
- Logstash 1
For systems that don't allow communication with Kafka I have a Logstash that makes this communication, this works perfectly. Send at a good speed, and swallows the peaks with relative ease.
- Logstash 2
The second Logstash is the one that would perform all the mapping and would have RAM and processor to spare, but it takes a lot to consume the events stored in Kafka, even without carrying out any mapping or filtering.
I have modified the input, choosing the pipe plugin with the Kafka consumer command, and uploading the autocommit to 1 minute to increase its performance, but even with resources to spare, it does not go to the rhythm that I would like ... And leaves too many events in Kafka.
**Elasticsearch**
The cluster with which I am practicing has three master nodes, three data nodes, and two client nodes (to redound Kibana). It works perfect everything, but I have several doubts about it:
- Regarding the number of indexes, types, shards and replicas.
Two questions:
```
-What would be the optimum configuration here to maximize profitability, both search time and storage? I have to find a good balance in both, especially for AllInOne environments.
-Would you have a relationship "indexes-types-shards" -> "speed" -> "size-storage"? I'm pretty confused about it ...
```
- Regarding the number of data added.
I have read since joins can not be made in Elasticsearch, searches involving several documents, (users with the largest number of failed logins in our entire system) should be done in pairs, first looking for the user X in our indexes Users, and then the number of logins you have made. And that the only way to increase search efficiency would consist in adding fields to avoid performing this type of joins.
```
-Does this greatly increase the size of the index?
-How much would we be talking about? Is it correct, positive (and, above all, acceptable) to increase the size of indexed documents by adding data from other documents to facilitate these multidocument searches?
```
- Regarding the compression of indexed data.
How much is estimated to weigh an index of about 100,000 million events (collected over two years), indexed by three or four fields, without data aggregations and compressed with "index.codec: best\_compression".
This question, I know that it is difficult to answer with exact information, but is that right now I do not even give an estimated number.
```
- Can re-indexing and re-compression be done to lighten the index? (But you still have to be able to search for this data, even if it is more expensive).
```
---
### Author:  [@rpoveda](https://discuss.elastic.co/u/rpoveda)
#### Post date: [November 29, 2016, 10:17am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-storage-and-events-per-second/67367/2 "2016-11-29T10:17:50Z")
> [@EPS Kafka-Logstash; Dimensionamiento Indices Elasticsearch](https://discuss.elastic.co/t/eps-kafka-logstash-dimensionamiento-indices-elasticsearch/67346/2):
>
> Logstash 1 -¿Como puedo perfeccionar la eficacia de este output y de este Logstash en realidad? -¿Cuál es la configuración de tópics más eficiente¿ (Actualmente, para las versiones más grandes, nos planteamos la posibilidad de 2 tópics, un grupo de productores y uno de consumidores por cliente). Aqui puedes encontrar informacion generica sobre esto: [https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1](https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1) Logstash 2 -¿Cómo podría llevar al máximo a este Logstash pa…
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 27, 2016, 10:18am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-storage-and-events-per-second/67367/3 "2016-12-27T10:18:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.