Loading of Data in Elaticsearch

Elastic Stack Elasticsearch
1

Hi,

I am working with something in which I loading data (around 2.5 GB) into ES per day and then Delete once in a week .I want a device that whether it will good to load data daily into single index or load the data daily into each respective index( 7 index for 7 day and then deleting all once in a week).Kindly explain with the reason as I am new to ES.

Thanks,
Rajeev

2

You can use Logstash for this - https://www.elastic.co/guide/en/logstash/current/index.html

But without more information it's a bit hard to help.

3

I am able to load data into ES. I am asking about the approac so that I can't have dataloss in the case of failing due to overload in an index or shred.

4

Then you want at least 3 nodes and 1 replica of the index.

5

At Present I am loading data into one node and one index and when I tried to load data today size of node became around 6.5 gb and after that I am getting error with cluster health read.Can you help me in that and also any way so that I can avoid these type of failure in future.

6

How much heap did you assign?

7

Initially It was set to default .So I now I changed it to 8GB by writing ES_HEAP_SIZE=8G and also locked the memory by setting bootstrap.mlockall: true in elasticsearch.yml file running on Window Server 2012 R2. Is it Correct. Suggest some points.

8

That'll help a lot!

9

Thanks for all the information. Can you please tell where to set all these value as I am getting error When I am setting these value in elasticsearch.yml file of config folder. I have 16GB of RAM.

10

Depends how you installed ES, the easiest way is to set an environment variable of ES_HEAP_SIZE.