ElasticSearch performance problem for dynamic field mapping

Elastic Stack Elasticsearch
1

I have an ES on linux and I index the documents using Nest framework. But the problem is there is one object that containes dynamic field. For exp: there is a class named Product and one of its properties is dynamic which means mapping changes everytime according to the new request. Lets say name of the dynamic property is Data and Data might be an object or string or integer or list of obejct ...etc. it might be everything. I don't know it's normal but there is no other option to do it so far.

1 gb data is stored every day into the es but when I search for a data which was indexed 1 or 2 weeks ago on kibana, it can not find it. It sais "Timeout ..." . So far there are 1800 fields in my index.

My question is what is the best practice for dynamic field mapping ? What should I do?

2

What version are you on?

That is likely causing problems, why is it so high?

3

The version is 5.5.0

I am trying to use the index as a log storage. I have like 20 applications and these apps creates logs (request and response). I consume these log data as json in my elastic search consumer application. It means I have json string on my elastic search client application. I convert every json data to jobject and than insert them to elasticsearch index.
Thats why there are like 1800 fields, every project has its own type. So what is your recommendation or what is the best practice to do this ?

4

Are you putting everything in one index?

5

Yes but it creates a new index everyday. I mean there is one index per day (index_name-2017-25-10 or index_name-2017-26-10).

6

You should split things out into different indices then. Put things that have similar structures together and you won't run into this problem.

1 Like
7

Spliting them into different indices might cause a searching performance ? I mean Kibana search for one index now, but if I split it will start searching for all the indices. Wont this cause performance problems ?

8

timeout settings in kibana
the default timeout of kibana is 30sec ,it small with a large data,can change it.

9

It won't cause problems.

Searching one index of 10 shards is the same as 10 indices of 1 shard, or 5 of 2 shards etc,

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.