Hi,
I am new to elk stack. We already have a ES 2.4 cluster running and are trying to upgrade to 5.1.1 version. I am feeding live data to old(2.4.x) and new(5.1.1) cluster from logstash(5.x). The data is now in sync between these two cluster but old cluster's index size is almost double when compared to new cluster's index size. Below is an example:
ES Version health status index pri rep docs.count docs.deleted store.size pri.store.size
2.4.x Old_Cluster green open index1-08 5 1 6520824 0 5.3gb 2.6gb
5.1.1 New_Cluster green open index2-08 5 1 6520824 0 9.3gb 4.6gb
I have made modification to the mappings thinking this might change the index size. Following is my current mapping on 5.1 cluster: https://gist.github.com/anonymous/563f88950342f7d910a579995cad1fe6
This didnt make any change to the index size as shown below:
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open new_cluster_index 5 1 6425708 0 9.2gb 4.6gb
I am not sure but i think this behaviour is resulting in high search response times in kibana when compared to the old kibana running with ES 2.4 version.
Can anyone provide any suggestions on this? I am not sure where the problem is occurring in kibana or elasticsearch. Is this something expected when coming to ES 5.1.1 cluster?
Thanks for getting back on this. Following is the link with the current template used for creating new indexes: https://gist.github.com/anonymous/563f88950342f7d910a579995cad1fe6
I dont have the settings for the old indices now but they are same to the new indices.
Below is an example of the settings of new indices:
{
"new_index-2017.01.14": {
"settings": {
"index": {
"refresh_interval": "5s",
"number_of_shards": "5",
"provided_name": "new_index-2017.01.14",
"creation_date": "1484354558913",
"number_of_replicas": "1",
"uuid": "P6bh78qxRS2WusRY037FOw",
"version": {
"created": "5010199"
}
}
}
},
".kibana": {
"settings": {
"index": {
"creation_date": "1484354612684",
"number_of_shards": "1",
"number_of_replicas": "1",
"uuid": "a0ul15NsT_eU_fwtHgw_AQ",
"version": {
"created": "5010199"
},
"provided_name": ".kibana"
}
}
}
}
Hi @JKhondhu, is there something wrong with the mapping? Can you provide any information on how this can be fixed. Please let me know if you need anymore information.
Thanks
Hi,
I want to state that even though the single time based (log) index has grown (possibly mapping related). My current assumption is that the normalization we are doing on the logs being digested, (even though the documents are of the same count) we may have pulled out more data itself. This is to be found out as of yet. Are you using Logstash?
So be it that this is one days index and the assumption lies that the server is architected to hold multiple days indices, and you shall be growing your Elasticstack on version 5.1.x. If your daily indices will be in the top range of 10G per day with a 30 day retention period then you need a data folder with a minimum of 300G.
If you need assistance with architecture planning, emergency support on your elastic products. I would urge you to consider our support. https://www.elastic.co/subscriptions
@JKhondhu, Yes, The indices are being created by logstash on a daily basis. In my case, per day in a dev environment, i am storing around 22 million docs which is resulting to an index size of 32 GB in 5.1.1 cluster. My main concern is not about the storage but this is resulting in high latencies in kibana searches. Is there anyway i can reduce the response times from kibana?
Alrighty, this is no longer an elastic search issue but a new Kibana (v5.1.1?) query issue.
Do you have a question already raised in the Kibana section that you could point me to?
This high latency could come from the depth your Kibana queries go into to poll the shards within your index. You really need to review the queries themselves. And, if you are running an expensive query then having to wait 60 seconds for a response is not unheard of.
@JKhondhu, I don't think this is just related to kibana because I used the profiler API on my ES 5 cluster and tried to execute a simple GET for a url on the indexes. It took around 2 minutes to complete the execution. The same query on kibana took around the same two minutes. The query is not a complicated query but a simple url search.
On ES 2.4 cluster, the same pattern is executed on the cluster and kibana and the response time was around 1 minute.
We see a similar issue with 5.1.2 after upgrading from 2.4. Previously average doc get times were around 2-3 ms, now they are in the range of 300, while the maximum times have gone up from the range of 300 ms to 28 seconds.
Elasticsearch changed the way that it sets the JVM heap from how it used to be done in Elasticsearch 2.x and earlier.
This is now meant to be set in the config/jvm.options file. Please make sure that your nodes have the heap that you expect them to have. This should also be visible in X-Pack Monitoring, if you're using it (free with the Basic license).
@pickypg,
I dont have x-pack monitoring setup on my ES. I am running on 8 GB of RAM and the
-Xms4g
-Xmx4g
is configured in the jvm.options file on my 2 ES nodes.
Hi @pickypg, I have just enabled x-pack monitoring on a sample cluster running ES 5.1.2 and Kibana 5.1.2. This cluster has 15 indices and around 64 M docs. I am trying to conduct searches on this cluster and the results are similar with high latencies. Can you provide any information on what type of monitoring should i be looking into to get some information on what is causing these latencies?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
