I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here is
the situation - I am currently running everything on single node - which I
know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is like
3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config params
too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
You're probably at the limits of a single node. You should upgrade to a
later version of ES as there is always performance improvements or add more
heap or nodes. The default settings of ES are more than suitable up to tens
of nodes, you shouldn't need to change anything there in the immediate term
and you will definitely see performance improvements.
Given you're currently only using one node, you can also disable replicas,
that will eliminate those unassigned shards and you can easily add them
later.
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here is
the situation - I am currently running everything on single node - which I
know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config params
too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
How are your queries build up. In a default situation
Kibana queries all indices, it can help to query only
the indices that you need. Beside that there is a
dashboard option (preload fields) to improve your
query when it concerns a lot of data tot query every
time you refresh your panels.
hth,,
Arie.
Op woensdag 31 december 2014 21:25:37 UTC+1 schreef Bhumir Jhaveri:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here is
the situation - I am currently running everything on single node - which I
know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config params
too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
I have plan to upgrade this to ES1.4x but just need to make sure that
version upgrade is smooth and free of any issues - I already have around
300gigs of data so hope that ES1.4 would give me better performance.
On Wednesday, December 31, 2014 7:14:46 PM UTC-8, Mark Walkom wrote:
You're probably at the limits of a single node. You should upgrade to a
later version of ES as there is always performance improvements or add more
heap or nodes. The default settings of ES are more than suitable up to tens
of nodes, you shouldn't need to change anything there in the immediate term
and you will definitely see performance improvements.
Given you're currently only using one node, you can also disable replicas,
that will eliminate those unassigned shards and you can easily add them
later.
On 1 January 2015 at 07:25, Bhumir Jhaveri <bhum...@gmail.com
<javascript:>> wrote:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here
is the situation - I am currently running everything on single node - which
I know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config
params too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
No it queries only the those set of groups which I require on dashboard - I
am not using _all - but having done that it is still huge set of data which
I am using here to generate the dashboard.
One problem which I noticed is like the documents which are there consists
of 14 columns - as in I am storing data for 14 different fields in one
single document - and by default ES puts indexes on all 14 columns so that
is one issue which I am aware off not sure if I change that to only lets
say 4 to 5 columns on which I will be performing search/query via kibana
dashboard - so that is something I still need to try off n see.
Not sure once you have already loaded this much of data - can you just take
indexes on few columns? like this situation.
On Thursday, January 1, 2015 11:18:03 AM UTC-8, Arie wrote:
Hi
How are your queries build up. In a default situation
Kibana queries all indices, it can help to query only
the indices that you need. Beside that there is a
dashboard option (preload fields) to improve your
query when it concerns a lot of data tot query every
time you refresh your panels.
hth,,
Arie.
Op woensdag 31 december 2014 21:25:37 UTC+1 schreef Bhumir Jhaveri:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here
is the situation - I am currently running everything on single node - which
I know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config
params too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
Also I am planning to create multiple nodes here as -
one single client node - which would be like with 16cpu and 60 gigs of RAM
and then alll data nodes with 8vcpu and 24gigs of ram and last the master
with either 24gigs or 60 gigs - will this be a kind of correct design or I
should change this to something.
On Wednesday, December 31, 2014 12:25:37 PM UTC-8, Bhumir Jhaveri wrote:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here is
the situation - I am currently running everything on single node - which I
know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config params
too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
I agree with what has been said up to this point; adding more nodes and
upgrading ES are good things to do. However, before beginning to spec out
new hardware you should probably evaluate what your existing node is doing.
Are you keeping metrics on what ES is doing? When I try to size my clusters
I try to pay attention to what percent of the heap is being used for what
purposes. How does disk I/O look? What are your CPUs doing when those
queries come in?
On Fri, Jan 2, 2015 at 11:19 AM, Bhumir Jhaveri bhumir81@gmail.com wrote:
Also I am planning to create multiple nodes here as -
one single client node - which would be like with 16cpu and 60 gigs of RAM
and then alll data nodes with 8vcpu and 24gigs of ram and last the master
with either 24gigs or 60 gigs - will this be a kind of correct design or I
should change this to something.
On Wednesday, December 31, 2014 12:25:37 PM UTC-8, Bhumir Jhaveri wrote:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here
is the situation - I am currently running everything on single node - which
I know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config
params too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
Sure I will check that - is marvel better way to check these details?
Though I have some rough idea of what the heap/cpus are doing - but since
its hardware sizing so dont want to size them on guess work.
On Friday, January 2, 2015 9:34:13 AM UTC-8, Chris Rimondi wrote:
I agree with what has been said up to this point; adding more nodes and
upgrading ES are good things to do. However, before beginning to spec out
new hardware you should probably evaluate what your existing node is doing.
Are you keeping metrics on what ES is doing? When I try to size my clusters
I try to pay attention to what percent of the heap is being used for what
purposes. How does disk I/O look? What are your CPUs doing when those
queries come in?
On Fri, Jan 2, 2015 at 11:19 AM, Bhumir Jhaveri <bhum...@gmail.com
<javascript:>> wrote:
Also I am planning to create multiple nodes here as -
one single client node - which would be like with 16cpu and 60 gigs of
RAM - and then alll data nodes with 8vcpu and 24gigs of ram and last the
master with either 24gigs or 60 gigs - will this be a kind of correct
design or I should change this to something.
On Wednesday, December 31, 2014 12:25:37 PM UTC-8, Bhumir Jhaveri wrote:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here
is the situation - I am currently running everything on single node - which
I know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config
params too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
Marvel is good for looking at metrics. I find the _cat API very helpful for
spot checking. _cat/nodes?v and _cat/indices?v are two I use pretty
frequently. We have a pretty tight statsd/graphite integration as well.
On Fri, Jan 2, 2015 at 12:38 PM, Bhumir Jhaveri bhumir81@gmail.com wrote:
Sure I will check that - is marvel better way to check these details?
Though I have some rough idea of what the heap/cpus are doing - but since
its hardware sizing so dont want to size them on guess work.
On Friday, January 2, 2015 9:34:13 AM UTC-8, Chris Rimondi wrote:
I agree with what has been said up to this point; adding more nodes and
upgrading ES are good things to do. However, before beginning to spec out
new hardware you should probably evaluate what your existing node is doing.
Are you keeping metrics on what ES is doing? When I try to size my clusters
I try to pay attention to what percent of the heap is being used for what
purposes. How does disk I/O look? What are your CPUs doing when those
queries come in?
On Fri, Jan 2, 2015 at 11:19 AM, Bhumir Jhaveri bhum...@gmail.com
wrote:
Also I am planning to create multiple nodes here as -
one single client node - which would be like with 16cpu and 60 gigs of
RAM - and then alll data nodes with 8vcpu and 24gigs of ram and last the
master with either 24gigs or 60 gigs - will this be a kind of correct
design or I should change this to something.
On Wednesday, December 31, 2014 12:25:37 PM UTC-8, Bhumir Jhaveri wrote:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) - Here
is the situation - I am currently running everything on single node - which
I know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config
params too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which I
should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
I have never used it with _cat api but I just tried it - its good - I
already have kopf but doesnt tell you exactly when you're running in any
query and its taking - so what all things its doing during that duration.
On Friday, January 2, 2015 10:21:32 AM UTC-8, Chris Rimondi wrote:
Marvel is good for looking at metrics. I find the _cat API very helpful
for spot checking. _cat/nodes?v and _cat/indices?v are two I use pretty
frequently. We have a pretty tight statsd/graphite integration as well.
On Fri, Jan 2, 2015 at 12:38 PM, Bhumir Jhaveri <bhum...@gmail.com
<javascript:>> wrote:
Sure I will check that - is marvel better way to check these details?
Though I have some rough idea of what the heap/cpus are doing - but since
its hardware sizing so dont want to size them on guess work.
On Friday, January 2, 2015 9:34:13 AM UTC-8, Chris Rimondi wrote:
I agree with what has been said up to this point; adding more nodes and
upgrading ES are good things to do. However, before beginning to spec out
new hardware you should probably evaluate what your existing node is doing.
Are you keeping metrics on what ES is doing? When I try to size my clusters
I try to pay attention to what percent of the heap is being used for what
purposes. How does disk I/O look? What are your CPUs doing when those
queries come in?
On Fri, Jan 2, 2015 at 11:19 AM, Bhumir Jhaveri bhum...@gmail.com
wrote:
Also I am planning to create multiple nodes here as -
one single client node - which would be like with 16cpu and 60 gigs of
RAM - and then alll data nodes with 8vcpu and 24gigs of ram and last the
master with either 24gigs or 60 gigs - will this be a kind of correct
design or I should change this to something.
On Wednesday, December 31, 2014 12:25:37 PM UTC-8, Bhumir Jhaveri wrote:
Hey guys,
I am facing some performance issues with ES(1.2.2)/Kibana(3.0.2) -
Here is the situation - I am currently running everything on single node -
which I know is biggest problem n bottleneck.
VM on which I am running is - 24gb ram- 8proc - 2TB storage space
Here is snapshot of what I have on ES
12GB heap(used only 50% of what is available)
50 indices
370 shards
185 unassigned
757051595 docs
183.71GB is total space occupied (this is of last 7 days of data only)
Now when I am running loading these data on my kibana dashboard, its
running very slow.
I agree that I need to split the data across multiple nodes - which is
like 3 to 5 data node, 1 client node, 1 master node - so even with that
configuration - it should run fine I guess may be for 10 to 15 days of data
since data would be distributed across multiple data nodes (Again this is
my assumption that it will go smooth) -
so would this be sufficient or I should also tweak few other config
params too - with which I will see some good performance and improvement?
I know I am seriously not tweaking some of the important config which
I should be but being little new to this world - I am still researching
further on this.
Should I move to some newer version of ES may be 1.3x or 1.4x which is
compatible with Kibana 3.1.2(Definitely don't want to go kibana 4 at the
moment)?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.