What can I do to make the "readings" do not disturb "writings"?

Hello,

I use Elasticsearch to store logs into a database, I receive approximately
50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of logs
per index.

I have a problem with indexing and reading at the same time with
elasticsearch.
Everything is working fine until kibana is querying into the elasticsearch
database, (some queries last 15 minutes) and no more data is written (or
nearly, sometimes i got 25% of the logs)

What can I do to make the "readings" do not disturb "writings" ?

The scheme is :

logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)

  • I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) :

he is receiving logs on port 512
and output is elasticsearch_cluster (node03)

config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}

  • I have an Elasticsearch cluster : (elasticsearch-0.90.5) :

node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port
9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port
9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data:* false*port 9302

  • and a dedicated server for kibana:

he is reading logs into elasticsearch node03

every VM are CentOS(6.4)
*
*
Thanks
RD.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Basically elasticsearch uses shards for read and write operations.
You don't need to have some "passive" nodes.

If you feel that you can't search fast enough then you should consider for example increase the number of replicas (and the number of nodes).

SSD drives, more memory should help as well.
May be having 1 shard per index or using routing could help.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 nov. 2013 à 15:03, Regis Depraute regis.depraute@gmail.com a écrit :

Hello,

I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of logs per index.

I have a problem with indexing and reading at the same time with elasticsearch.
Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs)

What can I do to make the "readings" do not disturb "writings" ?

The scheme is :

logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)

I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) :
he is receiving logs on port 512
and output is elasticsearch_cluster (node03)

config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}

I have an Elasticsearch cluster : (elasticsearch-0.90.5) :
node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: false port 9302

and a dedicated server for kibana:
he is reading logs into elasticsearch node03

every VM are CentOS(6.4)

Thanks
RD.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hello,

Is there a way to limit the cpu usage when i'm querying the elasticsearch
database ?
I'm loosing data (new logs - incoming data) when the cluster is too busy in
querying.

RD.

Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit :

Basically elasticsearch uses shards for read and write operations.
You don't need to have some "passive" nodes.

If you feel that you can't search fast enough then you should consider for
example increase the number of replicas (and the number of nodes).

SSD drives, more memory should help as well.
May be having 1 shard per index or using routing could help.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 nov. 2013 à 15:03, Regis Depraute <regis.d...@gmail.com <javascript:>>
a écrit :

Hello,

I use Elasticsearch to store logs into a database, I receive approximately
50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of logs
per index.

I have a problem with indexing and reading at the same time with
elasticsearch.
Everything is working fine until kibana is querying into the elasticsearch
database, (some queries last 15 minutes) and no more data is written (or
nearly, sometimes i got 25% of the logs)

What can I do to make the "readings" do not disturb "writings" ?

The scheme is :

logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)

  • *I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) :

he is receiving logs on port 512
and output is elasticsearch_cluster (node03)

config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}

  • I have an Elasticsearch cluster : (elasticsearch-0.90.5) :

node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port
9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port
9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data:* false*port 9302

  • and a dedicated server for kibana:

he is reading logs into elasticsearch node03

every VM are CentOS(6.4)
*
*
Thanks
RD.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You should never loose any data. At least, your injection process should be aware that an index operation failed.
And you can manage errors on a client level.

I guess it's because of UDP or something?

Unsure about your exact use case or architecture but you could consider to add another layer such as rabbitMQ to store first your documents in a queue and then consume them from rabbitMQ to elasticsearch although it's absolutely not something mandatory. Elasticsearch could perfectly manage your injection load if your architecture is well designed.

How many shard do you have per index? How many index?
Do you have enough file descriptors?

Anything in logs?

Also, I think you don't have enough memory to consume your data with Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

5 novembre 2013 at 09:06:34, Regis Depraute (regis.depraute@gmail.com) a écrit:

Hello,

Is there a way to limit the cpu usage when i'm querying the elasticsearch database ?
I'm loosing data (new logs - incoming data) when the cluster is too busy in querying.

RD.

Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit :
Basically elasticsearch uses shards for read and write operations.
You don't need to have some "passive" nodes.

If you feel that you can't search fast enough then you should consider for example increase the number of replicas (and the number of nodes).

SSD drives, more memory should help as well.
May be having 1 shard per index or using routing could help.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 nov. 2013 à 15:03, Regis Depraute regis.d...@gmail.com a écrit :

Hello,

I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of logs per index.

I have a problem with indexing and reading at the same time with elasticsearch.
Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs)

What can I do to make the "readings" do not disturb "writings" ?

The scheme is :

logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)

I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) :
he is receiving logs on port 512
and output is elasticsearch_cluster (node03)

config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}

I have an Elasticsearch cluster : (elasticsearch-0.90.5) :
node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: false port 9302

and a dedicated server for kibana:
he is reading logs into elasticsearch node03

every VM are CentOS(6.4)

Thanks
RD.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I guess it's because of UDP or something?
Yes, it's UDP

I'm going to add "redis" to make a buffer, do you think it's a good idea ?
Or it's better to use rabbitMQ ?

How many shard do you have per index? How many index?
I have 5 shards
7 index
each index weight +- 22 gb

Do you have enough file descriptors?
What is file descriptors?

Anything in logs?
I think I should increase RAM Memory :

[2013-11-04 09:21:53,606][WARN ][index.engine.robin ] [node01] [
logstash-2013.11.04][0] failed engine
java.lang.OutOfMemoryError: Java heap space
[2013-11-04 09:22:28,988][WARN ][cluster.action.shard ] [node01]sending failed shard
for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED],reason
[engine failure, message [OutOfMemoryError[Java heap space]]]
[2013-11-04 09:22:28,988][WARN ][cluster.action.shard ] [node01]received shard failed
for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED],reason
[engine failure, message [OutOfMemoryError[Java heap space]]]

6 Gb RAM means 3Gb for elasticsearch VM?
yes

Le mardi 5 novembre 2013 09:47:59 UTC+1, David Pilato a écrit :

You should never loose any data. At least, your injection process should
be aware that an index operation failed.
And you can manage errors on a client level.

I guess it's because of UDP or something?

Unsure about your exact use case or architecture but you could consider to
add another layer such as rabbitMQ to store first your documents in a queue
and then consume them from rabbitMQ to elasticsearch although it's
absolutely not something mandatory. Elasticsearch could perfectly manage
your injection load if your architecture is well designed.

How many shard do you have per index? How many index?
Do you have enough file descriptors?

Anything in logs?

Also, I think you don't have enough memory to consume your data with
Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

5 novembre 2013 at 09:06:34, Regis Depraute (regis.d...@gmail.com<javascript:>)
a écrit:

Hello,

Is there a way to limit the cpu usage when i'm querying the elasticsearch
database ?
I'm loosing data (new logs - incoming data) when the cluster is too busy
in querying.

RD.

Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit :

Basically elasticsearch uses shards for read and write operations.
You don't need to have some "passive" nodes.

If you feel that you can't search fast enough then you should consider
for example increase the number of replicas (and the number of nodes).

SSD drives, more memory should help as well.
May be having 1 shard per index or using routing could help.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 nov. 2013 à 15:03, Regis Depraute regis.d...@gmail.com a écrit :

Hello,

I use Elasticsearch to store logs into a database, I receive
approximately 50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of logs
per index.

I have a problem with indexing and reading at the same time with
elasticsearch.
Everything is working fine until kibana is querying into the
elasticsearch database, (some queries last 15 minutes) and no more data is
written (or nearly, sometimes i got 25% of the logs)

What can I do to make the "readings" do not disturb "writings" ?

The scheme is :

logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)

  • I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar)
    :

he is receiving logs on port 512
and output is elasticsearch_cluster (node03)

config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}

  • I have an Elasticsearch cluster : (elasticsearch-0.90.5) :

node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true
port 9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true
port 9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: falseport 9302

  • and a dedicated server for kibana:

he is reading logs into elasticsearch node03

every VM are CentOS(6.4)
*
*
Thanks
RD.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

So add more memory or more nodes.
I think you don't need to "complicate" things with another layer.

Check this webinar: Elasticsearch Platform — Find real-time answers at scale | Elastic

It will help you a lot!

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 nov. 2013 à 10:44, Regis Depraute regis.depraute@gmail.com a écrit :

I guess it's because of UDP or something?
Yes, it's UDP

I'm going to add "redis" to make a buffer, do you think it's a good idea ?
Or it's better to use rabbitMQ ?

How many shard do you have per index? How many index?
I have 5 shards
7 index
each index weight +- 22 gb

Do you have enough file descriptors?
What is file descriptors?

Anything in logs?
I think I should increase RAM Memory :

[2013-11-04 09:21:53,606][WARN ][index.engine.robin ] [node01] [logstash-2013.11.04][0] failed engine
java.lang.OutOfMemoryError: Java heap space
[2013-11-04 09:22:28,988][WARN ][cluster.action.shard ] [node01] sending failed shard for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]
[2013-11-04 09:22:28,988][WARN ][cluster.action.shard ] [node01] received shard failed for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]

6 Gb RAM means 3Gb for elasticsearch VM?
yes

Le mardi 5 novembre 2013 09:47:59 UTC+1, David Pilato a écrit :

You should never loose any data. At least, your injection process should be aware that an index operation failed.
And you can manage errors on a client level.

I guess it's because of UDP or something?

Unsure about your exact use case or architecture but you could consider to add another layer such as rabbitMQ to store first your documents in a queue and then consume them from rabbitMQ to elasticsearch although it's absolutely not something mandatory. Elasticsearch could perfectly manage your injection load if your architecture is well designed.

How many shard do you have per index? How many index?
Do you have enough file descriptors?

Anything in logs?

Also, I think you don't have enough memory to consume your data with Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

5 novembre 2013 at 09:06:34, Regis Depraute (regis.d...@gmail.com) a écrit:

Hello,

Is there a way to limit the cpu usage when i'm querying the elasticsearch database ?
I'm loosing data (new logs - incoming data) when the cluster is too busy in querying.

RD.

Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit :

Basically elasticsearch uses shards for read and write operations.
You don't need to have some "passive" nodes.

If you feel that you can't search fast enough then you should consider for example increase the number of replicas (and the number of nodes).

SSD drives, more memory should help as well.
May be having 1 shard per index or using routing could help.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 nov. 2013 à 15:03, Regis Depraute regis.d...@gmail.com a écrit :

Hello,

I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of logs per index.

I have a problem with indexing and reading at the same time with elasticsearch.
Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs)

What can I do to make the "readings" do not disturb "writings" ?

The scheme is :

logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)

I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) :
he is receiving logs on port 512
and output is elasticsearch_cluster (node03)

config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}

I have an Elasticsearch cluster : (elasticsearch-0.90.5) :
node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: false port 9302

and a dedicated server for kibana:
he is reading logs into elasticsearch node03

every VM are CentOS(6.4)

Thanks
RD.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks :slight_smile:

Le mardi 5 novembre 2013 13:35:28 UTC+1, David Pilato a écrit :

So add more memory or more nodes.
I think you don't need to "complicate" things with another layer.

Check this webinar:
Elasticsearch Platform — Find real-time answers at scale | Elastic

It will help you a lot!

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 5 nov. 2013 à 10:44, Regis Depraute <regis.d...@gmail.com <javascript:>>
a écrit :

I guess it's because of UDP or something?
Yes, it's UDP

I'm going to add "redis" to make a buffer, do you think it's a good idea ?
Or it's better to use rabbitMQ ?

How many shard do you have per index? How many index?
I have 5 shards
7 index
each index weight +- 22 gb

Do you have enough file descriptors?
What is file descriptors?

Anything in logs?
I think I should increase RAM Memory :

[2013-11-04 09:21:53,606][WARN ][index.engine.robin ] [node01] [
logstash-2013.11.04][0] failed engine
java.lang.OutOfMemoryError: Java heap space
[2013-11-04 09:22:28,988][WARN ][cluster.action.shard ] [node01]sending failed shard
for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED
], reason [engine failure, message [OutOfMemoryError[Java heap space]]]
[2013-11-04 09:22:28,988][WARN ][cluster.action.shard ] [node01]received shard failed
for [logstash-2013.11.04][0], node[NoxOucsFSciCBOfwHVVCfA], [R], s[STARTED
], reason [engine failure, message [OutOfMemoryError[Java heap space]]]

6 Gb RAM means 3Gb for elasticsearch VM?
yes

Le mardi 5 novembre 2013 09:47:59 UTC+1, David Pilato a écrit :

You should never loose any data. At least, your injection process should
be aware that an index operation failed.
And you can manage errors on a client level.

I guess it's because of UDP or something?

Unsure about your exact use case or architecture but you could consider
to add another layer such as rabbitMQ to store first your documents in a
queue and then consume them from rabbitMQ to elasticsearch although it's
absolutely not something mandatory. Elasticsearch could perfectly manage
your injection load if your architecture is well designed.

How many shard do you have per index? How many index?
Do you have enough file descriptors?

Anything in logs?

Also, I think you don't have enough memory to consume your data with
Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

5 novembre 2013 at 09:06:34, Regis Depraute (regis.d...@gmail.com) a
écrit:

Hello,

Is there a way to limit the cpu usage when i'm querying the elasticsearch
database ?
I'm loosing data (new logs - incoming data) when the cluster is too busy
in querying.

RD.

Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit :

Basically elasticsearch uses shards for read and write operations.
You don't need to have some "passive" nodes.

If you feel that you can't search fast enough then you should consider
for example increase the number of replicas (and the number of nodes).

SSD drives, more memory should help as well.
May be having 1 shard per index or using routing could help.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 nov. 2013 à 15:03, Regis Depraute regis.d...@gmail.com a écrit :

Hello,

I use Elasticsearch to store logs into a database, I receive
approximately 50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of
logs per index.

I have a problem with indexing and reading at the same time with
elasticsearch.
Everything is working fine until kibana is querying into the
elasticsearch database, (some queries last 15 minutes) and no more data is
written (or nearly, sometimes i got 25% of the logs)

What can I do to make the "readings" do not disturb "writings" ?

The scheme is :

logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)

  • I have a server dedicated to logstash
    (logstash-1.2.2-flatjar.jar) :

he is receiving logs on port 512
and output is elasticsearch_cluster (node03)

config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}

  • I have an Elasticsearch cluster : (elasticsearch-0.90.5) :

node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true
port 9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true
port 9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: falseport 9302

  • and a dedicated server for kibana:

he is reading logs into elasticsearch node03

every VM are CentOS(6.4)
*
*
Thanks
RD.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.