You should never loose any data. At least, your injection process should be aware that an index operation failed.
And you can manage errors on a client level.
I guess it's because of UDP or something?
Unsure about your exact use case or architecture but you could consider to add another layer such as rabbitMQ to store first your documents in a queue and then consume them from rabbitMQ to elasticsearch although it's absolutely not something mandatory. Elasticsearch could perfectly manage your injection load if your architecture is well designed.
How many shard do you have per index? How many index?
Do you have enough file descriptors?
Anything in logs?
Also, I think you don't have enough memory to consume your data with Kibana as Kibana is using facets. 6 Gb RAM means 3Gb for elasticsearch VM?
--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr
5 novembre 2013 at 09:06:34, Regis Depraute (regis.depraute@gmail.com) a écrit:
Hello,
Is there a way to limit the cpu usage when i'm querying the elasticsearch database ?
I'm loosing data (new logs - incoming data) when the cluster is too busy in querying.
RD.
Le lundi 4 novembre 2013 18:32:34 UTC+1, David Pilato a écrit :
Basically elasticsearch uses shards for read and write operations.
You don't need to have some "passive" nodes.
If you feel that you can't search fast enough then you should consider for example increase the number of replicas (and the number of nodes).
SSD drives, more memory should help as well.
May be having 1 shard per index or using routing could help.
HTH
--
David 
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 4 nov. 2013 à 15:03, Regis Depraute regis.d...@gmail.com a écrit :
Hello,
I use Elasticsearch to store logs into a database, I receive approximately 50G of logs per day,
After logstash process, and elasticsearch storing, I have +- 22Go of logs per index.
I have a problem with indexing and reading at the same time with elasticsearch.
Everything is working fine until kibana is querying into the elasticsearch database, (some queries last 15 minutes) and no more data is written (or nearly, sometimes i got 25% of the logs)
What can I do to make the "readings" do not disturb "writings" ?
The scheme is :
logs goes to 512(UDP or TCP) => logstash => ES_cluster <= kibana(reader)
I have a server dedicated to logstash (logstash-1.2.2-flatjar.jar) :
he is receiving logs on port 512
and output is elasticsearch_cluster (node03)
config logstash :
input {
syslog {
type => "syslog"
format => "json"
port => 514
}
}
filter {
grok {
... too long ...
}
}
output {
elasticsearch {
host => "aaa.bbb.ccc.ddd" --->here is the ip address of the node03
port => "9302"
node_name => "logstash-node"
}
}
I have an Elasticsearch cluster : (elasticsearch-0.90.5) :
node01 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9300
node02 : 2CPU/6GB RAM/500GB HDD : node.master: true node.data: true port 9301
node03 : 1CPU/2GB RAM/16GB HDD : node.master: true node.data: false port 9302
and a dedicated server for kibana:
he is reading logs into elasticsearch node03
every VM are CentOS(6.4)
Thanks
RD.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
