Kafka River Plugin reference problem

Elastic Stack Elasticsearch
1

Hi,I am writing a kafka river plugin

here is the thing:
for example
There is 5 partition in kafka

So I think the river plugin should only read 1 partition ,like that

Map<String, Integer> topicCountMap = new ConcurrentHashMap<String,
Integer>();
topicCountMap.put(topic, new Integer(1));

Map<String, List<KafkaMessageStream>> consumerMap = consumer
.createMessageStreams(topicCountMap);

And the elastic search cluster should be consist of not more than 5 node,is
that right?

thanks very much.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I'm afraid I did not understand but let me say something about rivers.

Whatever the number of nodes you have, one single river will only run on a single node.
So I don't see any relative numbers between the amount of kafka partitions and the number of nodes.

Does it help?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 13 mars 2013 à 09:34, 田晓亮 xiaoliang.tian@gmail.com a écrit :

Hi,I am writing a kafka river plugin

here is the thing:
for example
There is 5 partition in kafka

So I think the river plugin should only read 1 partition ,like that

  Map<String, Integer> topicCountMap = new ConcurrentHashMap<String, Integer>();
  topicCountMap.put(topic, new Integer(1));

  Map<String, List<KafkaMessageStream<Message>>> consumerMap = consumer
  		.createMessageStreams(topicCountMap);

And the Elasticsearch cluster should be consist of not more than 5 node,is that right?

thanks very much.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks,u said u don't understand, mean u don't understand my word or kafka?

according to your word,I think If I use a kafka river in es,The es nodes
number must be not more than the number of kafka partition.

thanks again

2013/3/13 David Pilato david@pilato.fr

I'm afraid I did not understand but let me say something about rivers.

Whatever the number of nodes you have, one single river will only run on a
single node.
So I don't see any relative numbers between the amount of kafka partitions
and the number of nodes.

Does it help?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 13 mars 2013 à 09:34, 田晓亮 xiaoliang.tian@gmail.com a écrit :

Hi,I am writing a kafka river plugin

here is the thing:
for example
There is 5 partition in kafka

So I think the river plugin should only read 1 partition ,like that

Map<String, Integer> topicCountMap = new ConcurrentHashMap<String,
Integer>();
topicCountMap.put(topic, new Integer(1));

Map<String, List<KafkaMessageStream>> consumerMap = consumer
.createMessageStreams(topicCountMap);

And the Elasticsearch cluster should be consist of not more than 5
node,is that right?

thanks very much.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/R2tpcHxotX8/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.