RabbitMQ: no queues and state idle when indexing

drjz · January 22, 2015, 11:27am

Hi all,

I have installed the RabbitMQ plugin and added a user "elasticsearch" with
administrator rights in RabbitMQ.

The plugin is loaded fine:
[2015-01-22 11:33:01,974][INFO ][plugins ] [node2] loaded
[river-rabbitmq], sites []

I created the river with these settings:

curl -XPUT 'http://10.99.35.231:9200/_river/rabbitriver/_meta' -d '{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "10.99.35.231",
"port" : 5672,
"user" : "elasticsearch",
"pass" : "elasticsearch",
"vhost" : "/",
"queue" : "elasticsearch",
"exchange" : "elasticsearch",
"routing_key" : "elasticsearch",
"exchange_declare" : true,
"exchange_type" : "direct",
"exchange_durable" : true,
"queue_declare" : true,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false,
"heartbeat" : "30m",
"qos_prefetch_size" : 0,
"qos_prefetch_count" : 10,
"nack_errors" : true
},
"index" : {
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}'

echo ""

curl -XPUT 'http://10.99.35.231:9200/_river/_settings' -d '{
"index" : {
"number_of_replicas" : 0
}
}'

However, when I index to ES using the Bulk API, no queues are created and
the status stays idle, but the data gets indexed to ES "normally"
apparently without using RabbitMQ. I am using the RabbitMQ Web GUI for
monitoring.

Question: what am I doing wrong here?

Thanks!

/JZ

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d9b1a183-71fa-479e-8577-b870254bd71d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · January 22, 2015, 2:21pm

Wondering what you are expecting to see?
Rivers are meant to get data from a source, transform to JSON and index to elasticsearch, like an ETL.

So you push data in RabbitMQ and you should see data coming into elasticsearch.

Are you looking for something else?
BTW, you should look at logstash.

HTH.

David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 janv. 2015 à 12:27, drjz juntezhang@gmail.com a écrit :

Hi all,

I have installed the RabbitMQ plugin and added a user "elasticsearch" with administrator rights in RabbitMQ.

The plugin is loaded fine:
[2015-01-22 11:33:01,974][INFO ][plugins ] [node2] loaded [river-rabbitmq], sites

I created the river with these settings:

curl -XPUT 'http://10.99.35.231:9200/_river/rabbitriver/_meta' -d '{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "10.99.35.231",
"port" : 5672,
"user" : "elasticsearch",
"pass" : "elasticsearch",
"vhost" : "/",
"queue" : "elasticsearch",
"exchange" : "elasticsearch",
"routing_key" : "elasticsearch",
"exchange_declare" : true,
"exchange_type" : "direct",
"exchange_durable" : true,
"queue_declare" : true,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false,
"heartbeat" : "30m",
"qos_prefetch_size" : 0,
"qos_prefetch_count" : 10,
"nack_errors" : true
},
"index" : {
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}'

echo ""

curl -XPUT 'http://10.99.35.231:9200/_river/_settings' -d '{
"index" : {
"number_of_replicas" : 0
}
}'

However, when I index to ES using the Bulk API, no queues are created and the status stays idle, but the data gets indexed to ES "normally" apparently without using RabbitMQ. I am using the RabbitMQ Web GUI for monitoring.

Question: what am I doing wrong here?

Thanks!

/JZ

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d9b1a183-71fa-479e-8577-b870254bd71d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAE5063D-3D8C-4885-8F39-07BA54648873%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

drjz · January 22, 2015, 2:42pm

I have a similar issue as explained
here: https://github.com/elasticsearch/elasticsearch-river-rabbitmq/issues/74

What I expect to see is that RabbitMQ will create a queue and show me the
messages and what is going on.

The solution there is using carriage returns. I have used normal new lines
so far with the Bulk API and I really wondering if that explains why
RabbitMQ stays idle?

/JZ

On Thursday, January 22, 2015 at 3:22:20 PM UTC+1, David Pilato wrote:

Wondering what you are expecting to see?
Rivers are meant to get data from a source, transform to JSON and index to
elasticsearch, like an ETL.

So you push data in RabbitMQ and you should see data coming into
elasticsearch.

Are you looking for something else?
BTW, you should look at logstash.

HTH.

David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 22 janv. 2015 à 12:27, drjz <junte...@gmail.com <javascript:>> a
écrit :

Hi all,

I have installed the RabbitMQ plugin and added a user "elasticsearch" with
administrator rights in RabbitMQ.

The plugin is loaded fine:
[2015-01-22 11:33:01,974][INFO ][plugins ] [node2] loaded
[river-rabbitmq], sites

I created the river with these settings:

curl -XPUT 'http://10.99.35.231:9200/_river/rabbitriver/_meta' -d '{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "10.99.35.231",
"port" : 5672,
"user" : "elasticsearch",
"pass" : "elasticsearch",
"vhost" : "/",
"queue" : "elasticsearch",
"exchange" : "elasticsearch",
"routing_key" : "elasticsearch",
"exchange_declare" : true,
"exchange_type" : "direct",
"exchange_durable" : true,
"queue_declare" : true,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false,
"heartbeat" : "30m",
"qos_prefetch_size" : 0,
"qos_prefetch_count" : 10,
"nack_errors" : true
},
"index" : {
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}'

echo ""

curl -XPUT 'http://10.99.35.231:9200/_river/_settings' -d '{
"index" : {
"number_of_replicas" : 0
}
}'

However, when I index to ES using the Bulk API, no queues are created and
the status stays idle, but the data gets indexed to ES "normally"
apparently without using RabbitMQ. I am using the RabbitMQ Web GUI for
monitoring.

Question: what am I doing wrong here?

Thanks!

/JZ

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d9b1a183-71fa-479e-8577-b870254bd71d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9b1a183-71fa-479e-8577-b870254bd71d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4cb16268-32c8-4b2a-886d-a154205cc0ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.