ElasticSearch and RabbitMQ as data source

Elastic Stack Elasticsearch
1

Hi all,

I am new to RabbitMQ and ElasticSearch. I am trying to create a setup where
RabbitMQ collects messages from some sources and publish then in
ElasticSearch for indexing.
What I have done so far:

Installed ElasticSearch 0.9.2 on Ubuntu 12.04
Created index named rabbit2 with these mappings:
{
"settings" : {
"number_of_shards" : 5
},
"mappings" : {
"logentry" : {
"_all" : {"enabled" : false},
"Metadata" : { "type" : "string", "index" : "not_analyzed" },
"Categories" : {"type" : "string", "index" : "not_analyzed" },
"ExtendedProperties" : {
"type" : "object",
"properties" : {
"DocumentKey": {"type" : "string", "index" : "not_analyzed"},
"Message": {"type" : "string", "index" : "analyzed"},
"ErrorCode": {"type" : "integer", "store" : "yes" },
"InnerMessage": {"type" : "string", "index": "analyzed"}
}
},
"Message": {"type" : "string", "index" : "not_analyzed"},
"Timestamp": {"type" : "date", "format" : "date_time" }
}
}
}

Installed RabbitMQ river and configured it with this settings
{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "192.168.106.11",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "playground",
"queue" : "platform.flume.queue",
"exchange" : "platform.flume.exchange",
"routing_key" : "flume",
"exchange_declare" : false,
"exchange_type" : "direct",
"exchange_durable" : false,
"queue_declare" : false,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false
},
"index" : {
"name": "rabbit2",
"type" : "logentry",
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}

ES log shows no error, RabbitMQ Web UI shows that there is a consumer
connected with the IP address of the ES server
RabbitMQ exchange and queue are predefined and already existing.

I have published from RabbitMQ Web UI following message:

{ "create": { "_index": "rabbit2", "_type": "logentry", "_id": 2 }}
{"Metadata":
["PLATFORM-PL1","10.176.130.201;","Platform.CPBC.Container.exe","Platform.CPBC.Container","5584",
null,"4020","0","Error"],"Categories":
["PLATFORM-PL1.platform.cpbc.data"],"ExtendedProperties": {"DocumentKey":
"tenant-alerts_1e5d2fa7-0e7a-4f02-b91a-cf09e13e66af", "Message": "Failed to
process response: Not found", "ErrorCode": 1, "InnerMessage": "Failed to
process response: Not found" }, "Message": "Document was not found.",
"Timestamp": "2013-06-26T18:00:07.6122425Z"}

Unfortunately the message is not consumed by ES and stays as
"Unacknowledged".

My question what I am doing wrong? Did I created wrong configuration or
missed something?
Is there somthing wrong with the message that I published?

Thanks for your help in advance.

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Anything in logs?

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 4 juil. 2013 à 11:06, Николай Колев lem00na@gmail.com a écrit :

Hi all,

I am new to RabbitMQ and Elasticsearch. I am trying to create a setup where RabbitMQ collects messages from some sources and publish then in Elasticsearch for indexing.
What I have done so far:

Installed Elasticsearch 0.9.2 on Ubuntu 12.04
Created index named rabbit2 with these mappings:
{
"settings" : {
"number_of_shards" : 5
},
"mappings" : {
"logentry" : {
"_all" : {"enabled" : false},
"Metadata" : { "type" : "string", "index" : "not_analyzed" },
"Categories" : {"type" : "string", "index" : "not_analyzed" },
"ExtendedProperties" : {
"type" : "object",
"properties" : {
"DocumentKey": {"type" : "string", "index" : "not_analyzed"},
"Message": {"type" : "string", "index" : "analyzed"},
"ErrorCode": {"type" : "integer", "store" : "yes" },
"InnerMessage": {"type" : "string", "index": "analyzed"}
}
},
"Message": {"type" : "string", "index" : "not_analyzed"},
"Timestamp": {"type" : "date", "format" : "date_time" }
}
}
}

Installed RabbitMQ river and configured it with this settings
{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "192.168.106.11",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "playground",
"queue" : "platform.flume.queue",
"exchange" : "platform.flume.exchange",
"routing_key" : "flume",
"exchange_declare" : false,
"exchange_type" : "direct",
"exchange_durable" : false,
"queue_declare" : false,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false
},
"index" : {
"name": "rabbit2",
"type" : "logentry",
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}

ES log shows no error, RabbitMQ Web UI shows that there is a consumer connected with the IP address of the ES server
RabbitMQ exchange and queue are predefined and already existing.

I have published from RabbitMQ Web UI following message:

{ "create": { "_index": "rabbit2", "_type": "logentry", "_id": 2 }}
{"Metadata": ["PLATFORM-PL1","10.176.130.201;","Platform.CPBC.Container.exe","Platform.CPBC.Container","5584", null,"4020","0","Error"],"Categories": ["PLATFORM-PL1.platform.cpbc.data"],"ExtendedProperties": {"DocumentKey": "tenant-alerts_1e5d2fa7-0e7a-4f02-b91a-cf09e13e66af", "Message": "Failed to process response: Not found", "ErrorCode": 1, "InnerMessage": "Failed to process response: Not found" }, "Message": "Document was not found.", "Timestamp": "2013-06-26T18:00:07.6122425Z"}

Unfortunately the message is not consumed by ES and stays as "Unacknowledged".

My question what I am doing wrong? Did I created wrong configuration or missed something?
Is there somthing wrong with the message that I published?

Thanks for your help in advance.

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi David,

There were some errors while I was trying to figure what I have to do but
now it seems that it is correct and works:
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing]
[_river] creating index, cause [auto(index api)], shards [1]/[1], mappings

[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] creating rabbitmq river, addresses
[[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)

Is there and option to make it more verbose?

best regards,
nickolay kolev

04 юли 2013, четвъртък, 12:22:57 UTC+3, David Pilato написа:

Anything in logs?

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 4 juil. 2013 à 11:06, Николай Колев <lem...@gmail.com <javascript:>> a
écrit :

Hi all,

I am new to RabbitMQ and Elasticsearch. I am trying to create a setup
where RabbitMQ collects messages from some sources and publish then in
Elasticsearch for indexing.
What I have done so far:

Installed Elasticsearch 0.9.2 on Ubuntu 12.04
Created index named rabbit2 with these mappings:
{
"settings" : {
"number_of_shards" : 5
},
"mappings" : {
"logentry" : {
"_all" : {"enabled" : false},
"Metadata" : { "type" : "string", "index" : "not_analyzed" },
"Categories" : {"type" : "string", "index" : "not_analyzed" },
"ExtendedProperties" : {
"type" : "object",
"properties" : {
"DocumentKey": {"type" : "string", "index" : "not_analyzed"},
"Message": {"type" : "string", "index" : "analyzed"},
"ErrorCode": {"type" : "integer", "store" : "yes" },
"InnerMessage": {"type" : "string", "index": "analyzed"}
}
},
"Message": {"type" : "string", "index" : "not_analyzed"},
"Timestamp": {"type" : "date", "format" : "date_time" }
}
}
}

Installed RabbitMQ river and configured it with this settings
{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "192.168.106.11",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "playground",
"queue" : "platform.flume.queue",
"exchange" : "platform.flume.exchange",
"routing_key" : "flume",
"exchange_declare" : false,
"exchange_type" : "direct",
"exchange_durable" : false,
"queue_declare" : false,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false
},
"index" : {
"name": "rabbit2",
"type" : "logentry",
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}

ES log shows no error, RabbitMQ Web UI shows that there is a consumer
connected with the IP address of the ES server
RabbitMQ exchange and queue are predefined and already existing.

I have published from RabbitMQ Web UI following message:

{ "create": { "_index": "rabbit2", "_type": "logentry", "_id": 2 }}
{"Metadata":
["PLATFORM-PL1","10.176.130.201;","Platform.CPBC.Container.exe","Platform.CPBC.Container","5584",
null,"4020","0","Error"],"Categories":
["PLATFORM-PL1.platform.cpbc.data"],"ExtendedProperties": {"DocumentKey":
"tenant-alerts_1e5d2fa7-0e7a-4f02-b91a-cf09e13e66af", "Message": "Failed to
process response: Not found", "ErrorCode": 1, "InnerMessage": "Failed to
process response: Not found" }, "Message": "Document was not found.",
"Timestamp": "2013-06-26T18:00:07.6122425Z"}

Unfortunately the message is not consumed by ES and stays as
"Unacknowledged".

My question what I am doing wrong? Did I created wrong configuration or
missed something?
Is there somthing wrong with the message that I published?

Thanks for your help in advance.

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

These are not errors but only info.
When you created the river, a new document type was created (rabbit_river2). That's what these logs are saying.

If you want to turn debug on, modify logging.yml file and set river.rabbitmq: trace

You should see some details.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 4 juil. 2013 à 11:26, Николай Колев lem00na@gmail.com a écrit :

Hi David,

There were some errors while I was trying to figure what I have to do but now it seems that it is correct and works:
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings
[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing] [_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing] [rabbitmq][rabbit_river2] creating rabbitmq river, addresses [[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing] [_river] update_mapping [rabbit_river2] (dynamic)

Is there and option to make it more verbose?

best regards,
nickolay kolev

04 юли 2013, четвъртък, 12:22:57 UTC+3, David Pilato написа:
Anything in logs?

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 4 juil. 2013 à 11:06, Николай Колев lem...@gmail.com a écrit :

Hi all,

I am new to RabbitMQ and Elasticsearch. I am trying to create a setup where RabbitMQ collects messages from some sources and publish then in Elasticsearch for indexing.
What I have done so far:

Installed Elasticsearch 0.9.2 on Ubuntu 12.04
Created index named rabbit2 with these mappings:
{
"settings" : {
"number_of_shards" : 5
},
"mappings" : {
"logentry" : {
"_all" : {"enabled" : false},
"Metadata" : { "type" : "string", "index" : "not_analyzed" },
"Categories" : {"type" : "string", "index" : "not_analyzed" },
"ExtendedProperties" : {
"type" : "object",
"properties" : {
"DocumentKey": {"type" : "string", "index" : "not_analyzed"},
"Message": {"type" : "string", "index" : "analyzed"},
"ErrorCode": {"type" : "integer", "store" : "yes" },
"InnerMessage": {"type" : "string", "index": "analyzed"}
}
},
"Message": {"type" : "string", "index" : "not_analyzed"},
"Timestamp": {"type" : "date", "format" : "date_time" }
}
}
}

Installed RabbitMQ river and configured it with this settings
{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "192.168.106.11",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "playground",
"queue" : "platform.flume.queue",
"exchange" : "platform.flume.exchange",
"routing_key" : "flume",
"exchange_declare" : false,
"exchange_type" : "direct",
"exchange_durable" : false,
"queue_declare" : false,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false
},
"index" : {
"name": "rabbit2",
"type" : "logentry",
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}

ES log shows no error, RabbitMQ Web UI shows that there is a consumer connected with the IP address of the ES server
RabbitMQ exchange and queue are predefined and already existing.

I have published from RabbitMQ Web UI following message:

{ "create": { "_index": "rabbit2", "_type": "logentry", "_id": 2 }}
{"Metadata": ["PLATFORM-PL1","10.176.130.201;","Platform.CPBC.Container.exe","Platform.CPBC.Container","5584", null,"4020","0","Error"],"Categories": ["PLATFORM-PL1.platform.cpbc.data"],"ExtendedProperties": {"DocumentKey": "tenant-alerts_1e5d2fa7-0e7a-4f02-b91a-cf09e13e66af", "Message": "Failed to process response: Not found", "ErrorCode": 1, "InnerMessage": "Failed to process response: Not found" }, "Message": "Document was not found.", "Timestamp": "2013-06-26T18:00:07.6122425Z"}

Unfortunately the message is not consumed by ES and stays as "Unacknowledged".

My question what I am doing wrong? Did I created wrong configuration or missed something?
Is there somthing wrong with the message that I published?

Thanks for your help in advance.

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi David,

I have added this option and restarted ES server. Here is the log:
[2013-07-04 11:23:23,772][INFO ][cluster.metadata ] [Man-Thing]
[_river] deleting index
[2013-07-04 11:23:23,774][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] closing rabbitmq river
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing]
[_river] creating index, cause [auto(index api)], shards [1]/[1], mappings

[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] creating rabbitmq river, addresses
[[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 12:38:56,592][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: stopping ...
[2013-07-04 12:38:56,632][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] closing rabbitmq river
[2013-07-04 12:38:56,705][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: stopped
[2013-07-04 12:38:56,705][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: closing ...
[2013-07-04 12:38:56,720][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: closed
[2013-07-04 12:38:59,131][INFO ][node ] [X-23]
{0.90.2}[27353]: initializing ...
[2013-07-04 12:38:59,149][INFO ][plugins ] [X-23] loaded
[river-rabbitmq], sites [bigdesk, HQ, head]
[2013-07-04 12:39:03,145][INFO ][node ] [X-23]
{0.90.2}[27353]: initialized
[2013-07-04 12:39:03,145][INFO ][node ] [X-23]
{0.90.2}[27353]: starting ...
[2013-07-04 12:39:03,296][INFO ][transport ] [X-23]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
192.168.104.134:9300]}
[2013-07-04 12:39:06,402][INFO ][cluster.service ] [X-23]
new_master [X-23][Jwk5Jl-PR-68g4c1VA4vSQ][inet[/192.168.104.134:9300]],
reason: zen-disco-join (elected_as_master)
[2013-07-04 12:39:06,450][INFO ][discovery ] [X-23]
elasticsearch/Jwk5Jl-PR-68g4c1VA4vSQ
[2013-07-04 12:39:06,480][INFO ][http ] [X-23]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
192.168.104.134:9200]}
[2013-07-04 12:39:06,481][INFO ][node ] [X-23]
{0.90.2}[27353]: started
[2013-07-04 12:39:07,638][INFO ][gateway ] [X-23]
recovered [7] indices into cluster_state
[2013-07-04 12:39:07,713][INFO ][river.rabbitmq ] [X-23]
[rabbitmq][rabbit_river2] creating rabbitmq river, addresses
[[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 12:43:43,981][TRACE][river.rabbitmq ] [X-23]
[rabbitmq][rabbit_river2] executing bulk with [0] actions
[2013-07-04 12:44:30,979][TRACE][river.rabbitmq ] [X-23]
[rabbitmq][rabbit_river2] executing bulk with [0] actions

To me it looks like the message that I push in RabbitMQ queue is incorrect
but I cannot figure out what is the problem. Any suggestions? Are any
docs about the structure of bulk message – I was not able to find any

Thanks

best regards,
Nickolay kolev

2013/7/4 David Pilato david@pilato.fr

These are not errors but only info.
When you created the river, a new document type was created
(rabbit_river2). That's what these logs are saying.

If you want to turn debug on, modify logging.yml file and set
river.rabbitmq: trace

You should see some details.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 4 juil. 2013 à 11:26, Николай Колев lem00na@gmail.com a écrit :

Hi David,

There were some errors while I was trying to figure what I have to do but
now it seems that it is correct and works:
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing]
[_river] creating index, cause [auto(index api)], shards [1]/[1], mappings

[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] creating rabbitmq river, addresses
[[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)

Is there and option to make it more verbose?

best regards,
nickolay kolev

04 юли 2013, четвъртък, 12:22:57 UTC+3, David Pilato написа:

Anything in logs?

David Pilato | Technical Advocate | Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
|* @scrutmydocs https://twitter.com/scrutmydocs

Le 4 juil. 2013 à 11:06, Николай Колев lem...@gmail.com a écrit :

Hi all,

I am new to RabbitMQ and Elasticsearch. I am trying to create a setup
where RabbitMQ collects messages from some sources and publish then in
Elasticsearch for indexing.
What I have done so far:

Installed Elasticsearch 0.9.2 on Ubuntu 12.04
Created index named rabbit2 with these mappings:
{
"settings" : {
"number_of_shards" : 5
},
"mappings" : {
"logentry" : {
"_all" : {"enabled" : false},
"Metadata" : { "type" : "string", "index" : "not_analyzed" },
"Categories" : {"type" : "string", "index" : "not_analyzed" },
"ExtendedProperties" : {
"type" : "object",
"properties" : {
"DocumentKey": {"type" : "string", "index" : "not_analyzed"},
"Message": {"type" : "string", "index" : "analyzed"},
"ErrorCode": {"type" : "integer", "store" : "yes" },
"InnerMessage": {"type" : "string", "index": "analyzed"}
}
},
"Message": {"type" : "string", "index" : "not_analyzed"},
"Timestamp": {"type" : "date", "format" : "date_time" }
}
}
}

Installed RabbitMQ river and configured it with this settings
{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "192.168.106.11",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "playground",
"queue" : "platform.flume.queue",
"exchange" : "platform.flume.exchange",
"routing_key" : "flume",
"exchange_declare" : false,
"exchange_type" : "direct",
"exchange_durable" : false,
"queue_declare" : false,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false
},
"index" : {
"name": "rabbit2",
"type" : "logentry",
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}

ES log shows no error, RabbitMQ Web UI shows that there is a consumer
connected with the IP address of the ES server
RabbitMQ exchange and queue are predefined and already existing.

I have published from RabbitMQ Web UI following message:

{ "create": { "_index": "rabbit2", "_type": "logentry", "_id": 2 }}
{"Metadata": ["PLATFORM-PL1","10.176.130.201;","Platform.CPBC.
Container.exe","Platform.CPBC.**Container","5584",
null,"4020","0","Error"],"**Categories": ["PLATFORM-PL1.platform.cpbc.**data"],"ExtendedProperties":
{"DocumentKey": "tenant-alerts_1e5d2fa7-0e7a-4f02-b91a-cf09e13e66af",
"Message": "Failed to process response: Not found", "ErrorCode": 1,
"InnerMessage": "Failed to process response: Not found" }, "Message":
"Document was not found.", "Timestamp": "2013-06-26T18:00:07.6122425Z"}

Unfortunately the message is not consumed by ES and stays as
"Unacknowledged".

My question what I am doing wrong? Did I created wrong configuration or
missed something?
Is there somthing wrong with the message that I published?

Thanks for your help in advance.

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Check as well that your last line has a carriage return.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 juil. 2013 à 11:49, lem0na lem00na@gmail.com a écrit :

Hi David,

I have added this option and restarted ES server. Here is the log:
[2013-07-04 11:23:23,772][INFO ][cluster.metadata ] [Man-Thing] [_river] deleting index
[2013-07-04 11:23:23,774][INFO ][river.rabbitmq ] [Man-Thing] [rabbitmq][rabbit_river2] closing rabbitmq river
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings
[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing] [_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing] [rabbitmq][rabbit_river2] creating rabbitmq river, addresses [[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing] [_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 12:38:56,592][INFO ][node ] [Man-Thing] {0.90.2}[5943]: stopping ...
[2013-07-04 12:38:56,632][INFO ][river.rabbitmq ] [Man-Thing] [rabbitmq][rabbit_river2] closing rabbitmq river
[2013-07-04 12:38:56,705][INFO ][node ] [Man-Thing] {0.90.2}[5943]: stopped
[2013-07-04 12:38:56,705][INFO ][node ] [Man-Thing] {0.90.2}[5943]: closing ...
[2013-07-04 12:38:56,720][INFO ][node ] [Man-Thing] {0.90.2}[5943]: closed
[2013-07-04 12:38:59,131][INFO ][node ] [X-23] {0.90.2}[27353]: initializing ...
[2013-07-04 12:38:59,149][INFO ][plugins ] [X-23] loaded [river-rabbitmq], sites [bigdesk, HQ, head]
[2013-07-04 12:39:03,145][INFO ][node ] [X-23] {0.90.2}[27353]: initialized
[2013-07-04 12:39:03,145][INFO ][node ] [X-23] {0.90.2}[27353]: starting ...
[2013-07-04 12:39:03,296][INFO ][transport ] [X-23] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.104.134:9300]}
[2013-07-04 12:39:06,402][INFO ][cluster.service ] [X-23] new_master [X-23][Jwk5Jl-PR-68g4c1VA4vSQ][inet[/192.168.104.134:9300]], reason: zen-disco-join (elected_as_master)
[2013-07-04 12:39:06,450][INFO ][discovery ] [X-23] elasticsearch/Jwk5Jl-PR-68g4c1VA4vSQ
[2013-07-04 12:39:06,480][INFO ][http ] [X-23] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.104.134:9200]}
[2013-07-04 12:39:06,481][INFO ][node ] [X-23] {0.90.2}[27353]: started
[2013-07-04 12:39:07,638][INFO ][gateway ] [X-23] recovered [7] indices into cluster_state
[2013-07-04 12:39:07,713][INFO ][river.rabbitmq ] [X-23] [rabbitmq][rabbit_river2] creating rabbitmq river, addresses [[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 12:43:43,981][TRACE][river.rabbitmq ] [X-23] [rabbitmq][rabbit_river2] executing bulk with [0] actions
[2013-07-04 12:44:30,979][TRACE][river.rabbitmq ] [X-23] [rabbitmq][rabbit_river2] executing bulk with [0] actions

To me it looks like the message that I push in RabbitMQ queue is incorrect but I cannot figure out what is the problem. Any suggestions? Are any docs about the structure of bulk message – I was not able to find any

Thanks

best regards,
Nickolay kolev

2013/7/4 David Pilato david@pilato.fr

These are not errors but only info.
When you created the river, a new document type was created (rabbit_river2). That's what these logs are saying.

If you want to turn debug on, modify logging.yml file and set river.rabbitmq: trace

You should see some details.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 4 juil. 2013 à 11:26, Николай Колев lem00na@gmail.com a écrit :

Hi David,

There were some errors while I was trying to figure what I have to do but now it seems that it is correct and works:
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings
[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing] [_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing] [rabbitmq][rabbit_river2] creating rabbitmq river, addresses [[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing] [_river] update_mapping [rabbit_river2] (dynamic)

Is there and option to make it more verbose?

best regards,
nickolay kolev

04 юли 2013, четвъртък, 12:22:57 UTC+3, David Pilato написа:

Anything in logs?

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 4 juil. 2013 à 11:06, Николай Колев lem...@gmail.com a écrit :

Hi all,

I am new to RabbitMQ and Elasticsearch. I am trying to create a setup where RabbitMQ collects messages from some sources and publish then in Elasticsearch for indexing.
What I have done so far:

Installed Elasticsearch 0.9.2 on Ubuntu 12.04
Created index named rabbit2 with these mappings:
{
"settings" : {
"number_of_shards" : 5
},
"mappings" : {
"logentry" : {
"_all" : {"enabled" : false},
"Metadata" : { "type" : "string", "index" : "not_analyzed" },
"Categories" : {"type" : "string", "index" : "not_analyzed" },
"ExtendedProperties" : {
"type" : "object",
"properties" : {
"DocumentKey": {"type" : "string", "index" : "not_analyzed"},
"Message": {"type" : "string", "index" : "analyzed"},
"ErrorCode": {"type" : "integer", "store" : "yes" },
"InnerMessage": {"type" : "string", "index": "analyzed"}
}
},
"Message": {"type" : "string", "index" : "not_analyzed"},
"Timestamp": {"type" : "date", "format" : "date_time" }
}
}
}

Installed RabbitMQ river and configured it with this settings
{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "192.168.106.11",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "playground",
"queue" : "platform.flume.queue",
"exchange" : "platform.flume.exchange",
"routing_key" : "flume",
"exchange_declare" : false,
"exchange_type" : "direct",
"exchange_durable" : false,
"queue_declare" : false,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false
},
"index" : {
"name": "rabbit2",
"type" : "logentry",
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}

ES log shows no error, RabbitMQ Web UI shows that there is a consumer connected with the IP address of the ES server
RabbitMQ exchange and queue are predefined and already existing.

I have published from RabbitMQ Web UI following message:

{ "create": { "_index": "rabbit2", "_type": "logentry", "_id": 2 }}
{"Metadata": ["PLATFORM-PL1","10.176.130.201;","Platform.CPBC.Container.exe","Platform.CPBC.Container","5584", null,"4020","0","Error"],"Categories": ["PLATFORM-PL1.platform.cpbc.data"],"ExtendedProperties": {"DocumentKey": "tenant-alerts_1e5d2fa7-0e7a-4f02-b91a-cf09e13e66af", "Message": "Failed to process response: Not found", "ErrorCode": 1, "InnerMessage": "Failed to process response: Not found" }, "Message": "Document was not found.", "Timestamp": "2013-06-26T18:00:07.6122425Z"}

Unfortunately the message is not consumed by ES and stays as "Unacknowledged".

My question what I am doing wrong? Did I created wrong configuration or missed something?
Is there somthing wrong with the message that I published?

Thanks for your help in advance.

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hi David,

Missing carriage return was the problem. Thanks for your patience and
support.

best regards,
nickolay kolev

04 юли 2013, четвъртък, 13:02:29 UTC+3, David Pilato написа:

Check as well that your last line has a carriage return.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 juil. 2013 à 11:49, lem0na <lem...@gmail.com <javascript:>> a écrit :

Hi David,

I have added this option and restarted ES server. Here is the log:
[2013-07-04 11:23:23,772][INFO ][cluster.metadata ] [Man-Thing]
[_river] deleting index
[2013-07-04 11:23:23,774][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] closing rabbitmq river
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing]
[_river] creating index, cause [auto(index api)], shards [1]/[1], mappings

[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] creating rabbitmq river, addresses
[[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 12:38:56,592][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: stopping ...
[2013-07-04 12:38:56,632][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] closing rabbitmq river
[2013-07-04 12:38:56,705][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: stopped
[2013-07-04 12:38:56,705][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: closing ...
[2013-07-04 12:38:56,720][INFO ][node ] [Man-Thing]
{0.90.2}[5943]: closed
[2013-07-04 12:38:59,131][INFO ][node ] [X-23]
{0.90.2}[27353]: initializing ...
[2013-07-04 12:38:59,149][INFO ][plugins ] [X-23] loaded
[river-rabbitmq], sites [bigdesk, HQ, head]
[2013-07-04 12:39:03,145][INFO ][node ] [X-23]
{0.90.2}[27353]: initialized
[2013-07-04 12:39:03,145][INFO ][node ] [X-23]
{0.90.2}[27353]: starting ...
[2013-07-04 12:39:03,296][INFO ][transport ] [X-23]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
192.168.104.134:9300]}
[2013-07-04 12:39:06,402][INFO ][cluster.service ] [X-23]
new_master [X-23][Jwk5Jl-PR-68g4c1VA4vSQ][inet[/192.168.104.134:9300]],
reason: zen-disco-join (elected_as_master)
[2013-07-04 12:39:06,450][INFO ][discovery ] [X-23]
elasticsearch/Jwk5Jl-PR-68g4c1VA4vSQ
[2013-07-04 12:39:06,480][INFO ][http ] [X-23]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
192.168.104.134:9200]}
[2013-07-04 12:39:06,481][INFO ][node ] [X-23]
{0.90.2}[27353]: started
[2013-07-04 12:39:07,638][INFO ][gateway ] [X-23]
recovered [7] indices into cluster_state
[2013-07-04 12:39:07,713][INFO ][river.rabbitmq ] [X-23]
[rabbitmq][rabbit_river2] creating rabbitmq river, addresses
[[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 12:43:43,981][TRACE][river.rabbitmq ] [X-23]
[rabbitmq][rabbit_river2] executing bulk with [0] actions
[2013-07-04 12:44:30,979][TRACE][river.rabbitmq ] [X-23]
[rabbitmq][rabbit_river2] executing bulk with [0] actions

To me it looks like the message that I push in RabbitMQ queue is incorrect
but I cannot figure out what is the problem. Any suggestions? Are any
docs about the structure of bulk message – I was not able to find any

Thanks

best regards,
Nickolay kolev

2013/7/4 David Pilato <da...@pilato.fr <javascript:>>

These are not errors but only info.
When you created the river, a new document type was created
(rabbit_river2). That's what these logs are saying.

If you want to turn debug on, modify logging.yml file and set
river.rabbitmq: trace

You should see some details.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 4 juil. 2013 à 11:26, Николай Колев <lem...@gmail.com <javascript:>>
a écrit :

Hi David,

There were some errors while I was trying to figure what I have to do but
now it seems that it is correct and works:
[2013-07-04 11:23:40,261][INFO ][cluster.metadata ] [Man-Thing]
[_river] creating index, cause [auto(index api)], shards [1]/[1], mappings

[2013-07-04 11:23:40,305][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)
[2013-07-04 11:23:40,308][INFO ][river.rabbitmq ] [Man-Thing]
[rabbitmq][rabbit_river2] creating rabbitmq river, addresses
[[192.168.106.11:5672]], user [guest], vhost [playground]
[2013-07-04 11:23:40,338][INFO ][cluster.metadata ] [Man-Thing]
[_river] update_mapping [rabbit_river2] (dynamic)

Is there and option to make it more verbose?

best regards,
nickolay kolev

04 юли 2013, четвъртък, 12:22:57 UTC+3, David Pilato написа:

Anything in logs?

David Pilato | Technical Advocate | Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
|* @scrutmydocs https://twitter.com/scrutmydocs

Le 4 juil. 2013 à 11:06, Николай Колев lem...@gmail.com a écrit :

Hi all,

I am new to RabbitMQ and Elasticsearch. I am trying to create a setup
where RabbitMQ collects messages from some sources and publish then in
Elasticsearch for indexing.
What I have done so far:

Installed Elasticsearch 0.9.2 on Ubuntu 12.04
Created index named rabbit2 with these mappings:
{
"settings" : {
"number_of_shards" : 5
},
"mappings" : {
"logentry" : {
"_all" : {"enabled" : false},
"Metadata" : { "type" : "string", "index" : "not_analyzed" },
"Categories" : {"type" : "string", "index" : "not_analyzed" },
"ExtendedProperties" : {
"type" : "object",
"properties" : {
"DocumentKey": {"type" : "string", "index" : "not_analyzed"},
"Message": {"type" : "string", "index" : "analyzed"},
"ErrorCode": {"type" : "integer", "store" : "yes" },
"InnerMessage": {"type" : "string", "index": "analyzed"}
}
},
"Message": {"type" : "string", "index" : "not_analyzed"},
"Timestamp": {"type" : "date", "format" : "date_time" }
}
}
}

Installed RabbitMQ river and configured it with this settings
{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "192.168.106.11",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "playground",
"queue" : "platform.flume.queue",
"exchange" : "platform.flume.exchange",
"routing_key" : "flume",
"exchange_declare" : false,
"exchange_type" : "direct",
"exchange_durable" : false,
"queue_declare" : false,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false
},
"index" : {
"name": "rabbit2",
"type" : "logentry",
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false
}
}

ES log shows no error, RabbitMQ Web UI shows that there is a consumer
connected with the IP address of the ES server
RabbitMQ exchange and queue are predefined and already existing.

I have published from RabbitMQ Web UI following message:

{ "create": { "_index": "rabbit2", "_type": "logentry", "_id": 2 }}
{"Metadata": ["PLATFORM-PL1","10.176.130.201;","Platform.CPBC.
Container.exe","Platform.CPBC.**Container","5584",
null,"4020","0","Error"],"**Categories": ["PLATFORM-PL1.platform.cpbc.**data"],"ExtendedProperties":
{"DocumentKey": "tenant-alerts_1e5d2fa7-0e7a-4f02-b91a-cf09e13e66af",
"Message": "Failed to process response: Not found", "ErrorCode": 1,
"InnerMessage": "Failed to process response: Not found" }, "Message":
"Document was not found.", "Timestamp": "2013-06-26T18:00:07.6122425Z"
}

Unfortunately the message is not consumed by ES and stays as
"Unacknowledged".

My question what I am doing wrong? Did I created wrong configuration or
missed something?
Is there somthing wrong with the message that I published?

Thanks for your help in advance.

best regards,
nickolay kolev

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Hi All,

I am using the rabbitmq-river plugin for elasticsearch. My configuration
for the river is as follows:

curl -XPUT 'localhost:9200/_river/rabbit_river/_meta' -d '{
"type" : "rabbitmq",
"rabbitmq" : {
"host" : "lbha1.ir.clemson.edu",
"port" : 5672,
"user" : "guest",
"pass" : "guest",
"vhost" : "/",
"queue" : "diagnostics_es",
"exchange" : "ir.diagnostics",
"routing_key" : "ir.diagnostics..",
"exchange_declare" : true,
"exchange_type" : "topic",
"exchange_durable" : true,
"queue_declare" : true,
"queue_bind" : true,
"queue_durable" : true,
"queue_auto_delete" : false,
"heartbeat" : "30m",
"nack_errors" : true
},
"index" : {
"bulk_size" : 100,
"bulk_timeout" : "10ms",
"ordered" : false,
"replication" : "default"
}
}'

However, the logs of elasticsearch keeps give me the following message:
[2014-06-10 11:12:33,354][TRACE][river.rabbitmq ] [Madame Web]
[rabbitmq][rabbit_river] executing bulk with [0] actions

Do you have any idea why this happens?

Thanks!

best,

Yuheng

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/da7442f8-7eef-4044-b4c8-a78110f98186%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.