Bulk Processor

Elastic Stack Elasticsearch
1

I don't quite undertsand what the bulk processor is doing this, I would
like someone to explain how it is upposed to work to make sure I designed
this correctly.
I specify the number of actions 1000.
my feeder keeos pushing documents to it "Its more like a loop iterating
documents folders", and I push eash document to the bulk. I expected the
bulk to queue things until it reaches 1000 docs, then processes the bulk?

Yet, this is how it logs, thie comes from the call back functions of the
bulk processor.

Bulk Called: ID= 1, Actions=33, MB=5.46250
Bulk Called: ID= 2, Actions=29, MB=5.51660
Bulk Succeeded: ID= 1, took= 921 ms
Bulk Called: ID= 3, Actions=12, MB=5.691812
Bulk Succeeded: ID= 2, took= 1526 ms

.....

Bulk Called: ID= 23, Actions=8, MB=5.45294
Bulk Succeeded: ID= 23, took= 751 ms
Bulk Called: ID= 24, Actions=19, MB=5.383918
Bulk Succeeded: ID= 24, took= 331 ms
Bulk Called: ID= 25, Actions=22, MB=5.347542
Bulk Succeeded: ID= 25, took= 694 ms
Bulk Called: ID= 26, Actions=58, MB=5.249195
Bulk Succeeded: ID= 26, took= 583 ms
Bulk Called: ID= 27, Actions=89, MB=5.244396
Bulk Succeeded: ID= 27, took= 588 ms.....

Bulk Called: ID= 47, Actions=17, MB=5.245771 ...

Bulk Succeeded: ID= 47, took= 431 ms

Finished Processing the whole thing

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3f06e4bc-eb79-4dd8-b987-1bf86c028062%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

It also flush docs after a given time, let's say every 5 seconds.
BTW there is a small issue which basically flush the Bulk every n-1 docs instead of n.

Fix is on the way.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 12 mars 2014 à 20:51, ZenMaster80 sabdalla80@gmail.com a écrit :

I don't quite undertsand what the bulk processor is doing this, I would like someone to explain how it is upposed to work to make sure I designed this correctly.
I specify the number of actions 1000.
my feeder keeos pushing documents to it "Its more like a loop iterating documents folders", and I push eash document to the bulk. I expected the bulk to queue things until it reaches 1000 docs, then processes the bulk?

Yet, this is how it logs, thie comes from the call back functions of the bulk processor.

Bulk Called: ID= 1, Actions=33, MB=5.46250
Bulk Called: ID= 2, Actions=29, MB=5.51660
Bulk Succeeded: ID= 1, took= 921 ms
Bulk Called: ID= 3, Actions=12, MB=5.691812
Bulk Succeeded: ID= 2, took= 1526 ms

.....

Bulk Called: ID= 23, Actions=8, MB=5.45294
Bulk Succeeded: ID= 23, took= 751 ms
Bulk Called: ID= 24, Actions=19, MB=5.383918
Bulk Succeeded: ID= 24, took= 331 ms
Bulk Called: ID= 25, Actions=22, MB=5.347542
Bulk Succeeded: ID= 25, took= 694 ms
Bulk Called: ID= 26, Actions=58, MB=5.249195
Bulk Succeeded: ID= 26, took= 583 ms
Bulk Called: ID= 27, Actions=89, MB=5.244396
Bulk Succeeded: ID= 27, took= 588 ms.....

Bulk Called: ID= 47, Actions=17, MB=5.245771 ...

Bulk Succeeded: ID= 47, took= 431 ms

Finished Processing the whole thing

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3f06e4bc-eb79-4dd8-b987-1bf86c028062%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ECA7026F-218A-4AB0-8C50-7AA223FCD1FB%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

David,

Sorry, I didn't quite follow, does it do the flushing automatically or am I
supposed to tell it?

On Wednesday, March 12, 2014 4:05:49 PM UTC-4, David Pilato wrote:

It also flush docs after a given time, let's say every 5 seconds.
BTW there is a small issue which basically flush the Bulk every n-1 docs
instead of n.

Fix is on the way.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 12 mars 2014 à 20:51, ZenMaster80 <sabda...@gmail.com <javascript:>> a
écrit :

I don't quite undertsand what the bulk processor is doing this, I would
like someone to explain how it is upposed to work to make sure I designed
this correctly.
I specify the number of actions 1000.
my feeder keeos pushing documents to it "Its more like a loop iterating
documents folders", and I push eash document to the bulk. I expected the
bulk to queue things until it reaches 1000 docs, then processes the bulk?

Yet, this is how it logs, thie comes from the call back functions of the
bulk processor.

Bulk Called: ID= 1, Actions=33, MB=5.46250
Bulk Called: ID= 2, Actions=29, MB=5.51660
Bulk Succeeded: ID= 1, took= 921 ms
Bulk Called: ID= 3, Actions=12, MB=5.691812
Bulk Succeeded: ID= 2, took= 1526 ms

.....

Bulk Called: ID= 23, Actions=8, MB=5.45294
Bulk Succeeded: ID= 23, took= 751 ms
Bulk Called: ID= 24, Actions=19, MB=5.383918
Bulk Succeeded: ID= 24, took= 331 ms
Bulk Called: ID= 25, Actions=22, MB=5.347542
Bulk Succeeded: ID= 25, took= 694 ms
Bulk Called: ID= 26, Actions=58, MB=5.249195
Bulk Succeeded: ID= 26, took= 583 ms
Bulk Called: ID= 27, Actions=89, MB=5.244396
Bulk Succeeded: ID= 27, took= 588 ms.....

Bulk Called: ID= 47, Actions=17, MB=5.245771 ...

Bulk Succeeded: ID= 47, took= 431 ms

Finished Processing the whole thing

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3f06e4bc-eb79-4dd8-b987-1bf86c028062%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3f06e4bc-eb79-4dd8-b987-1bf86c028062%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9cb96ece-d30d-49a2-bcb4-bb09098094fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

It does it automatically.

You just have to properly call .close() when you stop you application.
It will process the pending requests before actually exiting.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 14 mars 2014 à 15:50:48, ZenMaster80 (sabdalla80@gmail.com) a écrit:

David,

Sorry, I didn't quite follow, does it do the flushing automatically or am I supposed to tell it?

On Wednesday, March 12, 2014 4:05:49 PM UTC-4, David Pilato wrote:
It also flush docs after a given time, let's say every 5 seconds.
BTW there is a small issue which basically flush the Bulk every n-1 docs instead of n.

Fix is on the way.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 12 mars 2014 à 20:51, ZenMaster80 sabda...@gmail.com a écrit :

I don't quite undertsand what the bulk processor is doing this, I would like someone to explain how it is upposed to work to make sure I designed this correctly.
I specify the number of actions 1000.
my feeder keeos pushing documents to it "Its more like a loop iterating documents folders", and I push eash document to the bulk. I expected the bulk to queue things until it reaches 1000 docs, then processes the bulk?

Yet, this is how it logs, thie comes from the call back functions of the bulk processor.

Bulk Called: ID= 1, Actions=33, MB=5.46250
Bulk Called: ID= 2, Actions=29, MB=5.51660
Bulk Succeeded: ID= 1, took= 921 ms
Bulk Called: ID= 3, Actions=12, MB=5.691812
Bulk Succeeded: ID= 2, took= 1526 ms

.....

Bulk Called: ID= 23, Actions=8, MB=5.45294
Bulk Succeeded: ID= 23, took= 751 ms
Bulk Called: ID= 24, Actions=19, MB=5.383918
Bulk Succeeded: ID= 24, took= 331 ms
Bulk Called: ID= 25, Actions=22, MB=5.347542
Bulk Succeeded: ID= 25, took= 694 ms
Bulk Called: ID= 26, Actions=58, MB=5.249195
Bulk Succeeded: ID= 26, took= 583 ms
Bulk Called: ID= 27, Actions=89, MB=5.244396
Bulk Succeeded: ID= 27, took= 588 ms.....

Bulk Called: ID= 47, Actions=17, MB=5.245771 ...

Bulk Succeeded: ID= 47, took= 431 ms

Finished Processing the whole thing

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3f06e4bc-eb79-4dd8-b987-1bf86c028062%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9cb96ece-d30d-49a2-bcb4-bb09098094fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.532319bc.1190cde7.1ccf%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.