# River-wikipedia does not index all pages

**URL:** <https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585>\
**Category:** Elasticsearch\
**Created:** [September 13, 2013, 5:26am UTC](https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585 "2013-09-13T05:26:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [September 13, 2013, 5:26am UTC](https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585/1 "2013-09-13T05:26:17Z")

</div>

Hi all,

I use river-wikipedia to index Japanese wikipedia xml.  
When I use parameter "bulk\_size" : 10000, the number of indexed documents is 1540000.  
But xml include 1546721 pages.

I have a question after seeing WikipediaRiver.java source code.

Probably, the reason is that PageCallback.processBulkIfNeeded() method index document only ,  
if the number of buffering documents is more than bulkSize .

I suppose WikipediaRiver.close() method index the remainining documents,  
only this method is useful deleting river settings.

What do you think about it?

* * *

Jun Ohtani  
twitter : [http://twitter.com/johtani](http://twitter.com/johtani)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 13, 2013, 6:05am UTC](https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585/2 "2013-09-13T06:05:58Z")

</div>

You're probably right. I fixed something like this in other rivers.  
Could you open an issue in wikipedia river project?

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 13 sept. 2013 à 07:26, Jun Ohtani [johtani@gmail.com](mailto:johtani@gmail.com) a écrit :

Hi all,

I use river-wikipedia to index Japanese wikipedia xml.  
When I use parameter "bulk\_size" : 10000, the number of indexed documents is 1540000.  
But xml include 1546721 pages.

I have a question after seeing WikipediaRiver.java source code.

Probably, the reason is that PageCallback.processBulkIfNeeded() method index document only ,  
if the number of buffering documents is more than bulkSize .

I suppose WikipediaRiver.close() method index the remainining documents,  
only this method is useful deleting river settings.

What do you think about it?

* * *

Jun Ohtani  
twitter : [http://twitter.com/johtani](http://twitter.com/johtani)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [September 13, 2013, 6:25am UTC](https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585/3 "2013-09-13T06:25:25Z")

</div>

Hi David,

Thanks for replay.

I open a issue #12 in github project.

I have another idea.  
Currently, processing bulk is only bulkSize check.  
I suggest that PageCallback.processBulkIfNeeded process bulk after interval specified parameter, like a Solr auto-commit .  
But I have no idea to implement this at the present time.

I will try to think about that a little as well.

* * *

Jun Ohtani  
blog : [http://blog.johtani.info](http://blog.johtani.info)

On 2013/09/13, at 15:05, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:

> You're probably right. I fixed something like this in other rivers.  
> Could you open an issue in wikipedia river project?
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 13 sept. 2013 à 07:26, Jun Ohtani [johtani@gmail.com](mailto:johtani@gmail.com) a écrit :
> 
> Hi all,
> 
> I use river-wikipedia to index Japanese wikipedia xml.  
> When I use parameter "bulk\_size" : 10000, the number of indexed documents is 1540000.  
> But xml include 1546721 pages.
> 
> I have a question after seeing WikipediaRiver.java source code.
> 
> Probably, the reason is that PageCallback.processBulkIfNeeded() method index document only ,  
> if the number of buffering documents is more than bulkSize .
> 
> I suppose WikipediaRiver.close() method index the remainining documents,  
> only this method is useful deleting river settings.
> 
> What do you think about it?
> 
> * * *
> 
> Jun Ohtani  
> twitter : [http://twitter.com/johtani](http://twitter.com/johtani)
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 13, 2013, 6:38am UTC](https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585/4 "2013-09-13T06:38:19Z")

</div>

We have now a nice BulkProcessor class which handle that properly.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 13 sept. 2013 à 08:25, Jun Ohtani [johtani@gmail.com](mailto:johtani@gmail.com) a écrit :

Hi David,

Thanks for replay.

I open a issue #12 in github project.

I have another idea.  
Currently, processing bulk is only bulkSize check.  
I suggest that PageCallback.processBulkIfNeeded process bulk after interval specified parameter, like a Solr auto-commit .  
But I have no idea to implement this at the present time.

I will try to think about that a little as well.

* * *

Jun Ohtani  
blog : [http://blog.johtani.info](http://blog.johtani.info)

On 2013/09/13, at 15:05, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:

> You're probably right. I fixed something like this in other rivers.  
> Could you open an issue in wikipedia river project?
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 13 sept. 2013 à 07:26, Jun Ohtani [johtani@gmail.com](mailto:johtani@gmail.com) a écrit :
> 
> Hi all,
> 
> I use river-wikipedia to index Japanese wikipedia xml.  
> When I use parameter "bulk\_size" : 10000, the number of indexed documents is 1540000.  
> But xml include 1546721 pages.
> 
> I have a question after seeing WikipediaRiver.java source code.
> 
> Probably, the reason is that PageCallback.processBulkIfNeeded() method index document only ,  
> if the number of buffering documents is more than bulkSize .
> 
> I suppose WikipediaRiver.close() method index the remainining documents,  
> only this method is useful deleting river settings.
> 
> What do you think about it?
> 
> * * *
> 
> Jun Ohtani  
> twitter : [http://twitter.com/johtani](http://twitter.com/johtani)
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [September 13, 2013, 7:56am UTC](https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585/5 "2013-09-13T07:56:17Z")

</div>

Nice !

Thanks David.  
I read deeply BulkProcessor class .

2013/9/13 David Pilato [david@pilato.fr](mailto:david@pilato.fr)

> We have now a nice BulkProcessor class which handle that properly.
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 13 sept. 2013 à 08:25, Jun Ohtani [johtani@gmail.com](mailto:johtani@gmail.com) a écrit :
> 
> Hi David,
> 
> Thanks for replay.
> 
> I open a issue #12 in github project.
> 
> I have another idea.  
> Currently, processing bulk is only bulkSize check.  
> I suggest that PageCallback.processBulkIfNeeded process bulk after  
> interval specified parameter, like a Solr auto-commit .  
> But I have no idea to implement this at the present time.
> 
> I will try to think about that a little as well.
> 
> * * *
> 
> Jun Ohtani  
> blog : [http://blog.johtani.info](http://blog.johtani.info)
> 
> On 2013/09/13, at 15:05, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:
> 
> > You're probably right. I fixed something like this in other rivers.  
> > Could you open an issue in wikipedia river project?
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 13 sept. 2013 à 07:26, Jun Ohtani [johtani@gmail.com](mailto:johtani@gmail.com) a écrit :
> > 
> > Hi all,
> > 
> > I use river-wikipedia to index Japanese wikipedia xml.  
> > When I use parameter "bulk\_size" : 10000, the number of indexed  
> > documents is 1540000.  
> > But xml include 1546721 pages.
> > 
> > I have a question after seeing WikipediaRiver.java source code.
> > 
> > Probably, the reason is that PageCallback.processBulkIfNeeded() method  
> > index document only ,  
> > if the number of buffering documents is more than bulkSize .
> > 
> > I suppose WikipediaRiver.close() method index the remainining documents,  
> > only this method is useful deleting river settings.
> > 
> > What do you think about it?
> > 
> > * * *
> > 
> > Jun Ohtani  
> > twitter : [http://twitter.com/johtani](http://twitter.com/johtani)
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > 
> > --  
> > You received this message because you are subscribed to the Google  
> > Groups "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send  
> > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

## --

Jun Ohtani  
blog : [http://blog.johtani.info](http://blog.johtani.info)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:16am UTC](https://discuss.elastic.co/t/river-wikipedia-does-not-index-all-pages/13585/6 "2017-07-06T02:16:42Z")

</div>

Closed on July 6, 2017, 2:16am UTC