Elasticsearch not indexing all files

Elastic Stack Elasticsearch
1

Hi,

I am trying to index all my files stored in MongoDB using Elasticsearch.
But only 180842 files are indexed whereas I have 1637870 files in my DB. I
tried to delete an create an index and everytime it stops after indexing
180842 files. Any idea why not all documents are indexed?

I checked Elasticsearch log files and there is no error. But I found the
below lines in my log file.

(1) [2013-09-11 02:20:57,539][INFO ][river.mongodb ] [Arsenic] [mongodb][mongodb] Add attachment: 522bef23649dd3bb06a61fd8
(2) [2013-09-11 02:20:57,539][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] Add Attachment: 522bef0fe819cc4b70875a48 to index mongoindex / type files
(3) [2013-09-11 02:20:57,539][INFO ][river.mongodb ] [Arsenic] [mongodb][mongodb] Caught file: 522bef230eb5b705cf8ccd91 - /data/Test.java

Line (2) means that the file is added to my index. Am not sure what Line
(1) and (3) means. Does that mean that those files are not added to index?

NOTE: I used the below code to create index:

curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{ "type": "mongodb",
"mongodb": {
"db": "submission_data",
"collection": "fs",
"gridfs": true
},
"index": {
"name": "mongoindex",
"type": "files"
}
}'

Thanks,

Kiruthika

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Could anyone please help me in this issue??? I am not able to index all the
files using Elasticsearch....

Thanks,
Kiruthika

On Wednesday, September 11, 2013 2:46:47 AM UTC-7, KS wrote:

Hi,

I am trying to index all my files stored in MongoDB using Elasticsearch.
But only 180842 files are indexed whereas I have 1637870 files in my DB. I
tried to delete an create an index and everytime it stops after indexing
180842 files. Any idea why not all documents are indexed?

I checked Elasticsearch log files and there is no error. But I found the
below lines in my log file.

(1) [2013-09-11 02:20:57,539][INFO ][river.mongodb ] [Arsenic] [mongodb][mongodb] Add attachment: 522bef23649dd3bb06a61fd8
(2) [2013-09-11 02:20:57,539][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] Add Attachment: 522bef0fe819cc4b70875a48 to index mongoindex / type files
(3) [2013-09-11 02:20:57,539][INFO ][river.mongodb ] [Arsenic] [mongodb][mongodb] Caught file: 522bef230eb5b705cf8ccd91 - /data/Test.java

Line (2) means that the file is added to my index. Am not sure what Line
(1) and (3) means. Does that mean that those files are not added to index?

NOTE: I used the below code to create index:

curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{ "type": "mongodb",
"mongodb": {
"db": "submission_data",
"collection": "fs",
"gridfs": true
},
"index": {
"name": "mongoindex",
"type": "files"
}
}'

Thanks,

Kiruthika

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

It could be useful to start your elasticsearch node in debug mode and check
the queries sent to the mongodb. It would give you hints on why there are
less documents.

To enable debug mode logging: open /config/logging.yml and change the first
line from rootLogger: INFO to rootLogger: DEBUG

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi, I tried it with DEBUG option too, I am not getting any error in the log
file but still I am not able to index all the files.

Thanks,
Kiruthika

On Thursday, September 12, 2013 12:03:23 AM UTC-7, sina.t...@gmail.com
wrote:

It could be useful to start your elasticsearch node in debug mode and
check the queries sent to the mongodb. It would give you hints on why there
are less documents.

To enable debug mode logging: open /config/logging.yml and change the
first line from rootLogger: INFO to rootLogger: DEBUG

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

If you are running elasticsearch in debug mode you should be able to see
the queries issued against mongodb. By looking at those queries you could
verify if you are getting all the documents from mongodb to elasticsearch.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.