I have an index with 6 Crores of records. My usecase is to read the
entire index, check each record, whether it is present in new index or
not.If not I have to index into new index. I used scan and scroll operation
to read the index using JAVA Api. But this process is taking lot of time
i.e., to process 50,000 rcds it is taking 8 min. Can anyone suggest me how
I can configure or change my queries.
I have an index with 6 Crores of records. My usecase is to read the
entire index, check each record, whether it is present in new index or
not.If not I have to index into new index. I used scan and scroll operation
to read the index using JAVA Api. But this process is taking lot of time
i.e., to process 50,000 rcds it is taking 8 min. Can anyone suggest me how
I can configure or change my queries.
I know from experience that scan/scroll can handle batch sizes in the low
thousands without trouble so you should give that a shot. Each scroll call
should be quite quick. It might be a good idea to post a JSON recreation
of your problem so we can see what is happening. Usually the slow part of
the scan/scroll into new index is the batch calls to add the documents into
the new index. And whether or not that is "slow" is really dependant on
the size of the documents, the complexity of any scripts you use on import,
your disk speed, the complexity of your analysis, your cpu speed, the merge
settings you use. That list is roughly in order of how likely I've seen
things effect import speed.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.