This might seem silly, but What happens when I index files in a repository?
just want to elaborate my question here..
I have around 20GB of files, when I indexed them using fsCrawler it created an index of size 56mb.
(stats from GET _cat/indices?)
I ran a search query to find a name, lets say "Sam".
In the results I can see full file contents in the "hits>source>content".
My question is how can you convert a 20GB repository to 56mb index with full file contents?
what happens when I index a file or repository of files?
Could you please explain or point me to any documentation on this topic. I searched on the official docs but couldnt find any explanation on this.
FSCrawler extracts text and some metadata from a file. Imagine you are crawling a video file of 500mb, no text will be extracted but just few metadata. It means that you are going to index may be 1kb on 500mb. Same goes with PDF or Word documents which has pictures. The pictures won't add any text but contributes to the original size.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.