# Fscrawler doesn't seem to index against Includes **URL:** \ **Category:** Elasticsearch\ **Created:** [October 27, 2017, 3:03pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587 "2017-10-27T15:03:08Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Post date:** [October 27, 2017, 3:03pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/1 "2017-10-27T15:03:08Z")
here is my \_settings file. Basically I'm trying to create an index that includes only files of the types on the includes line. After it finishes, if I go to Kibana and query the index nothing is returned: ``` { "name" : "fullsitedocs", "fs" : { "url" : "\\\\tst-web-20\\W$\\inetpub\\wwwroot", "update_rate" : "15m", "includes" : ["*.htm", "*.html", "*.asp"], "json_support" : false, "filename_as_id" : false, "add_filesize" : true, "remove_deleted" : true, "add_as_inner_object" : false, "store_source" : false, "index_content" : true, "attributes_support" : false, "raw_metadata" : true, "xml_support" : false, "index_folders" : true, "lang_detect" : false, "continue_on_error" : false, "pdf_ocr" : true, "ocr" : { "language" : "eng" } }, "elasticsearch" : { "nodes" : [ { "host" : "10.128.128.106", "port" : 9200, "scheme" : "HTTP" } ], "index" : "fullsiteindex", "bulk_size" : 100, "flush_interval" : "5s", "username" : "elastic", "password" : "elastic123" }, "rest" : { "scheme" : "HTTP", "host" : "10.128.128.106", "port" : 8080, "endpoint" : "fscrawler" } } ``` from Kibana I enter: ``` GET fullsiteindex/_search { "query" : { "match" : { "file.extension": "*.htm" } } } ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 27, 2017, 3:15pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/2 "2017-10-27T15:15:39Z")
I'm not sure this query can work TBH: ```auto "match" : { "file.extension": "*.htm" } ``` Could you try with: ```auto "match" : { "file.extension": "htm" } ``` Instead? But you might be hitting this though: > ---
**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Post date:** [October 27, 2017, 3:27pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/3 "2017-10-27T15:27:11Z")
even if I do `"match_all" : { }` nothing is returned, its an empty index. fscrawler itself seems to run ok. when I add the --debug flag all the messages look good. ---
**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Post date:** [October 30, 2017, 8:37am UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/4 "2017-10-30T08:37:48Z")
Morning dadoonet. I thought I'd try removing includes completely from my settings file and instead just have the following Excludes statement: > "excludes" : ["\*.css", "\*.js", "\*.doc", "\*.pdf", "\*.xls", "\*.png", "\*.jpg", "\*.pps", "\*.asa", "\*.ico", "\*.idx", "\*.swf", "\*.mp4", "\*.scc"], then in kibana: ``` GET fullsiteindex/_search { "query" : { "match_all" : { } } } ``` returns no results. I have pages with .htm, .html and .asp extensions which I would have expected to have been picked up by fscrawler ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 30, 2017, 9:00am UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/5 "2017-10-30T09:00:10Z")
Did you restart FSCrawler with —restart option? —trace option should tell you when files are ignored or indexed. Could you check that html files are not skipped ? ---
**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Post date:** [October 30, 2017, 10:36am UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/6 "2017-10-30T10:36:23Z")
HI Dadoonet, I didn't use the --restart option, so I've just run it again with this option. when it finished, my index is no longer empty, but the excludes as with includes appears not to work. I can see .jpg files for example in my index. ---
**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Post date:** [October 30, 2017, 2:28pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/7 "2017-10-30T14:28:36Z")
HI dadoonet, I think I've cracked it 🙂 I still cant get the includes to work, but on the exclude's I didn't realise that it was case sensitive. So when my exclude had just \ ***.jpg** , it included \ ***.JPG**. That was my first issue in actually setting the correct data in the index. My second issue was with my query in Kibana. I cant use "match" but instead use "Wildcard" in the form: ``` GET fullsiteindex/_search/ { "query" : { "wildcard": { "file.extension": { "value": "jpg" } } } } ``` This seems to work for my test sample, so I'm now going to run the index against my full data set. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 30, 2017, 2:48pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/8 "2017-10-30T14:48:29Z")
> So when my exclude had just \*.jpg, it included \*.JPG. Interesting. Wanna open an issue about this? I think I should support it. About the query you ran, could you share a sample document? Using wildcard is not what I'd expect. `match` should work. `term` might be better though. ---
**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Post date:** [October 30, 2017, 3:49pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/9 "2017-10-30T15:49:00Z")
Not sure including a sample document would help. I wasn't so much interested in the content, but the type of document. Hence, trying to filter the type of document that goes into the index. My excludes ended up being: > ``` > "excludes" : ["*.jpg", "*.JPG", "*.css", "*.js", "*.doc", "*.DOC", "*.pdf", "*.PDF", "*.xls", "*.png", "*.pps", "*.asa", "*.db", "*.ico", "*.idx", "*.swf", "*.mp4", "*.scc", "*.xlsx", "*.xlsm", "*.gif", "*.tif", "*.tiff", "*.ttf", "*.rfa", "*.cur", "*.psd", "*.dwt", "*.cfg", "*.fla"], > > ``` After which My index when using my wildcard query returned no entries for any in the excludes list and for those that I wanted (htm, html, txt, asp) returned entries. ---
**Author:** ![bilpor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bilpor/32/24043_2.png) [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Post date:** [October 30, 2017, 3:57pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/10 "2017-10-30T15:57:57Z")
I have raised a new issue **#458** ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 27, 2017, 3:58pm UTC](https://discuss.elastic.co/t/fscrawler-doesnt-seem-to-index-against-includes/105587/11 "2017-11-27T15:58:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.