# How to avoid ��� chars when using FS Crawler? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 7, 2019, 5:47pm UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996 "2019-11-07T17:47:49Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Danilo\_Felix](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danilo_felix/32/53204_2.png) [@Danilo\_Felix](https://discuss.elastic.co/u/Danilo_Felix)\ **Post date:** [November 7, 2019, 5:47pm UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996/1 "2019-11-07T17:47:49Z")
Hello, I'm trying to use FS Crawler 2.6 in a Windows Server machine to index a huge number of files in my company. It's a very large Windows folders tree in a network drive: 14,92 Tb size, 7,2M files in 2,3M folders. Data are in a remote filer, in the same data center. In many indexed documents, there are ��� chars. Why this happens? How to avoid? My \_settings.json file: ``` { "name" : "disc-files-prd", "fs" : { "url" : "\\\\mycompany\\mycompany\\myfirstfolder", "update_rate" : "120h", "indexed_chars" : 100000, "includes": [ "*/*.doc", "*/*.pdf", "*/*.csv", "*/*.doc", "*/*.docx", "*/*.ods", "*/*.odp", "*/*.odt", "*/*.pdf", "*/*.pps", "*/*.ppsx", "*/*.ppt", "*/*.pptx", "*/*.rtf", "*/*.txt", "*/*.wps", "*/*.xls", "*/*.xlsx", "*/*.xlsm", "*/*.xps" ], "excludes": [ "*/~*", "*/*.tmp", "*/*.eml", "*/*.jpg", "*/*.png", "*/ISC/NP*", "*/ISC_INTL/NP*", "*/ISC_OSC/NP*", "*/_CONTINGENCIA/*", "*/_HISTORICO/*" ], "json_support" : false, "follow_symlink" : false, "filename_as_id" : false, "add_filesize" : true, "remove_deleted" : false, "add_as_inner_object" : true, "store_source" : false, "index_content" : true, "attributes_support" : false, "raw_metadata" : true, "xml_support" : false, "index_folders" : true, "lang_detect" : false, "continue_on_error" : true, "ignore_above": "20mb", "pdf_ocr" : false, "ocr" : { "language" : "eng" } }, "elasticsearch" : { "nodes" : [ { "url" : "https://elasticsearch.mycompany.com.br" } ], "bulk_size" : 100, "flush_interval" : "4s", "byte_size" : "5mb", "path_prefix" : "disc", "username" : "xxxxx-all", "password" : " *******", "index" : "xxxxxx-docs", "index_folder": "xxxxxxx-folders" }, "rest" : { "url" : "http://127.0.0.1:8080/fscrawler" } } ``` One indexed example: ``` { "took": 113, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": 718042, "max_score": 1.0, "hits": [ { "_index": "disc-files-docs", "_type": "_doc", "_id": "c4ca9c74b40f690fc7f1b4442a6a6a9", "_score": 1.0, "_source": { "content": "�� �ɘ����������� �ʰ�������{2474569B-416F-43D7-99A6-5F5AB8242EA0}��������������������������������������������������������������������������������������������@�����Ā�€���$GK$ ISC_INTL_MAPS_AI_NP-3 ``` How to avoid ���� chars ??? Thanks for help ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 7, 2019, 7:28pm UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996/2 "2019-11-07T19:28:06Z")
This is a file encoding issue I guess. Make sure you're using UTF8 everywhere. Do you have a sample document I can use to reproduce the problem? ---
**Author:** ![Danilo\_Felix](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danilo_felix/32/53204_2.png) [@Danilo\_Felix](https://discuss.elastic.co/u/Danilo_Felix)\ **Post date:** [November 7, 2019, 7:46pm UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996/3 "2019-11-07T19:46:25Z")
Yes, how i send to you? By email? It is a PDF file. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 8, 2019, 5:36am UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996/5 "2019-11-08T05:36:47Z")
You can share it on whatever sharing service and share the link here? ---
**Author:** ![Danilo\_Felix](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danilo_felix/32/53204_2.png) [@Danilo\_Felix](https://discuss.elastic.co/u/Danilo_Felix)\ **Post date:** [November 8, 2019, 12:07pm UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996/6 "2019-11-08T12:07:01Z")
It is here. > **[Sindipetro RJ \_ PCR\_ Jurídico do Sindipetro-RJ responde.pdf](https://drive.google.com/file/d/0B_d8ASy9pH9hSnlycUYzbWkxakxFSldYQVVJdmlMSExSNnc4/view?usp=sharing_eip&ts=5dc55965)** > > Google Drive file. Thank you very much ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 10, 2019, 1:12pm UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996/7 "2019-11-10T13:12:02Z")
So I tried your file and it seems that PDFBox is sending some warnings: ```auto 00:05:23,158 WARN [o.a.p.p.f.PDSimpleFont] No Unicode mapping for 0 (0) in font T1 00:05:23,173 WARN [o.a.p.p.f.PDSimpleFont] No Unicode mapping for 1 (2) in font T1 00:05:23,174 WARN [o.a.p.p.f.PDSimpleFont] No Unicode mapping for i255 (3) in font T1 00:05:23,174 WARN [o.a.p.p.f.PDSimpleFont] No Unicode mapping for 3 (4) in font T1 00:05:23,175 WARN [o.a.p.p.f.PDSimpleFont] No Unicode mapping for 4 (5) in font T1 00:05:23,176 WARN [o.a.p.p.f.PDSimpleFont] No Unicode mapping for 5 (6) in font T1 ``` Not sure if it's related though. I'd try to raise an issue in PDFBox project or in Tika mailing list and see if this is a bug or not. BTW I created a branch to test this: [https://github.com/dadoonet/fscrawler/tree/test/pdf-encoding](https://github.com/dadoonet/fscrawler/tree/test/pdf-encoding) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2019, 1:12pm UTC](https://discuss.elastic.co/t/how-to-avoid-chars-when-using-fs-crawler/206996/8 "2019-12-08T13:12:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.