The FSCrawler team is pleased to announce the FSCrawler 2.7 release!
FSCrawler
FS Crawler offers a simple way to index binary files into elasticsearch.
Usage
Download FSCrawler 2.7:
wget https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler-es7/2.7/fscrawler-es7-2.7.zip
Start FS crawler with:
bin/fscrawler job_name
FS crawler will read a local file (default to ~/.fscrawler/{job_name}/_settings.json
).
If the file does not exist, FS crawler will propose to create your first job.
$ bin/fscrawler job_name
18:28:58,174 WARN [f.p.e.c.f.FsCrawler] job [job_name] does not exist
18:28:58,177 INFO [f.p.e.c.f.FsCrawler] Do you want to create it (Y/N)?
y
18:29:05,711 INFO [f.p.e.c.f.FsCrawler] Settings have been created in [~/.fscrawler/job_name/_settings.json]. Please review and edit before relaunch
Create a directory named /tmp/es
or c:\tmp\es
, add some files you want to index in it and start again:
$ bin/fscrawler job_name
18:30:34,330 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
18:30:34,332 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
18:30:34,682 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started for [job_name] for [/tmp/es] every [15m]
More details in the documentation.
New features
- #991: Add Workplace Search connector.
- #1203: Add FTP crawler. By helsonxiao.
-
#1211: Add
file.content_type
field on folders. -
#1210: Add
file.filename
field on folders. - #1179: Automatically create Custom Sources.
- #1037: Split console logs and actual logs and add a banner :).
- #1036: Support ssl verification configurable. By TommyLike.
- #1035: Log index errors in documents.log.
- #1031: Add an external Log4J2 configuration file.
-
#907: Add
path_prefix
option. - #820: Generate FSCrawler docker images. By toto1310.
- #776: Report HEAP size at startup.
- #752: Add option to ignore symlinks. By budachst.
- #715: Allow custom index name in the REST API. By kikkauz.
- #698: Add Cross-Origin Resource Sharing (CORS) headers to RestServer. By isaac-ipl.
- #692: Allow running OCR but not on PDF files.
- #673: Add support for YAML configuration.
- #663: Add Patterns table to includes and excludes. By wrathagom.
Fixed Bugs
- #1224: Fix NPE in Console when running with Docker.
- #1217: Check if date is null when formatting it to RFC3339.
- #1204: Split build and deploy phases for Docker images.
- #1201: 2.7 - Docker image broken. By agrantdeakin.
- #1194: Elasticsearch node settings should not be null by default.
- #1193: Corrupt PDF can lead to a StackOverflow.
- #1137: Ignore errors when parsing a 0 byte file.
- #1085: fscrawler.bat added a CD to move to the appropriate directory. By CircuitGuy.
- #1084: InputStream must have > 0 bytes. By yuanzhian.
- #1066: Start fscrawler instead of internal services.
- #1041: Fixed an issue that caused an error when running in a windows environment. By muraken720.
- #1006: Running fscrawler with no argument now lists existing jobs. By janhoy.
- #1005: Fix ENTRYPOINT in Dockerfile to allow variable substitution. By Maijin.
- #994: Using cloud id gives "invalid IPv6 Address". By tdaroly.
- #973: Fix SSH crawling from Windows machine.
- #899: FSCrawler can't index .doc or .docx elements. By LaaKii.
- #895: java.lang.NoSuchMethodError: parsing some Word files. By mwaltersbmc.
- #860: Bug Syntax error in fscrawler file, to init fscrawler. By CarlosRCDev.
- #847: sun.jnu.encoding=UTF-8 added in .bat and .sh both. By shahariaazam.
- #834: FS Crawler freezes when crawling a 0 byte TXT file. By dansfelix.
- #819: Fix Percentage computation.
- #760: Allow passing test parameters to Maven CLI.
- #714: fix release-drafter. By jetersen.
- #701: Change log level and display logs only if filters on content.
- #691: OCR without pdf_ocr. By Newmski.
- #686: Wait for healthy index when creating the index.
- #681: SSH dirs should be seen as dirs and not files.
- #680: trying to index remote files with ssh - files seen as folder. By sblanc0054.
- #660: Fix authentication when sending announcement email.
Main changes
- #1213: Switch back to Java 11.
- #1049: Update Dockerfile to use JDK14. By mario-89.
- #1212: Let's use JsonPath.
- #1207: Generate only 2 docker images.
- #1206: Detect when fscrawler runs in foreground and adapt logs.
- #1205: Add logs to the console when running a Docker instance.
- #1172: Move CI from Travis to GitHub actions.
- #872: Add more information to the _simulate API.
- #700: Add dependency convergence checks.
- #695: Exclude the PDFParser from the DefaultParser.
- #694: Display full names when catching parsing errors.
-
#693: Move
fs.pdf_ocr
setting tofs.ocr.pdf_strategy
. - #675: Warn in case of Tika error.
- #1219: Update to Elasticsearch 7.14.0 and 6.8.18.
- #1180: Bump tika.version from 1.26 to 1.27.
Removed
- #978: files lost. By bluebell1990.
Have fun!
-FSCrawler team