Suggestions for web crawler


I am using ES 5.6.3 with xpack and need suggestions for web crawler. Can I use ES site search for crawling and ingest in to my ES 5.6.3 index ? Saw suggestions on Apache storm. Does it have features like include/exclude uri’s ? Can it handle bigger ones ? Is it easy to customise?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.