G'Day all,
We have a number of web crawlers that index these forums to provide broader search functionality for people across the web. From time to time we put limits around some of these crawlers because of their behaviour.
From Monday the 4th of Febuary, 2019, we'll be blocking the Go-http-client/1.1 entirely, as it's currently crawling about 250% more than the next highest crawler and it's been this way for a few months.
We don't usually notify anyone about this sort of admin change, but we're writing this post because the agent is a generic one from the golang project, which means there's likely a number of people that have written crawlers of their own, and we wanted them to be aware when things break.
We are sorry we need to take this rather hard action, but in this instance we can't differentiate between good crawlers and bad ones. And ultimately, each crawling action has a cost associated with it in maintaining and providing this platform to our wider community.
Thanks,
The Elastic Community Team