Thanks Badger, I will try using throttle. My understanding (please correct if I am wrong) is that I can limit the number of identical log messages per unit of time from the same host by using key => "%{host}%{message}".
I am actually using ILM based on the index size, but it doesn't prevent from appearing a lot of identical useless log messages. Also AFAIK ILM can only roll/delete the whole index, whereas I would prefer just to get rid of these specific messages.
I also asked a similar question in filebeat forum to check if filebeat could handle this situation with huge files/repeated log messages, but didn't get any response there.