Now I'm afraid switching to EL 5.x requires some further understanding/configuration of those 'schemes'.
You can still treat it as 'automagic'. But if things go wrong you will have to understand how mappings work and might want to look further into templates. This is true as well for 1.x as it is for 5.x and upcoming 6.x. If it did work ok for you so far, I don't see why it should break with 5.x. Only problem I can think of might be the removal of _type in 6.x. In case you make use of _type, consider introducing a type field.
By reading you, I understand that by porting the filter/logic-analysis functionality of my logstash filters, to those "beat modules", I could have beats sending directly to EL without an intermediate logstash, but, in that case, EL should be accordingly configured with some of those 'templates' ... is this correct?
Yeah, with turning these into modules you can have filebeat directly ship to ES and ES would to the processing/parsing. When implementing modules, we also define the fields a module generates in a file called fields.yml. This fields.yml is also used to generate docs + generate index-pattern in kibana (e.g. this field is in bytes or a percentage), as kibana applies some additional mapping on an index via index-patterns. Once fields.yml is defined filebeat will automatically pick up the definitions and install templates and such.
'Unfortunately Filebeat modules do not support Logstash right now.' ... this scares me a little bit... can't filebeat send to logstash? I assume it can... or this is related only to just beats filtering/analisys capability to work together with logstash?
In the future we want modules to work across all products. But apparently it's not that easy, also due to logstash and ingest node configurations are somewhat different. With 6.x one can use filebeat modules, but needs to use filebeat setup in order to install the template and kibana index pattern + one will use logstash merely as a proxy forwarding events to the ingest pipeline as defined by filebeat.
One final cuestion arise by reading your reply... what about geoip .... is filebeats capable (thorugh a 'module') to perform geoip data-enrichment? or still logstash is needed for that?
filebeat does no processing of events. It's basically a shipper, focusing only on the task of forwarding logs to downstream systems. Geoip can be either done in logstash or Elasticsearch Ingest node via the geoip processor (requires geoip plugin to be installed, as it's very big and not shipped with ES by default).
Given you have all the processing already in logstash, developing filebeat modules in addition will be quite some overkill for you. As a first step I'd recommend to just have filebeat point to redis such that you don't have to change downstream event processing. Given logstash introduced persistent queuing (receive queue writes events to disk before additional processing), you might also consider to remove redis from your architecture (do some testing before). If you are still interested in ES Ingest Node and Filebeat modules, we can give you some more tips and pointers (feel free to open another discussion), but for now I'd say keep it simple and do/try only one change after another.
