Two quick rants about logstash 5.x.
[red herring]
I was able to fix my issues with logstash not being able to find the config files, you need to run it under the user 'logstash'. This issue that was closed could have resulted in a code change to at least give a message or warning that makes sense - "This error may be because you're not running logstash as the logstash user or as a service"
[red herring]
More very relevant complaints about the logstash 5.x documentation: Logstash 5 not running
I've modified the config files as suggested.
[red herring]
Now its complaining that it can't find a pipeline config even though I'm specifying it with a -f option... wow this is getting a tad ridiculous. I'll log a bug on the project page. For now I fixed it by putting my config in /etc/logstash/conf.d
Here is the warning I'm receiving:
{"type"=>"mapper_parsing_exception", "reason"=>"Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
At least now I'm getting some feedback from ES, although the ruby output plugin doesn't show the mongo output when I'm running logstash as the logstash user/as a service.
Thanks for your guidance! Hopefully logstash 5.x updates will add more relevant error messages and make it a little easier to use, with a consistent behavior across commandline & config options.