Allow me to explain my predicament and hopefully you all will be able to enlighten me
I'm completely new to the Elastic Stack and over the last couple of days/weeks have been investigating the possibilities of what I might be able to do with (what so far seems) this superb product stack.
In my day to day job I tend to work a lot with a commercial ITSM/Process Workflow tool which of course generates a lot of data and is very flexible but does not have any really great reporting/dashboarding/visualization options.
The main reporting option is currently Crystal Reports which in itself is fine but can get pretty complex and cumbersome to maintain. Also for business users designing their own reports is pretty much impossible.
For a dashboarding/visualization option the tool relies on some build in functionality that relies on scripting/coding to visualize the KPI data in some nice graphs.
Hence my search for something more 21th century and this is where my eye fell on Kibana!
With Kibana comes of course ElasticSearch as a source and Logstash to make some sense of all the data.
Now while this explains why I have a need what is my issue? (I seem to have this all figured out don't I ).
The main problem with the tool is getting the data out of it in to logstash.
- There is no direct SQL access to the backend database (we can export periodically a subset in our own format so that's an alternative)
- The only available real-time entry point is a SOAP based webservice.
- Another (but I don't like this that much) is a period dump to a specific file format (xml, csv,xls,..) but this might be a last possible fallback option through perhaps filebeat?
I did find the http_poller plugin (http_poller documentation) but it's unclear to me if this can work with SOAP (as I need to pass an XML request and that doesn't seem to be one of the possible parameters)
With this in mind, does anybody have some suggestions on what I should set up, and especially what plugins (or beats) I should use to either read from the (near-time) reporting database or what would be even better, a way to periodically perform a SOAP request to the tool endpoint and then parse the xml response in to ElasticSearch.
Hopefully some of you more experienced minds can help me out or at least send me towards some more documentation/videos to review that point me in the right direction.
Thanks already in advance for reading this thrilling story and I hope to become a member of the ElasticStack userbase!