Brian,
I agree completely with your expectations. If I am to replace Splunk
(ridiculously overpriced in my opinion) or Logrhythm, I need to be able to :
- Generate alerts that need immediate reaction.
- Generate reports
- Compliance related reporting
- Perform aggregations on the fly (e.g. social-networking traffic by
user by hour of day/week etc.)
- Generate Trend reports (Logon failures by Source IP address by
destination address by hour of day etc.)
- Provide an analyst a console with which he can quickly drill down to
events of interest.
Of the three broad categories, Kibana does an admirable job of the third.
Alerts can be generated by Logstash or nxlog that feeds ES though this is
kludgy and hard to maintain at best. (Graylog2 streams?)
I was looking to generate reports by scripted process. Basically create a
scripting framework and vary the query as necessary, but this is easier
said than done at the moment.
With respect to Kibana, it is a great product but I believe it took a step
back in functionality from Version 2 to version 3. The pace of development
appears to now be the same as OSSEC WEBUI and I do not see a roadmap for
future releases. It may be that folding the product into the Elasticsearch
umbrella has moved the company in a different direction (I smell a
commercial product coming soon).
I took a look at Graylog2 and I must say it has moved forward by leaps and
bounds over the last year and does have the ability to generate alerts. Not
sure how easy it would be to create custom reports as I have not tried.
What holds me back is that it only supports ES version 0.90.9. Considering
that version does not support aggregations (only facets), I am reluctant to
move my eggs into an old basket considering ES is pushing Ver. 1.3 and
upwards at the moment.
Until the smoke clears up, I believe we will have to rely on building
scripting skills. I am personally struggling with parsing JSON output with
nested fields as effectively into CSV so I can feed R etc. programatically
Regards
Ash Kumar
On Thursday, September 25, 2014 11:57:44 AM UTC-4, Brian wrote:
Ash,
JSON is a natural for Kibana's Javascript to read and therefore emit as
CSV. So what I really was asking is Kibana going to become a serious
conteder and allow user-written commands to be inserted into the pipeline
between data query/response and charting. After my few weeks with R, I have
gotten it to far exceed GNUPlot for plotting (even with the base plotting
functions; I haven't yet dived into ggplot2 package), and to also far
exceeds Kibana. For example, setting up a custom dashboard is tedious, and
it's not easily customizable.
Now, I am not suggesting that the ELK stack turn into Splunk directly. But
since it wants to become a serious contender, I an am strongly recommending
that the ELK team take the next step and allow a user-written command to be
run against the Kibana output and its charting. And I recommend that the
output be CSV because that's what R supports so naturally. And with R, I
can build out custom analysis scripts that are flexible (and not hard-coded
like Kibana dashboards).
For example, I have an R script that gives me the most-commonly used
functions that the Splunk timechart command offers. And with all of its
customizability: Selecting the fields to use as the analysis, the by field
(for example, plotting response time by host name), the statistics (mean,
max, 95th percentile, and so on), even splitting the colors so that the
plot instantly shows the distribution of load across 10 hosts that reside
within two data centers.
This is an excellent (and free) book that shows what Splunk can do by way
of clear examples:
Exploring Splunk: Search Processing Language (SPL) Primer and Cookbook | Splunk
Again, I don't suggest that Kibana duplicate this. But I strongly suggest
that Kibana gives me a way to insert my own commands into the processing so
that I can implement the specific functions that our group requires, and
can do it without my gorpy Perl script and copy-paste command mumbo-jumbo,
and instead in a much more friendly and accessible way that even the PMs
can run from their Windows laptops without touching the command line.
And as my part of the bargain, I will use Perl, R, or whatever else is at
my disposal to create custom commands that can run on the Kibana host and
perform all of the analysis that our group needs.
Brian
On Wednesday, September 24, 2014 4:34:43 PM UTC-4, Ashit Kumar wrote:
Brian,
I like the direction you are going down and am trying to do that myself.
However, being a perl fledgling, I am still battling Dumper etc. I would
appreciate it if you could share your code to convert and ES query to CSV.
I want to use aggregations and print/report/graph results. Kibana is very
pretty and does the basics well, but I want to know who used web mail and
order it by volume of data sent by hour of day and either graph / tabulate
/ csv out the result. I just cant see how to do that with Kibana.
Thanks
Ash
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c3d710ef-d49e-472d-8b8a-52c3860a06c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.