Recently I have set up ELK to gather syslogs and NETFlow from my router and it's all working fine. It's just so basic and when I try to add visualizations and won't find anything. I don't know what I should be looking at.
Some people have the most advanced and good looking dashboards, is it because my data is very simple? Do I need to configure the devices to send in more or something?
How would I, for example, be able to visualize data like this http://i.imgur.com/5HvoSmG.png? According to reddit /r/networking user sysvival this was done with pure syslog logging.
I'm mainly looking to visualize netflow traffic and commands on syslogs in general.
In order to create dashboards of that nature, you will need to find the data within the index (syslog) for which you can create a visualization on. If you look at the dashbard, you may find it helps to look at the available fields . Clicking on an available field will display a short list of unique fields allowing you to easily filter on that data.
A visualization displaying the number of syslogs broken down by severity might be a good place to start and get yourself introduced to Kibana. Start with a new Vertical Bar Chart from the Visualize tab. Start with a new search, selecting the index containing your syslog data. For the Y-axis, we will choose Count. Under buckets, choose X-Axis, and Date Histogram. Click "Add sub-bucket", choose Split Bars, and select Terms. The field will be syslog_severity. Click the green play button on the top of this left pane. You can save this visualization and add it to your dashboard. Play around with the visualizations and see what you come up with.
Well that is an open ended question as so much can be done with just the syslog data you have.
Lets try to create a Date Histogram visualization Select Vertical Bar Chart from the Create New Visualization screen and use the following:
From A new Search. (pick you index)
Under Buckets choose the xAxis arrow and choose
Aggregration = Date Histogram
Field = @timestamp
Interval = Auto or what ever time you want
Add a sub bucket Split Bars wit the following
Sub Aggregation = filters.
In filter 1 field put a value you would like to find from the raw data in your elasticsearch index. (look in discover tab for raw data ideas)
Same in filter 2, 3 ,4 etc for each term you are looking for.
Hit the green run arrow at the top and you should see some graphs.
If you have graphs save the visualization and add it to a dashboard.
After some data flows you will be able to see anomaly's or deviation from the normal as you will see a base line of normal traffic and if someone does try to break in or something generates a different traffic pattern it will stand out as a spike.
This is a good place to start they give a couple other examples.
Below is a screen shot of a similar histogram from My Kibana 4.2 that shows a BIG deviation.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
