Logstash and Kibana Visualization: A Simple Flag

Elastic Stack Logstash
#1

WARNING: extreme ELK newbie....

Good afternoon ELK friends,

I am extremely new to ELK... and I apologize in advanced if any of my questions seem elementary.

What I'm trying to do, is simply visualize in Kibana from a file called "/var/log/test". The only thing my file contains is the word "on".

I want to create a visualization, searching for the word "on" in the index I created via Logstash using the file, and any time the file in /var/log/test is changed to "off" or anything else, I want Kibana to dynamically show this.

The problem is, I can't seem to figure out how to use Kibana to search for a specific string and I'm not sure where to start. I'm not sure if it's Logstash configuration or Kibana configuration.

If anyone could help or point me to a topic that might have already discovered this... I would be extremely grateful!!

Thanks for your time!

#2

Are you getting any events into Elasticsearch? What do they look like?

#3

Thanks for getting back to quick!

Here's my logstash conf for /var/log/test file:

input {
  file {
    type => "syslog"
    path => ["/var/log/test"]
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{WORD:yes}"}
  }
}


output {
  elasticsearch {
    action  => "index"
    hosts => "localhost:9200"
    index => "test"
    workers => 1
  }
  stdout {}
}

I run "logstash -f" and it takes it no problem. Indexing it into Kibana has no issue either.

After adding it in Kibana as an index pattern... the field "yes" shows up with a unique count of "1". But if /var/log/test is modified to say "no", nothing happens. When I restart Logstash... the unique count of "yes" goes up to "2".

Essentially what I want is for that count to go down to "0" if the file ever says log with a minimal amount of restarting services. I'm not sure if I'm going about it entirely the wrong way.

Thanks again!

#4

After adding it in Kibana as an index pattern... the field "yes" shows up with a unique count of "1". But if /var/log/test is modified to say "no", nothing happens.

Exactly how are you modifying /var/log/test?

When I restart Logstash... the unique count of "yes" goes up to "2".

I'm not sure why this only happens when you restart Logstash, but I wonder if you're hitting a corner case in the file input. Does what you're trying to do now resemble what you eventually want to use Logstash for? Because the file input is made for continuously monitoring log files. Monitoring one-line files that are updated is atypical and debugging this further is not useful unless it's an actual use case.

Essentially what I want is for that count to go down to "0"

Why would you expect it to go down to zero? After your update your index contained two documents, one with yes equal to "yes" (or whatever the original contents of the file was) and one with yes equal to "no", thus two unique values.

#5

Morning,

Thanks for getting back to me again. Your responses are very helpful.

I'm modifying the file using vim.

In the long run what I'd like to use this for is checking logs for all occurrences of "yes" and show an average of the occurrences. The logs would be very large. The single word/line file was just for learning purposes.

That makes sense on the indexes. Is what I'm hoping to achieve here not possible via ELK?

#6

I'm modifying the file using vim.

Okay. I'm pretty sure vim will write the whole file to a new file and rename it into place. Given start_position => beginning this should work there might be some subtlety that I'm not getting. Anyway, since this was only a training exercise I wouldn't spend too much time on it.

In the long run what I'd like to use this for is checking logs for all occurrences of "yes" and show an average of the occurrences. The logs would be very large.

That's a standard use case that the Elastic stack will handle just fine, but when trying it out try to mimic what actually will happen, i.e. append to the file with e.g. echo foo >> filename.log instead of editing the file with vim.

#7

Hey again,

Thanks a lot for the help.

So I went back and configured my central syslog server to output to Logstash as json.

If I wanted to show in Kibana the number of occurrences of the field "programname" that equals "sshd" in visualizations, I'm a little confused as how to do that.

Is there a good guide I could follow, or could you provide some insight on that?

Regards,
Mike

#8

Just type "programname:sshd" into the query box in Kibana. Apart from what's on elastic.co I can't recommend any particular documentation.

#9