What is logstash, Kibana in simple term?

Elastic Stack Logstash
1

I am learning to use logstash to connect ES with MySQL, JSON file. I am a bit confused. what Logstash really is. some sort of connector or something else.

PS: I know its very immature question, but I want to learn the depth of ELK.

2

Follow this to learn logstash and do some examples, after that you will let to know about kibana,

https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html

3

thanks, but i am working on projects so i have already gone through those documentations. i am asking the simple terms. like with some real life example.

4

I don't know, if this helps, but ... whatever... Let's take some really stupid example: I want to do statistics about squirrels in my neighborhood. Every squirrel has a name and we know what they look like. Each neighbor makes a log entry whenever he sees a squirrel eating a nut.

ElasticSearch is a document database that structures data in so called indices. It is able to save pieces (shards) of those indices redundantly on multiple servers and gives you great search functionalities. so you can access huge amounts of data very quickly.

Here we might have finished events that look like this:

{
  "_index": "squirrels-2018",
  "_id": "zr7zejfhs7fzfud",
  "_version": 1,
  "_source": {
    "squirrel": "Bethany",
    "neighbor": "A",
    "@timestamp": "2018-10-26T15:22:35.613Z",
    "meal": "hazelnut",
  }
}

Logstash is the data collector and transformator. It's able to accept data from many different sources (files, databases, transport protocols, ...) with its input plugins. After using one of those input plugins all the data is stored in an Event object that can be manipulated with filters (add data, remove data, load additional data from other sources). When the data has the desired format, it can be distributed to many different outputs.

If neighbor A provides a MySQL database with the columns 'squirrel', 'time' and 'ate', but neighbor B likes to write CSVs with the columns 'name', 'nut' and 'when', we can use Logstash to accept both inputs. Then we rename the fields and parse the different datetime formats those neighbors might be using. If one of them likes to call Bethany 'Beth' we can change the data here to make it consistent. Eventually we send the result to ElasticSearch (and maybe other outputs as well).

Kibana is a visualization tool. It allows you to get an overview over your index structures and server status and create diagrams for your ElasticSearch data

Here we can do funny diagrams like 'Squirrel Sightings Per Minute' or 'Fattest Squirrel (based on nut intake)'

2 Likes
5

@Jenni you are an angel. now things are crystal. this stupid/awesome example I was actually looking for and you made it so clear. thanks a ton.. :slight_smile: and i am sure this example will help other beginners like me.

6

Last request : can u mention the role of filebeat in ur example ?

7

We'd use filebeat for neighbor B. He doesn't want to give us access to his server or send us an e-mail with his current list for this project every hour. So we use filebeat instead to get every update automatically with a minimal delay. He installs filebeat on his server and tells the software that his notes are located in "/var/log/The S-Files/*" and should be sent to our server on port 5044 (for example). Now all filebeat does all day long is watch the files there and send us every new line it can find. Our beats input listens to that port, happily waiting for new squirrel events :slight_smile:

1 Like
8

thanks @Jenni ..God bless

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.