General questions regarding ELK stack


(bowow) #1

Hi,
I have a few questions regarding ELK stack.

  1. what is the best option to pull data? Read from a log file or Database? Best option in terms of Performance/latency/data accuracy.
    

a. If Database, then any drawback like extra overload on Application database? How frequently it reads?
b. If both the cases, what happens if Kibana goes down and need to load older log files or data? Will it be done automatic, in real time?
2) Can single Kibana instance handle multiple apps? Or should i go for separate Kibana instance per app. If single, then any performance degradation and can we control user access to specific application data?


(Magnus B├Ąck) #2
  1. what is the best option to pull data? Read from a log file or Database? Best option in terms of Performance/latency/data accuracy.

There is no single best option that's preferable under all circumstances. All methods have advantages and drawbacks.

a. If Database, then any drawback like extra overload on Application database? How frequently it reads?

Reading from the database obviously comes with a cost. You can choose how often to do the polling.

b. If both the cases, what happens if Kibana goes down and need to load older log files or data? Will it be done automatic, in real time?

Kibana is just the visualization layer. If either Logstash or Elasticsearch goes down you'll probably be okay, but it depends on how things are implemented. There will always be opportunities to screw things up.

  1. Can single Kibana instance handle multiple apps?

Yes.

Or should i go for separate Kibana instance per app. If single, then any performance degradation and can we control user access to specific application data?

With the Shield product you get pretty granular access control.


(bowow) #3

Thanks so much @magnusbaeck :slight_smile:

When you say, reading from database comes with a cost, do you mean we have to pay for the service?

FYI: The database I am going to be reading from has very large sets of data.


(Mark Walkom) #4

He means that reading that data takes time; to have someone setup a process, to run the process, to ingest into ES.


(bowow) #5

Oh ok. As long as it's not slow compared to reading from a file..It will be fine.


(system) #6