Save request results in a new elastic index or a new database

Elastic Stack Elasticsearch
1

Hi, I'm quite new with elasticsearch and kibana and I wonder if there is a way to save the data used to make the graphs with kibana in a new index, or even in an outside database, and then make the data
processing of the saved data.
Because the way I understood it Kibana to make a graph aggregate data from all the logs in elasticsearch so when we activate auto-refresh every 2 minutes to have real time graph it re aggregate all the log it's in elasticsearch, so what I'm trying to accomplish is to take the last 2 minutes logs add it to my database and making the processe on this database so i won't overload my elasticsearch.
thank you for your help :slight_smile:

2

There's not at this stage sorry!

3

I won't overload my elasticsearch

Did you overload it yet?
I mean a lot of OS and ES cache is involved and it should be ok OOTB.

Are you sure you need something like this?

4

Well, not yet, but I feel that if I keep reconsidering every log in my elasticsearch each time I make an update to my graph it might happen sooner or later? but maybe I'm wrong like I said i'm new with ELK .

but to give you a more precise explanation of my probleme here is an example:
I have a bar graph that shows http requests, and group them as either a successful request or failed ones: and the thing is we have like 30 request now and in 2 minutes 5 new ones will be added, and still we reconsider all 35 logs instead of adding the 5 new in the graph

image.png1068×588 20.1 KB

thank you ;

5

I have a bar graph that shows http requests, and group them as either a successful request or failed ones: and the thing is we have like 30 request now and in 2 minutes 5 new ones will be added, and still we reconsider all 35 logs instead of adding the 5 new in the graph

Yes. But I don't think it's a problem IMO.
Specifically for time based indices which are append-only indices. Which means that segments, specifically the older ones are cached by the OS for a long time. And also with instant aggregations some cache on ES level makes that even faster. (You need to upgrade though).

In short, I'd say: "don't try to fix an issue that you don't have".

6

ok, So I worry for nothing :slight_smile: ,
by the way as for the upgrade we are currently woking on upgrading to kibana 5 :slight_smile: .

thank you for your implication and responsiveness .

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.