Storing Static data in index pattern and reusing it with live data

Elastic Stack Kibana
1

Hi All,

We are migrating Auditing in our project from Oracle to ELK. For this , while creating a report/Visualization , we need to add a new column instance , value of which would be like(S1,S2,S3,S4) based on two parameters consumer and circle. Earlier in oracle we had created a table with around 1000 entries with three columns Consumer , Circle and Instance.

We were fetching instance values from this table on the basis of Circle and Consumer at the time of generating report, As consumer and circle are available in live traffic

Can anyone help me to develop similar logic in ELK

2

Hello @rohitarora275

If I understand correctly - you want to be able to dynamically load data from another service when creating a report, correct?

I think the best way to resolve this would be to import the necessary data into Elasticsearch before running your report.

--Matt

3

Hi Matt,

I need to run report from ELK daily ; say last 24 hours. In the way you suggested , data once loaded will not be visible in future.

Currently I am showing 5 6(consumer name and circle are already there) columns in my table, I just need to add one more column i.e instance. If I start writing conditions to populate instance value then it would be more than 1000 condition like ( if consumer = Salesforce and circle = India then set instance = S1). Currently I am implementing this with static lookup. Just trying to figure out better way

4

In the way you suggested , data once loaded will not be visible in future.

Could the data be loaded into Elasticsearch as its updated?

5

Yes we can upload data into elasticsearch, my only concern here is suppose I stored data in elasticsearch today and I want to generate a report of last 24 hours next week, then the data which I uploaded today will not be visible that time. And I will not be able to display Instance column in my report.

6

@rohitarora275

You could always write a script to update documents before running the report and then trigger report generation - https://www.elastic.co/guide/en/kibana/current/reporting-integration.html Such a script would be written and executed as its own process outside of Kibana.

Thanks,
Matt

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.