Hi,
I have a requirement is save the latest data to the new index from old index。
The old index writes data regularly every day。
In fact, I just want to visualize with the latest data，But I didn't find the right way, so I can only try to save the latest data to the new index.
Then use kibana life cycle to delete automatically. Finally, the index will still be the latest data next time。
I try to use rollup jobs, but it seems to work for all data,I just want it to work for a new generation of data。
Is there any other way for me?
Hi,
you can use a transform (similar concept to rollup), we have an example of this use case in the docs.
Spoiler: The next release will ship an easier ootb capability for this use case. Release date? I can't say, but look at how often we ship a new release, the next one is not far away.
1 Like
What if I want to use the latest data for visualization?
I just want to see the latest dozens of data，
maybe I misunderstand your request. Can you define what you mean by latest?
Is it just tail -f what you are looking for? If so, you might take a look into the logs app in kibana. Alternatively it should be possible to create a table and use a search with reversed sort order by timestamp.
I was thinking you need the latest state, that means latest by an identifier which can e.g. be a hostname.
If not it seems rollup/transform is the wrong tool for your use case.
Sorry, I didn't make it clear。
My needs are like this：
The old index writes data regularly every day, triggering only once a day.
However, the number of data written each time may be different。
I want to visualize the data in this index，But I just want to see the latest data。
The amount of the latest data is not fixed。
The way I think of now is to write the latest data to another index when writing data。
This new index will be deleted regularly,
When a new index is generated next time, the data in it is still up-to-date。
Can you understand my needs?
Is my plan feasible? Or is there another way?
Does your daily data contain a timestamp?
If not, is there a reason not to add one, e.g. at ingest?
I do not see a need to write an extra index, you can limit the time range in the visualization using the time picker or a query.
If you desperately want separate indexes, you can use a date index name processor to create a new index every day. You can query that index directly if you only want the data of the last day. For all data you can query using a pattern (e.g. mydata-*).
yes,It contains a timestamp。
I can understand you in general，But if I use the filtering operation, I will have to select a new time next time I view it, right?
For example, write data at 8 a.m. every day.
If I check the latest data after 8 a.m., I can choose "today" in kibana.
If I check the latest data before 8 a.m., I can choose "last 24 hours" in kibana。
This means that I need to change the query criteria to see the latest data。
But I hope that every time I click to view the dashboard, the latest data will be displayed，
That's why I try to write the latest data to the new index. Before writing the data the next day, the new index is deleted and then created to keep the latest data.
I'll use this new index for visualization without any query restrictions。
Thanks for the explanation, now I understand your request.
However I don't see how a new index will help to solve the visualization problem in kibana. Even with a new index you will still end up with the time picker problem.
I am not a kibana expert, but as far as I know you can specify a relative time and choose to round down. It seems to me that you are looking for last 1d rounded down. The problem now: The day will start at 0:00, but you want 8:00. TSVB (not sure what you use) allows you to offset the series under Options. So I would try to round down by day and offset 8h.
The new index contains only one day's data。
After 24 hours of creation, it will be deleted and recreated。
The new index contains the latest data.
If I use this index pattern for visualization, its data should be latest.
You're absolutely right。
Since I write data in days, I am looking for a way to find the data of the latest day.
The day will start at 0:00, but I want 8:00.
So if I write data at 0:00, it seems that I can avoid such a problem, and then I can select "today" to view the dashboard.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.