Increased log retention for set of logs within an index

Elastic Stack Elasticsearch
1

Hi, is it possible to store logs within an index a greater period of time than other logs in the index?

Scenario:

  • Fleuntd collects container logs and stores them in an 'application' index in Elasticsearch
  • The application logs have a retention of 7 days
  • The test team using JMeter want to store their JMeter logs in Elasticsearch so they can create a comparative view of their latencies
  • Because of the quanity of logs that will be generated by containers in our clusters, we don't want all application logs to have a long retention period, however, we are happy for JMeter logs to be kept for a longer period.

Can we store these JMeter logs with greater retention within the Application index, or is there an alternative solution?

Many thanks

2

The retention in Elasticsearch is managed using Index Lifecycle Policies, those policies works with entire indices, not documents within an index.

If you want a longer retention for some documents you should store those documents on a different index, something like application-jmeter for example.

But how you would do this depends on how you are indexing your data since you will need to be able to filter for jmeter documents.

1 Like
3

Thanks for the reply Leandro - appreciate the help.

We are deploying to OpenShift and using OpenShift Logging (EFK stack). OpenShift Logging aggregates 3 types of logs: application, infrastructure and audit.

Are you talking about filtering within Elasticsearch or Kibana?

4

No, you need to be able to identify which logs are from jmeter and send those logs to a different index.

For example, your normal logs you will send to the application index and the logs from jmeter will be sent to the application-jmeter index.

I never used OpenShift Logging and don't know how this works, but you need to be able to send logs to different indices based on some conditionals that identify the log as being generated by jmeter requests or not.

2 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.