Inaccuracy and noises in elastic search rollup data

Elastic Stack Elasticsearch
1

I am new to the ELK stack and Elasticsearch. I am currently storing nginx logs in Elasticsearch and using Kibana to visualize the total bytes over time. The purpose of this is to monitor traffic. To reduce storage requirements, I am trying to roll up the data into 1-minute intervals. However, the resulting visualization has a lot of noise. If I increase the time interval to 20 or 30 minutes, the visualization improves significantly. However, I need to store the data in 1-minute intervals. Is this a common issue? Can it be resolved? My Elasticsearch stack version is 8.2.2.

I have rolled up data via kibana itself, and the request for that is like this:

PUT _rollup/job/test
{
  "id": "test",
  "index_pattern": "nginx-optimized-2023.03.04",
  "rollup_index": "test",
  "cron": "0 * * * * ?",
  "page_size": 1000,
  "groups": {
    "date_histogram": {
      "interval": "1m",
      "field": "@timestamp"
    }
  },
  "metrics": [
    {
      "field": "bytes",
      "metrics": [
        "sum"
      ]
    }
  ]
}
2

Welcome to our community! :smiley:

It'd be good if you could share what you mean by this, screenshots etc.

3

thanks alot.

I have an index for 1 day data of nginx logs that has the outgoing bytes on each request.
the index has 230127924 documents.
And the rollup has 1440 documents, with 1 minute intervals.

Screenshot from 2023-03-20 01-30-41
Screenshot from 2023-03-20 01-30-41922×842 52.6 KB

the top visualization belongs to the data before rolling up, and the bottom after rolling up.