Inaccuracy and noises in elastic search rollup data

[Mehrab_Azad]

I am new to the ELK stack and Elasticsearch. I am currently storing nginx logs in Elasticsearch and using Kibana to visualize the total bytes over time. The purpose of this is to monitor traffic. To reduce storage requirements, I am trying to roll up the data into 1-minute intervals. However, the resulting visualization has a lot of noise. If I increase the time interval to 20 or 30 minutes, the visualization improves significantly. However, I need to store the data in 1-minute intervals. Is this a common issue? Can it be resolved? My Elasticsearch stack version is 8.2.2.

I have rolled up data via kibana itself, and the request for that is like this:

PUT _rollup/job/test
{
  "id": "test",
  "index_pattern": "nginx-optimized-2023.03.04",
  "rollup_index": "test",
  "cron": "0 * * * * ?",
  "page_size": 1000,
  "groups": {
    "date_histogram": {
      "interval": "1m",
      "field": "@timestamp"
    }
  },
  "metrics": [
    {
      "field": "bytes",
      "metrics": [
        "sum"
      ]
    }
  ]
}

[warkolm (Mark Walkom)]

Welcome to our community!

@Mehrab_Azad

However, the resulting visualization has a lot of noise. If I increase the time interval to 20 or 30 minutes, the visualization improves significantly

It'd be good if you could share what you mean by this, screenshots etc.

[Mehrab_Azad]

thanks alot.

I have an index for 1 day data of nginx logs that has the outgoing bytes on each request.
the index has 230127924 documents.
And the rollup has 1440 documents, with 1 minute intervals.

the top visualization belongs to the data before rolling up, and the bottom after rolling up.

[Mehrab_Azad]

Anything would be really helpful...