Inaccuracy and noises in elastic search rollup data

I am new to the ELK stack and Elasticsearch. I am currently storing nginx logs in Elasticsearch and using Kibana to visualize the total bytes over time. The purpose of this is to monitor traffic. To reduce storage requirements, I am trying to roll up the data into 1-minute intervals. However, the resulting visualization has a lot of noise. If I increase the time interval to 20 or 30 minutes, the visualization improves significantly. However, I need to store the data in 1-minute intervals. Is this a common issue? Can it be resolved? My Elasticsearch stack version is 8.2.2.

I have rolled up data via kibana itself, and the request for that is like this:

PUT _rollup/job/test
{
  "id": "test",
  "index_pattern": "nginx-optimized-2023.03.04",
  "rollup_index": "test",
  "cron": "0 * * * * ?",
  "page_size": 1000,
  "groups": {
    "date_histogram": {
      "interval": "1m",
      "field": "@timestamp"
    }
  },
  "metrics": [
    {
      "field": "bytes",
      "metrics": [
        "sum"
      ]
    }
  ]
}

Welcome to our community! :smiley:

It'd be good if you could share what you mean by this, screenshots etc.

thanks alot.

I have an index for 1 day data of nginx logs that has the outgoing bytes on each request.
the index has 230127924 documents.
And the rollup has 1440 documents, with 1 minute intervals.

the top visualization belongs to the data before rolling up, and the bottom after rolling up.

Anything would be really helpful...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.