Decay query Running very slow at a very small requestpersecond

[jalaj (Jalaj Kumar)]

Hi,

I have an index named discover in which I have have only 2 fields timestamp (ts) and another meta field(oid).

PUT discover
{
  "settings": {
    "number_of_shards": 6,
    "number_of_replicas": 2
  }, 
  "mappings": {
    "content": {
      "properties": {
        "ts": {
          "type": "date" 
        },
        "oid":    {"type": "keyword","index": false}
      }
    }
  }
}

I have ingested around 10 million documents into this index. I have 6 machines (8 core 60 GB RAM ). ES process have 30 GB Xmx.

Now I ran a simple query to fetch documents decayed by time.My requirement is I want documents with recent documents first. Initially I am not doing any query so its a matchall but in future I might want to rank with some metadata and also decay by time.

My current search query is like this.

GET discover/content/_search?preference=_shards:1
{
  "_source": "{_id}", 
      "query": {
        "function_score": {
          "functions": [
            {
              "gauss": {
                "ts": {
                  "origin":1513887762 ,
                  "scale": "1m",
                  "decay": 0.5
                }
              }
            }
          ]
        }
      }
}

I am okay with getting data only from a single shard at a time.

I run the above query qith just 50-60 rps and load average of the cluster reaches 7-8 and mean_99 latencies are in ~900 ms which is very slow.

Am I doing something wrong or is there a better way to achieve what I am trying ?