# How to generate the same sampled dataset from spark?

**URL:** <https://discuss.elastic.co/t/how-to-generate-the-same-sampled-dataset-from-spark/82005>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [April 11, 2017, 3:10pm UTC](https://discuss.elastic.co/t/how-to-generate-the-same-sampled-dataset-from-spark/82005 "2017-04-11T15:10:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![geantbrun](https://avatars.discourse-cdn.com/v4/letter/g/ecb155/32.png) [@geantbrun](https://discuss.elastic.co/u/geantbrun)\
**Post date:** [April 11, 2017, 3:10pm UTC](https://discuss.elastic.co/t/how-to-generate-the-same-sampled-dataset-from-spark/82005/1 "2017-04-11T15:10:40Z")

</div>

Hi,  
In Spark, I want to sample a large dataset from my es index. I tried the following command:

```
df = spark.read.format("org.elasticsearch.spark.sql") \
            .option("query", myquery) \
            .option("pushdown", "true") \
            .load(spark.conf.get("spark.es.resource")) \
            .limit(mysize) 

```

where myquery is:

```
{
  "query": {
    "filtered": {
      "query": {
        "function_score": {
          "functions": [
            {
              "random_score": {
                "seed": 1
              }
            }
          ],
          "score_mode": "sum"
        }
      }
    }
  }
}

```

amd mysize~10K. The problem is that I'm not getting the same result each time I execute the command (with the same seed). Why? What is the best way to sample the data without having first to match all the data and making a sample command on this huge dataframe?

Thank you for your help

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 9, 2017, 3:18pm UTC](https://discuss.elastic.co/t/how-to-generate-the-same-sampled-dataset-from-spark/82005/2 "2017-05-09T15:18:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.