# Exporting Indexed Data

**URL:** <https://discuss.elastic.co/t/exporting-indexed-data/196969>\
**Category:** Elasticsearch\
**Created:** [August 27, 2019, 3:25pm UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969 "2019-08-27T15:25:15Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [August 27, 2019, 3:25pm UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969/1 "2019-08-27T15:25:15Z")

</div>

I am trying to export approximately 100 million indexed events to txt files but having a hard time finding out how to do so. Since I can't get the Elasticsearch input filter to work in Logstash (secured cluster, open ticket on github and in logstash forums), I am using PowerShell to connect to my cluster and pull the data. I've figured out the query I need to run in order to pull just what I want, but I am stuck at only being able to pull 10,000 events.

How do I get all of what I want OR is there a better way to accomplish this?

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [August 27, 2019, 8:54pm UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969/2 "2019-08-27T20:54:46Z")

</div>

I figured out how to do this with the scroll API, however...at 100 million records, it's gonna take almost 4 straight days to pull it all out. I tried using the slice parameter in my query to break it up and run 16 exports in parallel, but it throws an error saying it's not a legitimate parameter. Below is the PowerShell bits I am using...any ideas on how to incorporate slice or what I can do to speed this up?

If anybody ends up using this for their own setup, you'll need to modify a few lines, especially if you are not using SSL on the API.

```auto
#Force PowerShell to use TLS 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12

#Get username/password for kibana user
$cred = Get-Credential

#Specify index to query - wildcard ending implied
$is = Read-Host "Enter index to search"

#Set base elasticsearch URL
$esbase = "https://FQDN:9200"

#Set headers
$Options = @{
  headers = @{
    'kbn-version' = "7.2.0"
  }
  ContentType = "application/json"
  Method = "POST"
}
#This can be used as an example, it is querying a date field whose values are in epoch seconds.
$query = '{ "sort": ["_doc"], "query" : { "range" : { "edge.timestamp_start": { "gte": "1559329200", "lte": "1561939200" } } } }'

#Initial query specifying parameters and index to search
$isr = Invoke-RestMethod @Options -Uri "$esbase/$is*/_search?scroll=10m&size=10000" -Body $query -Credential $cred -Verbose

#Start loop to process the initial 10k results and subsequent results.
do {

  #Grab the current time to be used in the filename
  $date = Get-Date -Format MMddyyyy_hhmmss

  #Start loop to dump results into a text file
  foreach ($hit in $isr.hits.hits) {

    #Narrow file output data to only the raw message field. In my case, it's already JSON formatted.
    $out = $hit._source.message

    #Output message field to a text field, appending the existing file so as not to create 10k tiny files.
    $out | Out-File C:\Temp\June$Date.txt -Encoding utf8 -Append
  }

 #Compress the resulting txt file. In my use case, each txt file was 16MB, compressed was 1.5MB
  Compress-Archive -Path C:\Temp\June$Date.txt -Update -DestinationPath C:\Temp\June$Date.zip -CompressionLevel Fastest

  #Remove original txt file after compression is complete.
  Remove-Item D:\Temp\CloudFlare\June$Date.txt -Confirm:$false

  #Grab the scroll id from the previous successful query to feed into the next query.
  $sid = $isr._scroll_id

  #Reset query timeout to 5 minutes and insert scroll id
  $query = ' {"scroll" : "5m", "scroll_id" : "'+$sid+'" }'
  
  #Gather the next 10k results
  $isr = Invoke-RestMethod @Options -Uri "$esbase/_search/scroll" -Body $query -Credential $cred -Verbose

#Continue loop until the last query returns 0 results.
} until ($isr.hits.hits.count -eq 0)

```

---

<div class="post-metadata">

**Author:** ![sspilleman](https://avatars.discourse-cdn.com/v4/letter/s/9dc877/32.png) [@sspilleman](https://discuss.elastic.co/u/sspilleman)\
**Post date:** [August 28, 2019, 5:36am UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969/3 "2019-08-28T05:36:27Z")

</div>

You could try elasticdump with the bulk option, I use it quite often: [https://www.npmjs.com/package/elasticdump](https://www.npmjs.com/package/elasticdump)

---

<div class="post-metadata">

**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\
**Post date:** [August 28, 2019, 10:17am UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969/4 "2019-08-28T10:17:37Z")

</div>

Hi  
The [latest document of Logstash](https://www.elastic.co/guide/en/logstash/current/ls-security.html) says " The Logstash Elasticsearch plugins ([output](http://www.elastic.co/guide/en/logstash/7.3/plugins-outputs-elasticsearch.html), [input](http://www.elastic.co/guide/en/logstash/7.3/plugins-inputs-elasticsearch.html), [filter](http://www.elastic.co/guide/en/logstash/7.3/plugins-filters-elasticsearch.html) and [monitoring](http://www.elastic.co/guide/en/logstash/7.3/monitoring-logstash.html)) support authentication and encryption over HTTP."  
Couldn't it support your secured Elasticsearch?

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [August 28, 2019, 2:44pm UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969/5 "2019-08-28T14:44:03Z")

</div>

> [@wangqinghuan](#):
>
> Hi  
> The [latest document of Logstash](https://www.elastic.co/guide/en/logstash/current/ls-security.html) says " The Logstash Elasticsearch plugins ([output](http://www.elastic.co/guide/en/logstash/7.3/plugins-outputs-elasticsearch.html), [input](http://www.elastic.co/guide/en/logstash/7.3/plugins-inputs-elasticsearch.html), [filter](http://www.elastic.co/guide/en/logstash/7.3/plugins-filters-elasticsearch.html) and [monitoring](http://www.elastic.co/guide/en/logstash/7.3/monitoring-logstash.html)) support authentication and encryption over HTTP."  
> Couldn't it support your secured Elasticsearch?

You would think it would, but it throws an error that I've posted a topic about in the [Logstash forums](https://discuss.elastic.co/t/exception-faraday-sslerror-elasticsearch-input/196558) as well as on the [plugin's github page](https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/108), neither has received any attention.

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [August 28, 2019, 2:49pm UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969/6 "2019-08-28T14:49:57Z")

</div>

> [@sspilleman](#):
>
> You could try elasticdump with the bulk option, I use it quite often: [elasticdump - npm](https://www.npmjs.com/package/elasticdump)

I saw that, unfortunately, I run on a windows platform and it looks like ElasticDump is designed for Linux platforms.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 25, 2019, 2:50pm UTC](https://discuss.elastic.co/t/exporting-indexed-data/196969/7 "2019-09-25T14:50:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.