# Is there a quicker way to import data to Elastic?

**URL:** <https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275>\
**Category:** Elasticsearch\
**Created:** [March 8, 2023, 1:33pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275 "2023-03-08T13:33:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![eeijlar](https://avatars.discourse-cdn.com/v4/letter/e/96bed5/32.png) [@eeijlar](https://discuss.elastic.co/u/eeijlar)\
**Post date:** [March 8, 2023, 1:33pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275/1 "2023-03-08T13:33:36Z")

</div>

I have exported elastic indices using logstash with the following logstash configuration:

```auto
    - pipeline.id: export-process
      pipeline.workers: 4
      config.string: |
        input {
          elasticsearch {
            hosts => "http://elastic:80/elasticsearch/"
            user => "elastic"
            password => ""
            ssl => "false"
            index => "metricbeat-*"
            docinfo => true
            query => '{
                "query": {
                  "bool": {
                    "filter": {
                      "range": {
                          "@timestamp": {
                          "gte": "now-35m",
                          "lte": "now",
                          "format": "strict_date_optional_time||epoch_millis"
                          }
                      }
                    }
                  }
              }
            }'
          }
        }
        output {
          file {
            gzip => "true"
            path => "/usr/share/logstash/export/export_%{[@metadata][_index]}.json.gz"
          }
        }

```

Now I am trying to import it back into another instance. I have unzipped the gz json file, and I am going over each line in the document and doing:

`curl -s -XPOST http://1.2.3.4:9000/metricbeat/_doc/ -H "Content-Type: application/json" -d "$1"`

where $1 is a line item from the json file. This method is very slow. I started the import of one index which is 1.7Gb and it is still running after 90 minutes. Is there a better way of doing this?

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [March 8, 2023, 2:49pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275/2 "2023-03-08T14:49:25Z")

</div>

Hi John,

Are you able to use the [\_bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#bulk-curl) instead?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 8, 2023, 2:53pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275/3 "2023-03-08T14:53:13Z")

</div>

> [@eeijlar](#):
>
> This method is very slow. I started the import of one index which is 1.7Gb and it is still running after 90 minutes. Is there a better way of doing this?

Why not use Logstash with a `file` input and an `elasticsearch` input? Or even Filebeat?

Also, if you have communication between your instances you could try a remote reindex, or maybe create a snapshot on a cloud service and restore from the snapshot.

---

<div class="post-metadata">

**Author:** ![eeijlar](https://avatars.discourse-cdn.com/v4/letter/e/96bed5/32.png) [@eeijlar](https://discuss.elastic.co/u/eeijlar)\
**Post date:** [March 8, 2023, 3:26pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275/4 "2023-03-08T15:26:38Z")

</div>

Hi @carly.richmond When I try the bulk import I get this in the response:

```auto
< Warning: 299 Elasticsearch-8.3.3-801fed82df74dbe537f89b71b098ccaff88d2c56 "Unsupported action: [stream]. Supported values are [create], [delete], [index], and [update]. Unsupported actions are currently accepted but will be rejected in a future version."
< content-type: application/json;charset=utf-8
< content-length: 329
* HTTP error before end of send, stop sending
<
* Closing connection 0
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_STRING]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_STRING]"},"status":400}

```

So it looks like the format of the output is not as expected. I also had to update the `http.max_content_length` as 100mb was too small also.  
Thanks for the link!

---

<div class="post-metadata">

**Author:** ![eeijlar](https://avatars.discourse-cdn.com/v4/letter/e/96bed5/32.png) [@eeijlar](https://discuss.elastic.co/u/eeijlar)\
**Post date:** [March 8, 2023, 3:31pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275/5 "2023-03-08T15:31:01Z")

</div>

The instances I am exporting from are ephemeral, hence the reason for trying to harvest the data from them to be imported at a later date. I take it you mean `elasticsearch` output rather than input. That might be an option. I will try it out.

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [March 8, 2023, 3:49pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275/6 "2023-03-08T15:49:54Z")

</div>

Yes, I think @leandrojmp's great suggestion is using file input and Elasticsearch _output_ plugins as you've clarified. I would recommend trying his approach instead of bulk given the error above.

Let us know how you get on!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 5, 2023, 3:50pm UTC](https://discuss.elastic.co/t/is-there-a-quicker-way-to-import-data-to-elastic/327275/7 "2023-04-05T15:50:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.