# Dump JSON directly in to ElasticSearch

**URL:** <https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940>\
**Category:** Elasticsearch\
**Created:** [July 13, 2020, 10:03am UTC](https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940 "2020-07-13T10:03:19Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![saurabhbhatt](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@saurabhbhatt](https://discuss.elastic.co/u/saurabhbhatt)\
**Post date:** [July 13, 2020, 10:03am UTC](https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940/1 "2020-07-13T10:03:19Z")

</div>

Hi,

I have this requirement of dumping big JSON file with over 700K records into ES directly. However the JSON I have is formatted in PRETTY array format as below:

**STRING 1:**  
indent preformatted text by 4 spaces

```auto
[
  {
    "recordId": 1,
    "FirstName": "Julie",
    "LastName": "Klier",
    "FullName": null,
    "email": "asdasd@gmail.com",
    "name": "sd fghgfh",
    "company": null,
    "leadsource": "MOT",
    "status": "Qualified",
    "site_id__c": "155731983",
    "createdDate": "2019-08-13T20:41:34",
    "sf_lead": [],
    "sf_contact": [
      {
        "id": "0031I00000tjNCxQAM",
        "accountid": null,
        "lastname": null,
        "firstname": null,
        "name": "Julie Klier",
        "email": "sadsad@gmail.com",
        "leadsource": "MOT",
        "createddate": "2019-08-13T20:41:34Z",
        "site_id__c": "155731983",
        "asi__c": null
      }
    ],
    "sf_account": [],
    "sf_oppurtunity": [
      {
        "id": "0061I00000J4DZWQA3",
        "contactid__c": null
      }
    ],
    "sf_campaign": [
      {
        "id": "7013u000000mlLyAAI"
      }
    ],
    "asidta_customer": [],
    "asidta_ordrhist": [],
    "asidta02_customer": [],
    "asidta02_ordrhist": [],
    "asidta04_customer": [],
    "asidta04_ordrhist": [],
    "asidta05_customer": [],
    "asidta05_ordrhist": []
  }
]

```

Now I want to convert this into below format:

**STRING 2:**

```auto
      {"index":{"_index":"allreportingdata","_id":1}}
  {
    "recordId": 1,
    "FirstName": "Julie",
    "LastName": "Klier",
    "FullName": null,
    "email": "asdasd@gmail.com",
    "name": "sd fghgfh",
    "company": null,
    "leadsource": "MOT",
    "status": "Qualified",
    "site_id__c": "155731983",
    "createdDate": "2019-08-13T20:41:34",
    "sf_lead": [],
    "sf_contact": [
      {
        "id": "0031I00000tjNCxQAM",
        "accountid": null,
        "lastname": null,
        "firstname": null,
        "name": "Julie Klier",
        "email": "sadsad@gmail.com",
        "leadsource": "MOT",
        "createddate": "2019-08-13T20:41:34Z",
        "site_id__c": "155731983",
        "asi__c": null
      }
    ],
    "sf_account": [],
    "sf_oppurtunity": [
      {
        "id": "0061I00000J4DZWQA3",
        "contactid__c": null
      }
    ],
    "sf_campaign": [
      {
        "id": "7013u000000mlLyAAI"
      }
    ],
    "asidta_customer": [],
    "asidta_ordrhist": [],
    "asidta02_customer": [],
    "asidta02_ordrhist": [],
    "asidta04_customer": [],
    "asidta04_ordrhist": [],
    "asidta05_customer": [],
    "asidta05_ordrhist": []
  }

```

Can someone please help me to achieve this? Also what is this STRING 2 called ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 13, 2020, 10:31am UTC](https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940/2 "2020-07-13T10:31:40Z")

</div>

The bulk format does not support pretty printing. Each document need to be on a single line.

---

<div class="post-metadata">

**Author:** ![saurabhbhatt](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@saurabhbhatt](https://discuss.elastic.co/u/saurabhbhatt)\
**Post date:** [July 13, 2020, 10:34am UTC](https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940/3 "2020-07-13T10:34:15Z")

</div>

Ok, so is there a way to convert this pretty format into single line format using some tool ? Also what is this single line JSON called ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 13, 2020, 10:44am UTC](https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940/4 "2020-07-13T10:44:57Z")

</div>

I would recommend creating a script to reformat this, e.g. using Python. Also note that the bulk api is designed for sending multiple events per request. Each request should however generally be no larger than 5MB so your data set would result in multiple bulk requests.

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [July 13, 2020, 11:41am UTC](https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940/5 "2020-07-13T11:41:14Z")

</div>

> [@saurabhbhatt](#):
>
> Ok, so is there a way to convert this pretty format into single line format using some tool

There's a lot of different tools that you can use for this. Writing a script in Python as @Christian_Dahlqvist suggests is on way, using something like [`jq`](https://stedolan.github.io/jq/) is another e.g.

```sh
cat example.json | jq -c

```

> [@saurabhbhatt](#):
>
> Also what is this STRING 2 called ?

This format is newline-delimited JSON ([NDJSON](http://ndjson.org/)), whereby each line is an individual JSON document, separated by a newline (`\n`, and also `\r\n`).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 10, 2020, 11:41am UTC](https://discuss.elastic.co/t/dump-json-directly-in-to-elasticsearch/240940/6 "2020-08-10T11:41:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.