How to import json file to Elasticsearch

[Ahmet_Kartal (Ali Can)]

Hi, I have 1 billion json data. I want to import these data to my local elasticsearch. All the documents in one json file. What is the easiest way to do that?

I have tried this but did not work for me: curl -XPOST http://localhost:9200/project/_doc -H "Content-Type: application/json" -d @ahmet.json

[dadoonet (David Pilato)]

@Ahmet_Kartal

did not work for me

Any details about what did not work?
May be you did not follow the bulk request pattern? I guess you are missing header for each Json.

@Ahmet_Kartal

All the documents in one json file.

You probably need to write a script or use filebeat.
Do you have one line per JSon?

If you have one file per document, you can otherwise try FSCrawler. It has an option to import json files.

[Ahmet_Kartal (Ali Can)]

@dadoonet

Any details about what did not work?

I just typed that cURL command and it did not send data to my localhost:9200. It returned null line.

@dadoonet

Do you have one line per JSon?

Each json seperated from each other by new line.For example like this:

{"type": "server", "timestamp": "2019-12-07T19:09:46,351Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "heap size [989.8mb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2019-12-07T19:09:46,353Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "node name [f97588749c79], node ID [qJbs-xKRTxiaoIGXbrsVkg], cluster name [docker-cluster]" }

[Ahmet_Kartal (Ali Can)]

Anyone can help ?

[vijaykriishna (Vijay Krishna)]

Try this.

Data format must be as follows for every json
1. first line containing index name
2. actual data to be inserted into the index

sample data:

{"index": {"_index": "index_name", "_type": "index_type", "_id": "doc_id"}}
{"type": "server", "timestamp": "2019-12-07T19:09:46,351Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "heap size [989.8mb], compressed ordinary object pointers [true]" }
{"index": {"_index": "index_name", "_type": "index_type", "_id": "doc_id"}}
{"type": "server", "timestamp": "2019-12-07T19:09:46,353Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "node name [f97588749c79], node ID [qJbs-xKRTxiaoIGXbrsVkg], cluster name [docker-cluster]" }

curl -H 'Content-Type: application/json' -XPOST 'localhost:9200/{index}/{type}/_bulk?pretty' --data-binary @<file_name>.json