How to import json file to Elasticsearch

Elastic Stack Elasticsearch
1

Hi, I have 1 billion json data. I want to import these data to my local elasticsearch. All the documents in one json file. What is the easiest way to do that?

I have tried this but did not work for me: curl -XPOST http://localhost:9200/project/_doc -H "Content-Type: application/json" -d @ahmet.json

2

Any details about what did not work?
May be you did not follow the bulk request pattern? I guess you are missing header for each Json.

You probably need to write a script or use filebeat.
Do you have one line per JSon?

If you have one file per document, you can otherwise try FSCrawler. It has an option to import json files.

3

I just typed that cURL command and it did not send data to my localhost:9200. It returned null line.

Each json seperated from each other by new line.For example like this:

{"type": "server", "timestamp": "2019-12-07T19:09:46,351Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "heap size [989.8mb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2019-12-07T19:09:46,353Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "node name [f97588749c79], node ID [qJbs-xKRTxiaoIGXbrsVkg], cluster name [docker-cluster]" }
4

Anyone can help ?

5

Try this.

Data format must be as follows for every json
1. first line containing index name
2. actual data to be inserted into the index

sample data:

{"index": {"_index": "index_name", "_type": "index_type", "_id": "doc_id"}}
{"type": "server", "timestamp": "2019-12-07T19:09:46,351Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "heap size [989.8mb], compressed ordinary object pointers [true]" }
{"index": {"_index": "index_name", "_type": "index_type", "_id": "doc_id"}}
{"type": "server", "timestamp": "2019-12-07T19:09:46,353Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "f97588749c79", "message": "node name [f97588749c79], node ID [qJbs-xKRTxiaoIGXbrsVkg], cluster name [docker-cluster]" }

curl -H 'Content-Type: application/json' -XPOST 'localhost:9200/{index}/{type}/_bulk?pretty' --data-binary @<file_name>.json

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.