Import Json file by using Sense Interface

Elastic Stack Elasticsearch
1

Hi

I'm new in elasticsearch. I have created a new index and defined its mapping .
I want to import a json file from my computer into elasticsearch by using sense in order to analyze it in kibaba. How can I do it?

Thanks

2

What does the JSON file look like?

If you have a lot of docs, you can use Bulk API for that. But note that you need headers.

3

My json file looks like as following:
[{"id" : 24567, "source" : "Personal" , "startTime" : "2015-05-04 19:05:53"....},{"id" : 24568, "source" : "Personal" ,"startTime" : "2015-05-04 20:05:53"....},{"id" : 24569, "source" : "Personal" , "startTime" : "2015-05-04 19:10:53"....} ,...]

And number of document is very high (around 3 million)
when I am using the following code in sense , It is not working

POST /my_index -d @"C:\MyComputer...\file.json"

But the point is, how can i use Bulk API in sense and add the headers ?

thank you so much for your help

4

Sense is not intended to send 3m records. That's "just" a console to play with/manage elasticsearch.

Can you generate your 3m documents in another format than this one?
If not, it means that you will have to pre-process them with something like Logstash or any ETL tool you like.

If you can, that's better.

Generate in that case a file which is looking like:

{ "index" : { "_id" : "24567" } }
{"id" : 24567, "source" : "Personal" , "startTime" : "2015-05-04 19:05:53"....}
{ "index" : { "_id" : "24568" } }
{"id" : 24568, "source" : "Personal" ,"startTime" : "2015-05-04 20:05:53"....}

Generate a file for every 10000 doc for example. Then you can send those files using a curl command (although it could be hard on windows!).

If you generate one file per document, you could also use fscrawler with this option: https://github.com/dadoonet/fscrawler#indexing-json-docs but I'm unsure it will work well for 3m docs...

Another question is where is this data coming from? That might change the answer I'm giving...

5

Thank you so much for your quick response.
About format of document, I can say that:

The data is an output of R and the original format is .Rdata . I converted this format to .json in R to import into elasticsearch. But I can have .txt and .csv format of data file.

6

You would be better of using Logstash, or something else.