Get Raw JSON file with half a million lines into elasticsearch

Elastic Stack Elasticsearch
1

Hi guys,

I feel like 'ive tried just about everything to achieve this to no success, so I won't bother going into great detail about all my tests. So far ive attempted getting my Raw JSON file into elasticsearch using curl, jsonpyes and logstash but none of these methods have worked. I would use the bulk API but this would mean adding headers for every line in a giant raw JSON file which I don't know how to do.

If anyone else has worked with getting large JSON files into elasticsearch i'd love to hear how you achieved this. Seems quite ironic how hard it is to get large JSON files into elasticsearch considering JSON is the supposed preferred input for elasticsearch.

Cheers

2

How many documents do you have in that file? Elasticsearch does speak json indeed, but it does not mean you can push arbitrary json to it.

3

Im unsure what you mean by documents, but the structure of my JSON looks like the attached images. I've noticed that the last 3 values of every field are null but I don't think that is what is causing the error.

the jsonpyes tool analysed the raw JSON as valid but im not sure if its accurate or not either.

4

That looks like a very, very large document with a lot of fields. How many fields are there in the document? What does all the numeric keys shown in the screenshot represent?

5

They are Snort IDS alerts generated from the unified2 output and converted to JSON using u2json, For details on each field you can look here: http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node21.html

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.