# Get Raw JSON file with half a million lines into elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [March 29, 2017, 2:54pm UTC](https://discuss.elastic.co/t/get-raw-json-file-with-half-a-million-lines-into-elasticsearch/80499 "2017-03-29T14:54:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![shookshank](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@shookshank](https://discuss.elastic.co/u/shookshank)\ **Post date:** [March 29, 2017, 2:54pm UTC](https://discuss.elastic.co/t/get-raw-json-file-with-half-a-million-lines-into-elasticsearch/80499/1 "2017-03-29T14:54:06Z")
Hi guys, I feel like 'ive tried just about everything to achieve this to no success, so I won't bother going into great detail about all my tests. So far ive attempted getting my Raw JSON file into elasticsearch using curl, jsonpyes and logstash but none of these methods have worked. I would use the bulk API but this would mean adding headers for every line in a giant raw JSON file which I don't know how to do. If anyone else has worked with getting large JSON files into elasticsearch i'd love to hear how you achieved this. Seems quite ironic how hard it is to get large JSON files into elasticsearch considering JSON is the supposed preferred input for elasticsearch. Cheers ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [March 29, 2017, 3:32pm UTC](https://discuss.elastic.co/t/get-raw-json-file-with-half-a-million-lines-into-elasticsearch/80499/2 "2017-03-29T15:32:54Z")
How many documents do you have in that file? Elasticsearch does speak json indeed, but it does not mean you can push arbitrary json to it. ---
**Author:** ![shookshank](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@shookshank](https://discuss.elastic.co/u/shookshank)\ **Post date:** [March 29, 2017, 4:31pm UTC](https://discuss.elastic.co/t/get-raw-json-file-with-half-a-million-lines-into-elasticsearch/80499/3 "2017-03-29T16:31:06Z")
Im unsure what you mean by documents, but the structure of my JSON looks like the attached images. I've noticed that the last 3 values of every field are null but I don't think that is what is causing the error. ![](https://us1.discourse-cdn.com/elastic/original/3X/3/7/37b6ec05230c7440f56058498e06994afccfb073.png) ![](https://us1.discourse-cdn.com/elastic/original/3X/e/8/e8c7226d6f6b1c8d0057c450fa9e3780f1b89f8b.png) ![](https://us1.discourse-cdn.com/elastic/original/3X/3/7/3717f97f183fb220bc37a571fb5bad7c33556f38.png) the jsonpyes tool analysed the raw JSON as valid but im not sure if its accurate or not either. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 29, 2017, 4:43pm UTC](https://discuss.elastic.co/t/get-raw-json-file-with-half-a-million-lines-into-elasticsearch/80499/4 "2017-03-29T16:43:32Z")
That looks like a very, very large document with a lot of fields. How many fields are there in the document? What does all the numeric keys shown in the screenshot represent? ---
**Author:** ![shookshank](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@shookshank](https://discuss.elastic.co/u/shookshank)\ **Post date:** [March 29, 2017, 5:07pm UTC](https://discuss.elastic.co/t/get-raw-json-file-with-half-a-million-lines-into-elasticsearch/80499/5 "2017-03-29T17:07:51Z")
They are Snort IDS alerts generated from the unified2 output and converted to JSON using u2json, For details on each field you can look here: [http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node21.html](http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node21.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 26, 2017, 5:07pm UTC](https://discuss.elastic.co/t/get-raw-json-file-with-half-a-million-lines-into-elasticsearch/80499/6 "2017-04-26T17:07:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.