files about 1.2 gig
no idea how many lines...
my guess is that the json file is not newline delimited json and that's what's mucking it up now...
That would indeed be a problem. If that were the case it would ingest the whole file as a single event. This is java so characters are 2 bytes. So the event would be 2.4 GB, and the message it builds to send to elasticsearch would be another 2.4 GB. So you would need at least 5 GB of heap.
I would look into reformatting the input file.
Yea - I fixed the heap error.
I have the heap 8 gigs, and put 12 gigs ram on the box...
I ran a python script that I found hoping it would fix data, but it still hangs, so i'm not entirely sure if it's formatted correctly...
nope - looks like the commas are still in the file - the script looked like it would replace all the commas with carriage returns...
my guess is that, from your description, i would need something more than simply carriage returns if i wanted new documents per event within the file...
No, provided the JSON are newline (not CR) separated the file input should consume each line as an event.
Figured it out - now just need to nail down the issue with the "Event.date" field...
I get all the documents in just fine, but ES/Kibana doesn't like the date at all.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.