Bulk Import to Elasticsearch

Elastic Stack Elasticsearch
1

Hello,
I have a JSON file of 1 GB Logdata. The format of the file is

{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value2" }

I have tried increasing the size of file in Bulk Import. I can successfully import Json File up to 160 MB. All the files above the size are not indexed and is ignored by elasticsearch withour giving an error.

How can I send the 1 GB big log data at once to elasticsearch. I cannot break the log file to small piece and send it to Elasticsearch, as it is created automatically by a program.

2

How are you sending it to Elasticsearch?

3

I cannot break the log file to small piece and send it to Elasticsearch, as it is created automatically by a program.

You must break it into smaller pieces.

You can use Logstash for this, even a unix script can probably do it.

4

Thank you for a quick Reply. As I Early mentioned, the log file is written in a format for a Bulk import.
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test", "_type" : "type1" } }
{ "field1" : "value2" }

If I use Logstash it must ignore the every odd Lines { "index" : { "_index" : "test", "_type" : "type1" } }.

As I am new to logstash, can I do it easily?

5

I’d just split the file into sub files,
A unix script will do that easily IMO

6

curl -XPOST 'elasticsearch:9200/_bulk' -d @test.json

It works for small sized file but my file is more than 1 Gb. Now I am using Logstash and its working fine,

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.