How to ingest multiple json documents from a folder into ES using logstash

[Steve64 (Steve)]

Continuing the discussion from Ingesting JSON files, format problem?:

@CraigFoote

I have a series of JSON files, each with one rather large, single-line JSON document. I'm trying to ingest via the file input plugin:

input{
	file{
		path => "/path/to/files/*.txt"
		start-position => "beginning"
		sincedb_path => "/dev/null"
		codec => "json"
	}
}
output{
	stdout{ codec => rubydebug }
}

I think there's something wrong with my json format because it works when I put a test.txt file in that folder with content:
{"test1":"test2"}

The series of json files I want to ingest are of the format:

{"messages":[{"text": ["The result was retrieved successfully."], "level": "INFO"}], "results":[{"field1": "value1", "field2": [{"field2a": "value2a", "field2b": "value2b"}], "field3": "value3"}]}

Is there a problem here? I don't get any errors and the files are found but I see nothing from rubydebug. Could it be the length of the files, they are a couple MB each.

[rclarke (Robin Clarke)]

Hello Steve,

That should work. Could you check that the json files also match the pattern *.txt ?

Cheers,
-Robin-