ES Input plugin gives invalid json data

Elastic Stack Logstash
1

Hi All

I am using ES input and output as file, i get json data but this is multiline and not a valid json

here is my conf file

input {
  elasticsearch {
  hosts => ""
  index => "abcd"
  size => 10
  scroll => "5m"
  }
}

filter {

json {
source => "message"
target => "test"
}

  }

  output {
   file {
     path => "xyz.txt"
     codec => json
   }
  }

I get the result in out file as

{
	"P1" : "2",
	"Object" : "A005",
	"Count" : "2"
} {
	"P0" : "5",
	"Object" : "A001",
	"Count" : "5"
} {
	"P1" : "3",
	"Object" : "A002",
	"Count" : "3"
}

and my expected output is as below with comma between each document and stored in a array.

I have tried this with and without json filter but no luck. I think this is something to do with codes.

[{
	"P1" : "2",
	"Object" : "A005",
	"Count" : "2"
}, {
	"P0" : "5",
	"Object" : "A001",
	"Count" : "5"
} ,{
	"P1" : "3",
	"Object" : "A002",
	"Count" : "3"
}]

How can i achieve the above result, thanks

2

I don't think you can get logstash to produce that array. I would switch to the json_lines codec which gets you one item per line, then add the commas and square brackets outside of logstash.

3

oh i never thought this can be that challenging. I thought ES is a json storage and it should have capability to produce json as a output for all the document in a array

This was a surprising element for me. or we are missing something?

Is this not possible with ruby filter as well,please suggest..

4

Which it does. You might be better off reading the documents from elasticsearch using curl. logstash adds no value here, in fact it removes it, because it breaks up the array of results that elasticsearch returns. If you need help with submitting a query using curl I would ask in the elasticsearch forum.

5

okay let me try first with curl, Thanks for this idea.

6

curl command works, but it will flush out only 10 K records, i am dealing with millions of records. Then i will have to enable the Scroll API which is very costly and not a good solution.

Now i am stuck again with no solution...i feel ES input is really of no use.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.