Parse the json [{},{},{}][{},{},{}] with logstash

Elastic Stack Logstash
1

Hello,

I have a json from scrapy with this format:

[{"website1":"myurl1","importdate":"X"},{"website2":"myurl2","importdate":"X"}]
[{"website1":"myurl1","importdate":"Y"},{"website3":"myurl3","importdate":"Y"}]

So a new array each time when I run my script,and I would like to have the output with hits like this:
hit1: hit1.website1 = myurl1 hit1.importdate= X
hit2: hit2.website2 = myurl2 hit1.importdate= X
hit3: hit3.website3 = myurl3 hit1.importdate= Y

I tried:

    input {
  file {
    path => ['/tmp/myscriptresult/*.json']
    type => "json"
    codec => "json"
  }

}

filter 
{

}


output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		user => "myuser"
		password => "mypassword "
		ecs_compatibility => disabled
   	index => "cluster-%{+YYYY.MM.dd}"
    document_type => "%{[logger]}"
	}
}

But I have errors like

logstash_1 | [2020-11-17T08:44:21,928][ERROR][logstash.codecs.json ][main][3ef39f6257bc46c22e35cdf979584572f625dccfcd32b0b0841e54250aff99af] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected close marker ']': expected '}' (for root starting at [Source: (String)"]["; line: 1, column: 0])
logstash_1 | at [Source: (String)"]["; line: 1, column: 2]>, :data=>"]["}

Do you have an idea how to do the filter ?

Thanks,

Thib

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.