How to write filter part in my conf file for json file as input

Here is my conf file look like:

input
{
file
{
codec => "json"
path => "/ci/data/cirrus/report/*/status.json"
start_position => "beginning"
sincedb_path => "/ci/data/cirrus/sincedb_path/status.db"
type => "STATUS"
ignore_older => 0
}
}

filter {
json {
source => "path"
add_field =>
{
"status" => "%{field1}"
"cirrus_id" => "%{field2}"
}
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["localhost:9200"]
index => "scan"
}
}

what's wrong with conf file?? and what should i write inorder to get certain fields from my json file.

my json file look like:

{
"status": "abort",
"time_start": 1494845194.7839701,
"agent_pid": 16466,
"id": "ci-000c2969deb0_administrator_1",
"time_queued": 1494845187.88799,
"time_end": 1494845194.8816111,
"message": "catch exception [[Errno 2] No such file or directory: u'/ci/data/cirrus/jobs/ci-000c2969deb0_administrator_1/input/job.json_from_(no suts defined)'];1 stage(s) failed",
"stages": {
"run_sys_cmd_sut": {
"status": "fail",
"message": "failed to connect to sut, unknown SUT OS type"
}
}
}

i want to store get "status", "id" , "time_queued " from the above json file as an output. for this what should i write in my filter part.

your json file is "pretty printed"
In the file input with a json codec, each line is expected to be a full json object per line.
e.g.

{"field1":"value1","field2":"value2","field3":"value3","field4":"value4"}
{"field1":"value1","field2":"value2","field3":"value3","field4":"value4"}

You should consider using filebeat to do the multiline on the CI box and send the data to a Logstash box.

Failing that, consider using a script that reads files from /ci/data/cirrus/report/*/status.json, removes the newline characters and appends the result text as one line to a new file that you "tail" in the file input.

As a last resort you can try using the multiline codec in the file input but its not easy see stackoverflow

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.