How to write filter part in my conf file for json file as input


(Gopi K) #1

Here is my conf file look like:

input
{
file
{
codec => "json"
path => "/ci/data/cirrus/report/*/status.json"
start_position => "beginning"
sincedb_path => "/ci/data/cirrus/sincedb_path/status.db"
type => "STATUS"
ignore_older => 0
}
}

filter {
json {
source => "path"
add_field =>
{
"status" => "%{field1}"
"cirrus_id" => "%{field2}"
}
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["localhost:9200"]
index => "scan"
}
}

what's wrong with conf file?? and what should i write inorder to get certain fields from my json file.

my json file look like:

{
"status": "abort",
"time_start": 1494845194.7839701,
"agent_pid": 16466,
"id": "ci-000c2969deb0_administrator_1",
"time_queued": 1494845187.88799,
"time_end": 1494845194.8816111,
"message": "catch exception [[Errno 2] No such file or directory: u'/ci/data/cirrus/jobs/ci-000c2969deb0_administrator_1/input/job.json_from_(no suts defined)'];1 stage(s) failed",
"stages": {
"run_sys_cmd_sut": {
"status": "fail",
"message": "failed to connect to sut, unknown SUT OS type"
}
}
}

i want to store get "status", "id" , "time_queued " from the above json file as an output. for this what should i write in my filter part.


(Guy Boertje) #2

your json file is "pretty printed"
In the file input with a json codec, each line is expected to be a full json object per line.
e.g.

{"field1":"value1","field2":"value2","field3":"value3","field4":"value4"}
{"field1":"value1","field2":"value2","field3":"value3","field4":"value4"}

You should consider using filebeat to do the multiline on the CI box and send the data to a Logstash box.

Failing that, consider using a script that reads files from /ci/data/cirrus/report/*/status.json, removes the newline characters and appends the result text as one line to a new file that you "tail" in the file input.

As a last resort you can try using the multiline codec in the file input but its not easy see stackoverflow


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.