File Input codec => "json_lines" won't work

[panagiss]

I try to read from a json file but codec=>"json_lines" won't work, can't see any output. But when i try and set codec=>"json" it works and parses the data and i see the output.

So you ask me what is the format of the json file. Well the json file was records from beats that i save into a file. So the output must be by default json_lines as the docs state. So i cannot see why is this happening...

Config to save file:

input{
    beats {
        port => 5044
     }
}


output{
    stdout {
       codec => rubydebug 
    }
    file{
         path => "/usr/share/logstash/config/auditbeat.json"
    }
}

Now here is a sample of json file:

And now here is the config file i try to read from the file:

input{
    file{
        path => "/usr/share/logstash/config/auditbeat.json"
        start_position => "beginning"
        codec => "json_lines"
    }
}

output{
    stdout {
       codec => rubydebug 
    }
}

That config won't print anything on the stdout, but i switch to codec => json it will.

[panagiss]

Also i found out that while using the codec=>"json" in the input file plugin, i thought it worked ok but i got this error at the begging which is related to my filter:

Error parsing json {:source=>"destination", :raw=>{"ip"=>"127.0.0.1", "bytes"=>0, "port"=>16443, "packets"=>0}, :exception=>java.lang.ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash and org.jruby.RubyIO are in unnamed module of loader 'app'

Here is the updated conf with filter:

input{
    file{
        path => "/usr/share/logstash/config/auditbeat.json"
        start_position => "beginning"
        #ecs_compatibility => "v1"
        codec => "json"
    }
   
}

filter{
    json{
        source => "destination"
    }
}

output{
    stdout {
       codec => rubydebug 
    }
}

[Badger]

The documentation explains why json_lines does not work with a file input. json_lines expects a stream of lines separated by newlines. The file input strips the newline from a line, so the json_lines codec never thinks it has seen the end of the line.

The ClassCastException suggests to me that you do not need the json filter, that the [destination] field was already parsed by the json codec. You can see that [destination] is a RubyHash.