Remove_fields doesn't remove field in json

[Kostyantyn_Dobriohlo]

Cannot remove fields in json, 'cause always have a json parsing error. Tried many different ways, but inside the json nothing changes. And there is a warning in logs: Error parsing json

Config file:

input {
  tcp {
    port => 5959
    codec => json {
      target => "extra"
    }
  }
}
filter {
  json {
    source => "extra"
    remove_field => ["line"]
  }
  mutate {
    remove_field => ["message", "program", "host", "logsource"]
  }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "log-stash"
  }
}

Warning message:

logstash_1       | [WARN ] 2022-09-26 13:44:40.745 [[main]>worker4] json - Error parsing json {:source=>"extra", :raw=>{"interpreter_version"=>"3.9.2", "interpreter"=>"C:\\Users\\Kostya\\Desktop\\Работа\\testShit\\Scripts\\python.exe", "logger_name"=>"test", "path"=>"/21", "func_name"=>"log_to_file", "client"=>"127.0.0.1", "line"=>96, "status"=>400, "logstash_async_version"=>"2.5.0", "params"=>{"pk"=>"21"}, "time"=>"2022-09-26T13:44:29.000Z", "method"=>"GET", "process_name"=>"SpawnProcess-1", "thread_name"=>"MainThread"}, :exception=>#<Java::JavaLang::ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash and org.jruby.RubyIO are in unnamed module of loader 'app')>}

[stephenb (Stephen Brown)]

Hi @Kostyantyn_Dobriohlo Welcome to the community!

@Kostyantyn_Dobriohlo

{"interpreter_version"=>"3.9.2", "interpreter"=>"C:\\Users\\Kostya\\Desktop\\Работа\\testShit\\Scripts\\python.exe",

That does not look like valid json to me... => is not valid json so whatever is writing that log line is not writing valid json.

it would be

{"interpreter_version" : "3.9.2", "interpreter" : "C:\\Users\\Kostya\\Desktop\\Работа\\testShit\\Scripts\\python.exe",

Can you show a couple raw lines of your logs?

EDIT Ohh Good Eyes @Badger I did not see the first codec! Was wondering why it already looked "logstash-ized" !!

[Badger]

@Kostyantyn_Dobriohlo you have already parsed the json using the codec, there is no need for a json filter.

input { generator { count => 1 lines => [ '{ "a": 1, "b": 2 }' ] codec => json { target => "extra" } } }
filter { json { source => "extra" remove_field => ["line"] } }
output { stdout { codec => rubydebug { metadata => false } } }

will produce the error

 Error parsing json {:source=>"extra", :raw=>{"a"=>1, "b"=>2},

because [extra] is a hash, not a string.