Parsed JSON object/hash requires a target configuration option

Elastic Stack Logstash
1

My cluster architecture is elk + Kafka + filebeat. Use the following configuration to parse JSON logs:

input {
    kafka {
       bootstrap_servers => "192.168.1.176:9092,192.168.1.178:9092,192.168.1.177:9092"
       client_id => "logstash175"
       auto_offset_reset => "latest"
       topics => "games"
       group_id => "games"
       reconnect_backoff_ms => 1000
       security_protocol => "SASL_PLAINTEXT"
       sasl_mechanism => "PLAIN"
       jaas_path => "/etc/logstash/kafka-client-jaas.conf"
       codec => json
    }
}

filter {
        if [service] == "daxuan_room_prod" {
            #mutate {
            #    gsub => [ "message", "\t", "" ]
            #    gsub => [ "message", "\n", "" ]
            #}
            json {
                skip_on_invalid_json => true
                source => "message"
                #target => "jsonmsg"
                remove_field => ["message"]
            }
        } else {
            json {
                skip_on_invalid_json => true
                source => "message"
        }
    }
}

output {
    if [service] {
       elasticsearch {
           hosts => ["192.168.1.74:9200","192.168.1.165:9200","192.168.1.166:9200"]
           user => "elastic"
           password => "SaOGLWt"
           ssl => true
           ssl_certificate_verification => true
           cacert => "/etc/logstash/ssl/root.pem"
           manage_template => false
           index => "%{[service]}-%{+YYYY.MM.dd}"

Numerous warnings after startup:

[2021-10-08T17:08:12,118][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"  \"leave,money:59986,value:59986,time:2021-10-08 17:08:09\""}
[2021-10-08T17:08:12,139][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:12,140][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:12,940][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:12,949][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:12,959][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:13,217][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:13,225][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:13,236][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:14,198][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:14,200][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:14,469][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"  \"leave,money:114753,value:114753,time:2021-10-08 17:08:11\""}
[2021-10-08T17:08:14,475][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:14,480][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:14,482][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}
[2021-10-08T17:08:14,484][WARN ][logstash.filters.json    ][games][33373ed6490de41074cbfedd4242d4226c27daf2e272b02a4fe133a0de55aea7] Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\t"}

When I add the target configuration, the warning does not change. How do I fix it?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.