Error json parsing opensearch logs with logstash

Hello, i'm trying to parse suricata, logstash and opensearch logs with dictionary filter, here's part of my config

input {
  file {
    path => "/opt/logs/opensearchTest/opensearch_server.json"
    codec => "json"
    type => "opensearch_log"
  }
}
filter {
  if [type] == "opensearch_log" {
    if [level] == "ERROR" {
      mutate {
        add_field => { "translated_values" => "********** " }
      }
    } else {
      translate {
        source => "message"
        target => "translated_field"
        dictionary => {
          "*******" => "*******"
          "**********" => "****"
          "*********" => "**********"
        }
        fallback => ""
      }

      mutate {
        add_field => { "translated_values" => "%{translated_field}" }
        remove_field => [ "message", "translated_field" ]
      }
    }
  }
}
output {
 else if [type] == "logstash_log" or [type] == "suricata_log" or [type] == "opensearch_log" {
    if [translated_values] {
      http {
        http_method => "post"
        url => "http://************/"
        format => "form"

        headers => {
          "Authorization" => "Token *********"
          "Content-Type" => "application/json"
        }

        mapping => ["action_result", "True", "action_type", "add_audit", "comment", "%{translated_values}"]
      }
    }
  } 
}

and i've got the same pattern for logstash and suricata, that perfectly works, but with opensearch logs i receive this error:

JSON parse error, original data now in message field {:message=>"incompatible json object type=java.lang.String , only hash map or arrays are supported", :exception=>LogStash::Json::ParserError, :data=>"\"type\": \"server\", \"timestamp\": \"2023-10-19T00:00:00,834+03:00\", \"level\": \"INFO\", \"component\": \"o.o.c.m.MetadataUpdateSettingsService\", \"cluster.name\": \"opensearch\", \"node.name\": \"astra-altar-wazuh\", \"message\": \"updating number_of_replicas to [0] for indices [wazuh-monitoring-2023.42w]\", \"cluster.uuid\": \"7wgF5XaTRuqjNBptO52m-g\", \"node.id\": \"JBRbEq1VTey6vZjMxcWPfQ\"

here's example of opensearch logs:

{"type": "server", "timestamp": "2023-10-19T11:33:04,042+03:00", "level": "INFO", "component": "o.o.p.PluginsService", "cluster.name": "opensearch", "node.name": "astra-altar-wazuh", "message": "PluginService:onIndexModule index:[wazuh-monitoring-2023.37w/mbsyMgM7STWGQ6zkR6FdHw]", "cluster.uuid": "7wgF5XaTRuqjNBptO52m-g", "node.id": "JBRbEq1VTey6vZjMxcWPfQ"  }
{"type": "server", "timestamp": "2023-10-19T11:33:04,045+03:00", "level": "INFO", "component": "o.o.p.PluginsService", "cluster.name": "opensearch", "node.name": "astra-altar-wazuh", "message": "PluginService:onIndexModule index:[.kibana_1/0XetZAI8QqSmtSAOuOrsxg]", "cluster.uuid": "7wgF5XaTRuqjNBptO52m-g", "node.id": "JBRbEq1VTey6vZjMxcWPfQ"  }
{"type": "server", "timestamp": "2023-10-19T11:33:04,123+03:00", "level": "INFO", "component": "o.o.c.r.a.AllocationService", "cluster.name": "opensearch", "node.name": "astra-altar-wazuh", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana_1][0]]]).", "cluster.uuid": "7wgF5XaTRuqjNBptO52m-g", "node.id": "JBRbEq1VTey6vZjMxcWPfQ"  }
{"type": "server", "timestamp": "2023-10-19T11:34:25,979+03:00", "level": "INFO", "component": "o.o.n.Node", "cluster.name": "opensearch", "node.name": "astra-altar-wazuh", "message": "stopping ...", "cluster.uuid": "7wgF5XaTRuqjNBptO52m-g", "node.id": "JBRbEq1VTey6vZjMxcWPfQ"  }

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

The message indicates that your log entries are not surrounded by {}

but as you see they surrounded by {} :frowning:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.