Decode json in _source into fields

(Leandro Maciel) #1


I'm using Filebeat and Logstash to analyze logs from Wazuh, the logs are json files where each line is a json string, I can read the log correctly and push it into Elasticsearch, through Logstash, and it correctly populate the fields, but in the discover tab on Kibana, the _source field shows the raw json.

Is there any way to decode the _source raw json and show the fields in the discover tab?

This is what I'm getting now:

This is the way I want the _source field to behave:

My filebeat configuration is:

  - input_type: log
     - "/var/ossec/logs/alerts/alerts.json"
    document_type: wazuh-ossec
    codec: json_lines
    json.message_key: log
    json.keys_under_root: true
    json.overwrite_keys: true

   hosts: [""]

My logstash pipeline for this index is:

input {
	beats {
        host    => ""
        port    => 5000
        codec   => "json_lines"
        type    => "wazuh-ossec"
filter {
	if [type] == "wazuh-ossec" {
	        geoip {
                source => "srcip"
                target => "GeoLocation"
                fields => ["city_name", "continent_code", "country_code2", "country_name", "region_name", "location"]
	        date {
                match => ["timestamp", "ISO8601"]
                target => "@timestamp"
	        mutate {
                remove_field => [ "timestamp", "beat", "fields", "input_type", "tags", "count", "@version", "log", "offset"]
output {
	if [type] == "wazuh-ossec" {
	        elasticsearch {
                hosts                   => ["localhost:9200"]
                index                   => "wazuh-alerts-%{+YYYY.MM.dd.HH}"
                codec                   => "json_lines"
                document_type           => "wazuh-ossec"
                template                => "/etc/logstash/wazuh-elastic5-template.json"
                template_name           => "wazuh"
                template_overwrite      => true

What am I missing?

(Leandro Maciel) #2


With the help from the guys from the Wazuh mailing list I was able to solve the problem.

The solution was to explicit declare the _source field as the value on the metaFields variable in the advanced settings on Kibana.

I don't know if this is the normal behavior, if I missed something or if it is a bug, but explicit declaring the values on the metaFields variable solves the problem.

Just posting the solution before the topic closes.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.