Split fields in different docs

Elastic Stack Logstash
1

Hello friends. I am trying to reindex some data to another index but with a little differences.
In the first index I have this kind of hit:

{
  "France": {
    "Testing": {
      "status": "passed"
    }
  },
  "Spain": {
    "Testing": {
      "status": "passed"
    }
  },
  "Brazil": {
    "Testing": {
      "status": "passed"
    }
  },
  "USA": {
    "Testing": {
      "status": "failed"
    }
  }
}

So I want to split each country in other doc each one some like this:

{
  "Country": {
    "Testing": {
      "status": "passed"
    }
  }
}

So I was doing this with logstash but I failed, I have this config. Where I am tryin to get all the values from the fields but I don't know how to get it. I want to do some like in kibana where use the "*" but I don know how to use it in logstash


input {
  elasticsearch {
    hosts => "localhost:9200"
    index => "informationCountry"
    size => 500
    scroll => "5m"
    docinfo => true
  }
}
filter{
ruby {
           code => '
          f = event.get("[*][Testing][status]") 
        if f.is_a? Hash
            newF = []
            f.each { |k, v|
                newF << { "ENV": k }
            }
            event.set("[Country][Testing][status]", newF)
        end   '
    }

}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "testingInformation"
    document_id => "%{[@metadata][_id]}"
  }

}

Please help I tried with multiple ways but I can not got it

2

works well?

3

No :frowning:

4

You have to get the top level hash by event.get("message") and analyze it.

5

How can I get it? because i tried with puts event.get("message") to print the value but I don get nothing is empty

6

Sorry, you had to use event.to_hash with Elasticsearch input plugin.

Use this filter:

input {
  elasticsearch {
    docinfo => true
  }
}
filter{
    ruby {
        code => '
            keys = event.to_hash.keys
            array = []
            keys.each{|k|
                if !(k.start_with?("@")) then
                    array << {"ENV": k}
                    event.remove(k)
                end
            }
            event.set("[Country][Testing][status]", array)
        '
    }
}
output {
    stdout {
        codec => rubydebug{metadata => true}
    }
}

You will get:

{
    "@timestamp" => 2022-02-01T09:44:25.393Z,
     "@metadata" => {
        "_index" => "test_split_fields",
           "_id" => "QhaitH4Bf0nakUP8oFTM",
         "_type" => "_doc"
    },
      "@version" => "1",
       "Country" => {
        "Testing" => {
            "status" => [
                [0] {
                    "ENV" => "Spain"
                },
                [1] {
                    "ENV" => "USA"
                },
                [2] {
                    "ENV" => "France"
                },
                [3] {
                    "ENV" => "Brazil"
                }
            ]
        }
    }
}
7

Oh! yes I get this but how can add the data?, I was seeing the data on Kibana but the field are empty :thinking:

{
    "@timestamp" => 2022-02-01T17:03:43.789Z,
     "@metadata" => {
        "_index" => "informationCountry",
         "_type" => "_doc",
           "_id" => "0Xhuq34BrOPMccolsPMW"
    },
      "@version" => "1",
       "Country" => {
        "Testing" => {
            "status" => [
                [ 0] {
                    "ENV" => "France"
                },
                [ 1] {
                    "ENV" => "Spain"
                },
                [ 2] {
                    "ENV" => "Brazil"
                },
                [ 3] {
                    "ENV" => "USA"
                },
                [ 4] {
                    "ENV" => "localtime"
                }
            ]
        }
    }
}
8

Are you using appropriate output plugin?

What does " the field are empty" mean? You found indexed documents but fields are empty? Or documents themselves are not indexed? In such debugging situation, you should use dev tools and REST API to exclude other problems.

9

I am using this.

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "testingInformation"
    document_id => "%{[@metadata][_id]}"
  }

}

I get in the hit this:

 "Country" : {
            "Testing" : {
              "status" : [
                {
                  "ENV" : "Spain"
                },
                {
                  "ENV" : "USA"
                },
                {
                  "ENV" : "France"
                },
                {
                  "ENV" : "Brazil"
                }
              ]
            }
          }

but not the results of each country, the Env: passed or Env: failed. How can I get the values that are into each key?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.