Splitting JSON arrays

Elastic Stack Logstash
1

Hi,

I have indexed a json file without any filter and queried the corresponding index to first explore the structure before splitting it in an appropriate way.

I found the field of interest with the following query and would like to have separate events for each "myCalculations":

   GET /myindex/_search
    {
      "_source": {
        "includes": [ "myChildren.myChildren.myChildren.myChildren.myChildren.myChildren.myChildren.myChildren.myCalculations.UVALUE"]
      },
      "query": {
        "match_all": {}
      }
    }

Giving me the following result:

  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "doc",
        "_id" : "D97cAGkBR4-_LQPCdwKP",
        "_score" : 1.0,
        "_source" : {
          "myChildren" : [
            {
              "myChildren" : [
                {
                  "myChildren" : [
                    {
                      "myChildren" : [
                        {
                          "myChildren" : [
                            {
                              "myChildren" : [
                                {
                                  "myChildren" : [
                                    {
                                      "myChildren" : [
                                        {
                                          "myCalculations" : {
                                            "UVALUE" : [
                                              {
                                                "concernedObjectName" : "maçonnerie_dimension",
                                                "valueList" : null,
                                                "value" : 0.5166241434124577,
                                                "concernedObjectClass" : "Paroi"
                                              }
                                            ]
                                          }
                                        },
                                        {
                                          "myCalculations" : {
                                            "UVALUE" : [
                                              {
                                                "concernedObjectName" : "fen1",
                                                "valueList" : null,
                                                "value" : 1.6878,
                                                "concernedObjectClass" : "Paroi"
                                              }
                                            ]
                                          }
                                        }, ........

So each myCalculations should be a new event.

I thought that the config file below would give me the result I want but I get

Only String and Array types are splittable. field:[myChildren][myCalculations] is of type = NilClass

Suggesting my path filter is wrong ???

input {

    file {

    path => "pathtojson/myfile.json"
    start_position => "beginning"
    sincedb_path => "NUL"

      }
    }

    filter {
      json { source => "message" target => "theJSON" store_json => true force_array => false }
      mutate { rename => { "[theJSON][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren]" => "myChildren" } }
      split { field => "[myChildren][myCalculations]" }
      }

    output
    {
    	stdout {
    		codec => dots
    	}

     	elasticsearch {
          index => "myindex"
      	}

    }

Thanks in advance for any help !

2

What does the complete myfile.json look like? Or else theJSON from

 output { stdout { codec => rubydebug } }

Split is getting that error because [myChildren][myCalculations] does not exist, which means the mutate+rename failed.

1 Like
3

Hi,

Thanks and sorry for the late reply. The Json is really too large to paste it here.

I managed to solve my issue by doing successive splits, altough the solution is ugly it works.

filter {
  json { source => "message" target => "theJSON" remove_field => "message"}
  mutate { rename => { "[theJSON][myChildren]" => "test" } }
    split {field => [test]}
    split { field => "[test][myChildren]" }
    split { field => "[test][myChildren][myChildren]" }
    split { field => "[test][myChildren][myChildren][myChildren]" }
    split { field => "[test][myChildren][myChildren][myChildren][myChildren]" }
    split { field => "[test][myChildren][myChildren][myChildren][myChildren][myChildren]" }
    split { field => "[test][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren]" }
    split { field => "[test][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren]" }
    split { field => "[test][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren][myChildren]" }
}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.