Logstash Json Split Filter

Elastic Stack Logstash
1

I am trying to use the Split filter in Logstash to separate my Json data into separate documents in elastic. I have used this same Split filter in another case and did not have this problem. I have tried targeting a few different fields and get the same error message. Please let me know if I can clarify anything posted as this is my FIRST forum post. Thanks for taking the time to read.

Here is the Error message I'm getting in Logstash.

Error: [2022-05-04T07:18:01,115][WARN ][logstash.filters.split ][main][f8424efe02fef8508f1f30680490ad8a3d9a978ef5ebc0301f9217ded02aec62] Only String and Array types are splittable. field:Dispatch is of type = NilClass

Logstash Config:

    http_poller{
        urls =>{
            
            test2 => {
            
                method => get
                user => "Username"
                password => "XXXX"
                url => "https://360test.test.net/360API/director.php/360API/360data/V1/Dispatch?modifier=TestMobile"
                type => "json"
                codec => "json"
                headers => {
                    Accept => "application/json"
                    "Content-Type" => "application/json"
                    }
                }
            }
            request_timeout => 60
            schedule => { cron => "*/1 * * * * UTC" }
            
            
        }
 }
 filter {
      split {
               field => "Dispatch"   
             }
 }

       

 output{
     elasticsearch {
         hosts => ["https://CLOUD END POINT"]
         index => "360_data"
         user => "User"
         password => "XXXX"
     }
     stdout { codec => rubydebug }
 }

Return:

{
       "package" => {
        "serviceresponse" => [
            [0] {
                 "credentials" => {
                    "sessiontoken" => "N/A"
                },
                        "data" => {
                    "Dispatch" => [
                        [ 0] {
                               "Company" => "Bethel Company",
                              "Priority" => "5.00",
                                "CallNo" => "608357",
                            "CustomerNo" => "BETHELPO01",
                            "StatusCode" => "TOSCHEDULE          ",
                              "SiteName" => "Bethel Company",
                                  "href" => "https://360test.test.net/360API/director.php/Dispatch/24143858  ",
                             "CloseDate" => nil,
                                  "Date" => "2022-05-01 00:00:00.000",
                                "SiteNo" => "4229719   ",
                            "DispatchNo" => "24143858  "
                        },
                        [ 1] {
                               "Company" => "Boston Company                                                                     ",
                              "Priority" => "4.00",
                                "CallNo" => "608440",
                            "CustomerNo" => "BOSTONLA01",
                            "StatusCode" => "TOSCHEDULE          ",
                              "SiteName" => "Boston Company                                                                     ",
                                  "href" => "https://360test.test.net/360API/director.php/Dispatch/24350276  ",
                             "CloseDate" => nil,
                                  "Date" => "2022-05-01 00:00:00.000",
                                "SiteNo" => "4229754   ",
                            "DispatchNo" => "24350276  "
                        },
                        [ 2] {
                               "Company" => "Boston Company                                                                    ",
                              "Priority" => "5.00",
                                "CallNo" => "608424",
                            "CustomerNo" => "BOSTONPS01",
                            "StatusCode" => "OPEN                ",
                              "SiteName" => "Boston Company                                                                    ",
                                  "href" => "https://360test.test.net/360API/director.php/Dispatch/24277622  ",
                             "CloseDate" => nil,
                                  "Date" => "2022-04-30 00:00:00.000",
                                "SiteNo" => "4229762   ",
                            "DispatchNo" => "24277622  "
                        },
                        [ 3] {
                               "Company" => "Bourne;City                                                                                      ",
                              "Priority" => "4.00",
                                "CallNo" => "608439",
                            "CustomerNo" => "BOURNE;T01",
                            "StatusCode" => "TOSCHEDULE          ",
                              "SiteName" => "Bourne City",
                                  "href" => "https://360test.test.net/360API/director.php/Dispatch/24350269  ",
                             "CloseDate" => nil,
                                  "Date" => "2022-05-01 00:00:00.000",
                                "SiteNo" => "4229209   ",
                            "DispatchNo" => "24350269  "
                        },
                    ],
                      "Status" => {
                        "UserVariable" => "Success",
                              "nError" => 0,
                        "ErrorMessage" => "No Error",
                             "Message" => 0,
                                 "TTL" => "0"
                    }
                },
                     "service" => "360data",
                     "appname" => "360API",
                      "method" => "List",
                    "modifier" => "TestMobile",
                     "version" => "1",
                "dataprovider" => "DEFAULT",
                      "object" => "Dispatch",
                         "sid" => "477226"
            }
        ]
    },
      "@version" => "1",
    "@timestamp" => 2022-05-04T14:19:00.682Z,
          "tags" => [
        [0] "_split_type_failure"
    ]
}
2

Hi I have also asked this same question on stackoverflow

3

Hi,

Dispatch is not a field but a nested field. So when you ask logstash to split on Dispatch, logstash told you that the type of the field is NilClass because the Dispatch field do not exist.

You should do something more like this

 filter {
      split {
               field => "[package][serviceresponse][0][Dispatch]"   
             }
 }

Cad.

1 Like
4

Hi Cad, thanks for the speedy response!

I have tried replacing my filter with your recommendation and I'm still getting a similar error response.

[2022-05-04T08:23:00,955][WARN ][logstash.filters.split   ][main][f408d6865f097d692d3fa891073d4f438fd78766f838c38dea43f7b248ffb7e5] Only String and Array types are splittable. field:[package][serviceresponse][0][Dispatch] is of type = NilClass

Any addition thoughts on this?

Thank you

5

Then try.

filter {
      split {
               field => "[package][serviceresponse][0][data][Dispatch]"   
             }
 }

If it is still not working, i let you found the good path to the Dispatch field yourself ^^

Cad.

1 Like
6

Hi Cad

The new path you gave worked!

Thanks so much for the help

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.