Logstash not writing to elasticsearch if split filter is used

Elastic Stack Logstash
1

Hi Guys,

I'm trying to use the split filter. But Logstash doesn't write to elasticsearch if I do so. However, if I comment out the split filter it works like a charm. Would really appreciate the help, following is my conf file:

input {
    stdin {}
}

filter {
  #if [source] =~ "junitResult.xml" {
    multiline {
        pattern => ".*"
        what => "next"
    }

    #ruby {
    #  code => "event['index'] = event['source'].match(/jobs\/(.*)\//)[1]
    #  event['pipeline'] = event['source'].match(/jobs\/(.*)\/builds\//)[1]"
    #}
    ruby {
      code => "event['index'] = 1"
    }
    xml {
      source => "message"
      target => "parsed"
    }

  split {
    field => "[parsed][suites][suites][suite][suite][cases][cases][case]"
    add_field => {
      test_duration  => "%{[parsed][suites][suites][suite][suite][cases][cases][case][duration]}"
      class_name     => "%{[parsed][suites][suites][suite][suite][cases][cases][case][className]}"
      test_name      => "%{[parsed][suites][suites][suite][suite][cases][cases][case][testName]}"
      skipped        => "%{[parsed][suites][suites][suite][suite][cases][cases][case][skipped]}"
      result         => "%{[parsed][suites][suites][suite][suite][cases][cases][case][errorDetails]}"
    }
  }

  if [result] !~ "Failed" {
    mutate {
      update => {
        "result" => "Success"
      }
    }
  }

    mutate {
      remove_field => ["message", "parsed"]
    }

  #}
}

output {
  elasticsearch {
   hosts => ["localhost:9200"]
   sniffing => true
   manage_template => false
   index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
   document_type => "%{[@metadata][type]}"
  }
  stdout { codec => rubydebug }
}
2

ping : )

3

Wild guess: The split filter doesn't include the @metadata field so the index and document_type options won't get the values you expect?

Take ES out of the equation for now and just use the stdout output you already have (but change the rubydebug codec's option so it shows the contents of @metadata). When things look great there try again with the elasticsearch output enabled.

1 Like
4

This worked like a charm! Thanks. I just hard-coded the index and document_type fields to a static value.