Issue while parsing xml data from a file

Elastic Stack Logstash
1

I have xml data in a file and I am trying to parse it as a single event and visualise it in kibana
below is the Configuration file for logstash.
input {
file {
path => "/Users/varsaraf/outxr.xml"
start_position => "beginning"
sincedb_path => "/tmp/sincedbfile"
exclude => "*.gz"
type => "xml"
codec => multiline {
pattern => "<operation-data"
negate => "true"
what => "previous"
}
}
}

filter {
    xml {
        source => "message"
        store_xml => false
        target => "doc"
        xpath => [
            "/operation-data/operations/operation/operation-id/text()", "oper-id",
            "/operation-data/operations/operation/operation-time/text()", "oper-time",
            "/operation-data/operations/operation/statistics/specific-stats/icmp-path-jitter-stats/source-address/text()", "source-ip",
            "/operation-data/operations/operation/statistics/specific-stats/icmp-path-jitter-stats/dest-address/text()", "dest-ip",
            "/operation-data/operations/operation/statistics/specific-stats/icmp-path-jitter-stats/hop-address/text()", "hop-id",
            "/operation-data/operations/operation/statistics/specific-stats/icmp-path-jitter-stats/packet-interval/text()", "pkt-interval",
            "/operation-data/operations/operation/statistics/specific-stats/icmp-path-jitter-stats/response-time-count/text()", "RTC",
            "/operation-data/operations/operation/statistics/specific-stats/icmp-path-jitter-stats/packet-count/text()", "pkts",
            "/operation-data/operations/operation/statistics/specific-stats/icmp-path-jitter-stats/packet-loss-count/text()", "pkt-loss"
        ]
    }
}

output {
    elasticsearch {
        codec => json
        hosts => ["localhost:9200"]
        index => "test1"
    }
    stdout {
        codec => rubydebug
    }

I am able to create an index but the fields have data type as string I want them to be long or integer. how do I do it?

2

You could use mutate+convert. You will need to create a new index to see the type change.

3

I get the following output on running the new config file with the corrections of adding mutate and convert. the output is attached as Screenshot-

Screenshot 2020-03-27 at 3.42.14 PM
Screenshot 2020-03-27 at 3.42.14 PM3584×1520 1.75 MB

my config file is -

    input {
       file {
    path => "/Users/varsaraf/out.xml"        start_position => "beginning"
    sincedb_path => "/tmp/mysincedbfilez"
    exclude => "*.gz"
    type => "xml"
    codec => multiline {
        pattern => "<sla-path-jitter-stats>"
        negate => "true"
        what => "previous"
    }
}} filter {
xml {
    source => "message"
    store_xml => false
    target => "sla-path-jitter-stats"
    xpath => [
        "/sla-path-jitter-stats/oper-id/text()", "id",
        "/sla-path-jitter-stats/hop-address/text()", "ip_address",
        "/sla-path-jitter-stats/latest-start-time/text()", "start-time",
        "/sla-path-jitter-stats/latest-rtt-stats/sum-of-rtt/text()", "rtt-sum",
        "/sla-path-jitter-stats/latest-packet-loss-stats/timeouts/text()", "timeout",
        "/sla-path-jitter-stats/latest-packet-loss-stats/packet-loss-count/text()", "packet-loss"
    ]} mutate {convert => {"id" => "long"  "rtt-sum" => "long"  "timeout" => "long"   "packet-loss"=>"long"

} }}
output {
elasticsearch {
codec => json
hosts => ["localhost:9200"]
index => "ipslaxe"
}
stdout {
codec => rubydebug
}}

it doesn't create an index or send the data to Elasticsearch. while previously the same file without mutate and convert was sending the data.

4

"long" is not a supported type. Read the documentation.

5

I have changed the config file still it shows no output. attaching the config file below.

input {
    file {
        path => "/Users/varsaraf/out.xml"
        start_position => "beginning"
        sincedb_path => "/tmp/mysincedbfilez"
        exclude => "*.gz"
        type => "xml"
        codec => multiline {
            pattern => "<sla-path-jitter-stats>"
            negate => "true"
            what => "previous"
        }
    }
}

filter {
    xml {
        source => "message"
        store_xml => false
        target => "sla-path-jitter-stats"
        xpath => [
            "/sla-path-jitter-stats/oper-id/text()", "id",
            "/sla-path-jitter-stats/hop-address/text()", "ip_address",
            "/sla-path-jitter-stats/latest-start-time/text()", "start-time",
            "/sla-path-jitter-stats/latest-rtt-stats/sum-of-rtt/text()", "rtt-sum",
            "/sla-path-jitter-stats/latest-packet-loss-stats/timeouts/text()", "timeout",
            "/sla-path-jitter-stats/latest-packet-loss-stats/packet-loss-count/text()", "packet-loss"
        ]
    }
}
filter{
 mutate {
 convert => {
   "id" => "integer"
   "rtt-sum" => "integer"
   "timeout" => "integer"
   "packet-loss" => "integer"
   }
 }
}
output {
    elasticsearch {
        codec => json
        hosts => ["localhost:9200"]
        index => "ipslaxe"
    } }
    stdout {
        codec => rubydebug
    }
}

what is the error ?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.