Trying to fetch values from xml and feed into influxdb

[Niranjan_kaup]

Sample xml file:

Tove Jani Reminder Don't forget me this weekend!

Below is the configuration:

input {

file {
path => "/mnt/users.xml"
start_position => "beginning"
type => xml
}
}
filter {

xml {
    source => "message"
    store_xml => "false"
    xpath => [
        "/note/to/text()","id"
    ]
}

}
output {
stdout {}
}

and below is the output which I am getting curently

2017-10-06T09:52:07.006Z localhost.localdomain >
2017-10-06T09:52:19.024Z localhost.localdomain
2017-10-06T09:52:19.025Z localhost.localdomain Tove
2017-10-06T09:52:19.026Z localhost.localdomain Jani
2017-10-06T09:52:19.029Z localhost.localdomain Reminder
2017-10-06T09:52:19.030Z localhost.localdomain Don't forget me this weekend!
2017-10-06T09:52:19.030Z localhost.localdomain

But, I expect it should be in "id" -> "Tove" is the result so that I can feed into influxdb.

[magnusbaeck (Magnus Bäck)]

As you can see from your post it doesn't contain a shred of XML. Format XML snippets as preformatted text to avoid the de-HTMLificiation performed by the posting software.

The file input is line-oriented so if you want to read a file into a single event you'll have to use the multiline codec. Examples of this has been posted in the past.

[Niranjan_kaup]

Thanks for your reply:

I got that option and seems working. But, one issue is when it is getting parsed it stores null value at variable. Below is the configuration and codebug output:

input {

file {
path => "/mnt/users.xml"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "<repositories"
negate => "true"
what => "previous"
auto_flush_interval => 1
}
}
}
filter {

xml {
    source => "message"
    store_xml => "true"
    target => "influxdb"
    force_array => false
    xpath => [
        "/repositories-item/resourceURI/text()","resourceURI"
    ]
}

}

output {
influxdb {
host => localhost
retention_policy => autogen

use_event_fields_for_data_points => true

data_points => {
"resourceURI" => "%{resourceURI}"
}
}
stdout { codec => rubydebug }
#stdout {}
}

Output:

{
"path" => "/mnt/users.xml",
"@timestamp" => 2017-10-10T05:44:10.913Z,
"@version" => "1",
"host" => "localhost.localdomain",
"resourceURI" => [
[0] "http://localhost:8081/nexus/service/local/repositories/thirdparty"
],
"message" => " \n http://localhost:8081/nexus/service/local/repositories/thirdparty\n http://localhost:8081/nexus/content/repositories/thirdparty\n thirdparty\n 3rd party\n hosted\n RELEASE\n maven2\n org.sonatype.nexus.proxy.repository.Repository\n maven2\n true\n true\n file:/opt/sonatype-work/nexus/storage/thirdparty/\n ",
"influxdb" => {
"effectiveLocalStorageUrl" => "file:/opt/sonatype-work/nexus/storage/thirdparty/",
"providerRole" => "org.sonatype.nexus.proxy.repository.Repository",
"format" => "maven2",
"resourceURI" => "http://localhost:8081/nexus/service/local/repositories/thirdparty",
"repoType" => "hosted",
"contentResourceURI" => "http://localhost:8081/nexus/content/repositories/thirdparty",
"provider" => "maven2",
"name" => "3rd party",
"exposed" => "true",
"repoPolicy" => "RELEASE",
"id" => "thirdparty",
"userManaged" => "true"
},
"tags" => [
[0] "multiline"
]
}
{
"@version" => "1",
"host" => "localhost.localdomain",
"path" => "/mnt/users.xml",
"@timestamp" => 2017-10-10T05:44:10.802Z,
"message" => "\n ",
"tags" => [
[0] "multiline",
[1] "_xmlparsefailure"
]
}
{
"path" => "/mnt/users.xml",
"@timestamp" => 2017-10-10T05:44:10.807Z,
"@version" => "1",
"host" => "localhost.localdomain",
"resourceURI" => [
[0] "http://localhost:8081/nexus/service/local/repositories/snapshots"
],
"message" => " \n http://localhost:8081/nexus/service/local/repositories/snapshots\n http://localhost:8081/nexus/content/repositories/snapshots\n snapshots\n Snapshots\n hosted\n SNAPSHOT\n maven2\n org.sonatype.nexus.proxy.repository.Repository\n maven2\n true\n true\n file:/opt/sonatype-work/nexus/storage/snapshots/\n ",
"influxdb" => {
"effectiveLocalStorageUrl" => "file:/opt/sonatype-work/nexus/storage/snapshots/",
"providerRole" => "org.sonatype.nexus.proxy.repository.Repository",
"format" => "maven2",
"resourceURI" => "http://localhost:8081/nexus/service/local/repositories/snapshots",
"repoType" => "hosted",
"contentResourceURI" => "http://localhost:8081/nexus/content/repositories/snapshots",
"provider" => "maven2",
"name" => "Snapshots",
"exposed" => "true",
"repoPolicy" => "SNAPSHOT",
"id" => "snapshots",
"userManaged" => "true"
},
"tags" => [
[0] "multiline"
]
}

[Niranjan_kaup]

How to make sure null values will be ignored when we are sending to influxdb ?