Use API field as a variable in logstash input

[AurelienG]

Hello,

What I am trying to do seems quite similar to what @dorj1234 was asking help for a few months ago in this topic but the answer provided is not enough for me so I hope you won't mind me opening a similar topic.

What I would like to get is information about Jenkins builds configuration. From the API, I can have some information about all builds, but if I want to know more about a specific build, I have to add its name in the URL. So my idea is to make a first API request with logstash, maybe store the builds name somewhere (file? local variable?), then use logstash again for each build.

I use exec plugin to curl the Jenkins API and then I would like to GET jenkins/job/name-of-job/config.xml. Is there any way I could do that?

Thanks for the help

[Badger]

Well the answer is much the same. You can write a ruby filter, which would perhaps use Net::HTTP to do the second GET. Something like this, which uses the whole of message as a filename, and puts the response into somefield.

    ruby {
        code => '
            require "net/http"
            uri = URI( "http://127.0.0.1/geojson/" + event.get("message") )
            response = Net::HTTP.get(uri)
            event.set("somefield", response)
        '
    }

Error handling left as an exercise for the reader.

[AurelienG]

Ok thanks, I'll try that! I must admit I'm not really familiar with ruby, so thank you for giving the whole syntax

Edit: I looked a little further and came across the Nokogiri gem, which seems to be better to get specific elements of the XML file. What do you think?

Edit 2: I have tried a few things but I can't fix the issue I have. The address I'm trying to get the XML file from is HTTPS and I always get this error:
Ruby exception occurred: Non-HTTP proxy URI: proxy.********.fr:****

Here's my ruby filter:

ruby {
    code => '
      require "nokogiri"
      require "open-uri"
      config = URI.parse(event.get("[jobs][url]") + "config.xml").read
      jdk_version = config.xpath("//jdk")
      event.set("jdk_version", jdk_version)
    '
  }

[Badger]

Logstash has an xml filter, which is a wrapper around nokogiri.

[AurelienG]

I guess I shouldn't have tried to do everything within the Ruby filter...
I used your code to get all the XML in a field and now I just have to use the XML filter to keep what I need.

I think I should be able to do it by myself now, thanks a lot!

Edit: It's working perfectly, thanks again

[AurelienG]

Actually, I still have a problem...
I need to authenticate to access the config.xml file but I can't figure out how to do it with the https request. I have seen lots of "solutions" but only for http requests. Obviously, I don't want to use username and password, but rather the private token. I have tried to put it in the header but can't make it work.
Any idea for this?

Here's my code:

	ruby {
		# Get the config.xml for each job
		code => '
			require "net/https"
			url = event.get("[jobs][url]") + "config.xml"
			uri = URI(url)

			Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
				request = Net::HTTP::Get.new(uri)
				request["authorization"] = "Token token=abc123"
				response = http.request(request)
				event.set("config", response.body)
			end
		'
	}

I used the other syntax for HTTPS from the NET::HTTP class because it seemed to handle the header, but it still doesn't work.