Http_request_failure when trying to get logs with http_poller plugin logstash

Elastic Stack Logstash
1

Hi,
I want to fetch JSON formatted logs from a URL. What is the best approach to do it? I tried the http_poller plugin..but I am not sure if it doesn't work or i am doing something wrong. Can you please help?

I tried using the httppoll plugin in logstash...however,.. i get an http failure message

input {
http_poller {
urls => {
test1 => "http://nimhts23.hm230.org/solr/col_dashv1_shard2_replica2/select?q=%3A&wt=json&indent=true"
}

request_timeout => 60
interval => 60
codec => "json"
}
}

output {
elasticsearch {
hosts => "10.45.31.897"

}

}

I get the following message:

@version:1 @timestamp:August 2nd 2016, 15:29:24.663 tags:httprequest_failure http_request_failure.request.method:get http_request_failure.request.url:http://nimhts23.hm230.org/solr/col_dashv1_shard2_replica2/select?q=%3A&wt=json&indent=true http_request_failure.name:test1 http_request_failure.error:nimhts23.hm230.org: unknown error http_request_failure.backtrace: - http_request_failure.runtime_seconds:0.284 _id:AVZNYEFVvm06ECqo53r0 _type:logs _index:logstash-2016.08.02 _score:

2

I get this error message on stdout:

{
"@version" => "1",
"@timestamp" => "2016-08-03T16:55:40.437Z",
"http_poller_metadata" => {
"name" => "test2",
"host" => "aunmo",
"request" => {
"method" => "get",
"url" => "http://nimhts23.hm230.org/solr/col_dashv1_shard2_replica2/select?q=%3A&wt=json&indent=true""
},
"runtime_seconds" => nil
},
"tags" => [
[0] "_http_request_failure"
],
"http_request_failure" => {
"request" => {
"method" => "get",
"url" => http://nimhts23.hm230.org/solr/col_dashv1_shard2_replica2/select?q=%3A&wt=json&indent=true""
},
"name" => "test2",
"error" => "http://nimhts23.hm230.org: unknown error",
"backtrace" => nil,
"runtime_seconds" => 0.776
}
}

3

Can someone help with this? I am still stuck on the same issue. constantly trying to debug this issue with no luck..

What could be there reason behind this http_request_failure?

4

@warkolm, Sorry about having to tag you on this. But I have been stuck with this for long time and can't figure out where the problem is. If you could provide some insights, i would greatly appreciate it!

5

Try removing the JSON codec in the input and output to stdout with ruby debug codec instead of Elasticsearch to see exactly what the fetched events look like to Logstash. Then add filters and codecs as appropriate.

6

Still same error...

{
"@version" => "1",
"@timestamp" => "2016-08-08T18:08:10.701Z",
"report" => {
"name" => "test2",
"host" => "malibu",
"request" => {
"method" => "get",
"url" => "confidential"
},
"runtime_seconds" => nil
},
"tags" => [
[0] "_http_request_failure"
],
"http_request_failure" => {
"request" => {
"method" => "get",
"url" => "confidential"
},
"name" => "test2",
"error" => "confidential",
"backtrace" => nil,
"runtime_seconds" => 0.003
}
}

I guess the http_poller plugin is broken.

7

Are you able to curl to the specified URL from the server where Logstash is running? What does this return?

8

I did ping "url"

and the response i get is:
ping: unknown host myurl

So i guess it's a problem with the url..not the plugin?

9

Ping is for servers, not URLs, and it will not tell you whether you have connectivity to the appropriate port or not. Are you able to curl the URL quoted above?

10

Oh my bad. Silly me!
I tried the curl command and I am getting "curl: (6) Couldn't resolve host 'hdpttn24.mmc1800.org'