Http_poller input plugin error

Elastic Stack Logstash
1

Hello Team,

Kindly help me with below error when i try to pull data from a URL in csv format and store it in a csv file output. I have been trying googling around but could not get any answer. last hope is elastic community. kindly let me know what error I am doing with my configuration.

below is the error

[2019-04-02T19:44:09,476][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.0"}
[2019-04-02T19:44:21,765][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-02T19:44:22,070][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"every"=>"2s"}, :timeout=>nil}
[2019-04-02T19:44:22,125][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::HTTP_Poller schedule=>{\"every\"=>\"2s\"}, urls=>{\"minemeld\"=>\"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator|threatIP&f=confidence&f=sources|feeds\", \"codec\"=>\"line\"}, id=>\"b2c19b71025ef17abe0e54722f8bdb435e1a5296d1b6da6c1007e741a1fb431f\", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>\"json_4a209d70-7ae1-47ef-90f5-cf5951a5a4c6\", enable_metric=>true, charset=>\"UTF-8\">, request_timeout=>60, socket_timeout=>10, connect_timeout=>10, follow_redirects=>true, pool_max=>50, pool_max_per_route=>25, keepalive=>true, automatic_retries=>1, retry_non_idempotent=>false, validate_after_inactivity=>200, keystore_type=>\"JKS\", truststore_type=>\"JKS\", cookies=>true, metadata_target=>\"@metadata\">", :error=>"Invalid URL http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator|threatIP&f=confidence&f=sources|feeds", :thread=>"#<Thread:0x5e7c605a run>"}
[2019-04-02T19:44:23,987][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid URL http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator|threatIP&f=confidence&f=sources|feeds>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:105:in `validate_request!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:97:in `normalize_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:57:in `block in setup_requests!'", "org/jruby/RubyHash.java:1419:in `each'", "org/jruby/RubyEnumerable.java:833:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:57:in `setup_requests!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-4.0.5/lib/logstash/inputs/http_poller.rb:47:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:413:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176:in `block in start'"], :thread=>"#<Thread:0x5e7c605a run>"}
[2019-04-02T19:44:24,039][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2019-04-02T19:44:24,690][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Below is my config file

input {
  http_poller {
    schedule => { "every" => "2s" }
    urls => {
    minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator|threatIP&f=confidence&f=sources|feeds"
    codec => "line"
  }
 }
}
filter {
   csv {
     separator => ","
     columns => ["threatIP","confidence","feeds"]
  }
}
output {
 csv {
  fields => ["threatIP","confidence","feeds"]
  path => "/tmp/darpintel.csv"
 }
}
2

codec => "line" is not a valid URL specification. Also, you should change | to %7C in minemeld.

3

Hello @Badger Thank you . I removed Codec and i was able to get the output file successfully. but however the file is empty. it should be filled with IPs. Could you please help me what i am missing in collecting the data.

and what is %7C ? I am not aware of this code/plugin ? where and how should i use it ?

4

It is URL encoding.

5

Hello @Badger
should I have my URL with %7C ? like below ?
http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv%7C

I am not familiar with URL encoding. your detailed answer would be really useful. Thanks !

6

That should be

minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator%7CthreatIP&f=confidence&f=sources%7Cfeeds"
7

I got the URL encoding. Below is my URL encoded pipe with %7C. ILogstash provides output file, but the file is without data and it has just the seperator ","

minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator%7CthreatIP&f=confidence&f=sources%7Cfeeds"

Kindly help me on why I am not getting data in my output file. the same URL in browser provides data.

8

In a browser, use a debugging proxy like Fiddler. See what the request headers look like and try adding them to your http_poller configuration.

9

Hello @Badger - Thanks for the suggestion. Below is the request header after debugging the URL using fiddler.

image
image.png959×631 132 KB

Below is my current Input pipeline. What should I add in my input pipeline ?

input {
  http_poller {
    schedule => { "every" => "2s" }
    urls => {
    minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator%7CthreatIP&f=confidence&f=sources%7Cfeeds"
  }
 }
}
10

Hello @Badger - I just changed the URL to json format and tried saving the output in file with .json extention. logstash is running successfully without errors but the output file contains error messages indicating to certificate. pleas check below messages that was inside the output file and kindly guide me with fixing certificate issued with such URL. the URL is output of minemeld IOC.

{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.9196650000000001,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:14.260Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.04099,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:15.387Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.021399,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:17.536Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.020947,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:19.661Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.030288,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:21.789Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.014396,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:23.904Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.018939,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:26.038Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.01104,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:28.157Z"}

below is the logstash conf file .

input {
  http_poller {
    schedule => { "every" => "2s" }
    urls => {
    minemeld => {
    method => GET
    url => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json"
   }
  }
 }
}
output {
 file {
  path => "/tmp/darpintel.json"
 }
11

@Badger - I generated a self signed certification for minemeld using below commands.

$ wget https://gist.githubusercontent.com/jtschichold/f0977e5c1ec09b3ec7d66bf80687d9da/raw/7ec994a3a731637ffa335365adddddbfd92004f2/generate-certificate.sh
$ chmod a+x generate-certificate.sh
$ sudo ./generate-certificate.sh <minemeld hostname>

Now I have CA.crt created . when i tried converting the self signed certificate into valid jks format as per http_poller document, I get error as keytool error: java.lang.Exception: Input not an X.509 certificate

I used below one liner as per the document

openssl s_client -showcerts -connect http://192.168.56.30/feeds/DARP-output-feedgreen </dev/null 2>/dev/null|openssl x509 -outform PEM > downloaded_cert.pem; keytool -import -alias test -file downloaded_cert.pem -keystore downloaded_truststore.jks

Kindly guide me how to fix the cert issue. I get below message in my json output file

{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&amp;v=json","method":"get"},"runtime_seconds":0.9196650000000001,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:14.260Z"}

12

That is http, not https. There should not be any need for certificates unless it is redirecting.

13

I think it is redirecting to https. the output file is having message stating
unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&amp;amp;v=json

14

If it is redirecting to https then you may need either a truststore or the cacert option on the input. Can you find what it redirects to and use that URL instead?

15

Please see below snippet of the URL in browser.

image
image975×347 38.2 KB

not sure if it is taking https or http . I created a self signed cert , used CAcert in the http_poller input pipeline but same result.
Also i tried to create a truststore using the one liner in document, but i get error as keytool error: java.lang.Exception: Input not an X.509 certificate

I used below onliner command as per document for converting self signed cert to jks extention.

`

openssl s_client -showcerts -connect https://192.168.56.30/feeds/DARP-output-feedgreen </dev/null 2>/dev/null|openssl x509 -outform PEM > downloaded_cert.pem; keytool -import -alias test -file downloaded_cert.pem -keystore downloaded_truststore.jks

`

16

Clearly in the browser it has redirected to https. So change your http_poller input to use https. Then ask a new question about how to configure SSL on an http_poller input.

17

Thanks @Badger - I changed my http_poller input with https and also used CAcert path . stil I get http_request failure error.

"http_request_failure":{"runtime_seconds":0.015593000000000001,"name":"minemeld","request":{"cacert":"/opt/CA.cert","method":"get","url":"https://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json"},"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","backtrace":null},"@version":"1","@timestamp":"2019-04-03T14:36:30.591Z","tags":["_http_request_failure"]}

my conf file

input {
http_poller {
schedule => { "every" => "2s" }
urls => {
minemeld => {
method => GET
url => "https://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json"
cacert => "/opt/CA.cert"
}
}
}
}
output {
file {
path => "/tmp/darpintel.json"
}
}

kindly let me know if i need to create new discussion on certificate error. the original query created initially was addressed . Thank you. :slight_smile: now encountering with this cert error

18

Yes, start a new thread.

19

Thanks and appreciate your effort in narrowing down the issue.

Regards,
Haran

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.