, action_result: false", :backtrace=>nil}
[2019-04-02T19:44:24,690][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
```
**Below is my config file**
```
input {
http_poller {
schedule => { "every" => "2s" }
urls => {
minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator|threatIP&f=confidence&f=sources|feeds"
codec => "line"
}
}
}
filter {
csv {
separator => ","
columns => ["threatIP","confidence","feeds"]
}
}
output {
csv {
fields => ["threatIP","confidence","feeds"]
path => "/tmp/darpintel.csv"
}
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 2, 2019, 1:06pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/2 "2019-04-02T13:06:20Z")
codec =\> "line" is not a valid URL specification. Also, you should change | to %7C in minemeld.
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 2, 2019, 1:09pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/3 "2019-04-02T13:09:40Z")
Hello @Badger Thank you . I removed Codec and i was able to get the output file successfully. but however the file is empty. it should be filled with IPs. Could you please help me what i am missing in collecting the data.
and what is %7C ? I am not aware of this code/plugin ? where and how should i use it ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 2, 2019, 1:11pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/4 "2019-04-02T13:11:49Z")
> [@HaranKumar](#):
>
> and what is %7C ?
It is [URL encoding](https://en.wikipedia.org/wiki/Percent-encoding).
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 2, 2019, 2:29pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/5 "2019-04-02T14:29:16Z")
Hello @Badger
should I have my URL with %7C ? like below ?
`http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv%7C`
I am not familiar with URL encoding. your detailed answer would be really useful. Thanks !
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 2, 2019, 2:45pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/6 "2019-04-02T14:45:51Z")
> [@HaranKumar](#):
>
> minemeld =\> "[http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator|threatIP&f=confidence&f=sources|feeds](http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator%7CthreatIP&f=confidence&f=sources%7Cfeeds)"
That should be
```
minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator%7CthreatIP&f=confidence&f=sources%7Cfeeds"
```
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 2, 2019, 2:46pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/7 "2019-04-02T14:46:58Z")
I got the URL encoding. Below is my URL encoded pipe with %7C. ILogstash provides output file, but the file is without data and it has just the seperator ","
`minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator%7CthreatIP&f=confidence&f=sources%7Cfeeds"`
Kindly help me on why I am not getting data in my output file. the same URL in browser provides data.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 2, 2019, 3:26pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/8 "2019-04-02T15:26:28Z")
In a browser, use a debugging proxy like Fiddler. See what the request headers look like and try adding them to your http\_poller configuration.
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 3, 2019, 1:50am UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/9 "2019-04-03T01:50:49Z")
Hello @Badger - Thanks for the suggestion. Below is the request header after debugging the URL using fiddler.
Below is my current Input pipeline. What should I add in my input pipeline ?
```
input {
http_poller {
schedule => { "every" => "2s" }
urls => {
minemeld => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=csv&f=indicator%7CthreatIP&f=confidence&f=sources%7Cfeeds"
}
}
}
```
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 3, 2019, 9:48am UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/10 "2019-04-03T09:48:45Z")
Hello @Badger - I just changed the URL to json format and tried saving the output in file with .json extention. logstash is running successfully without errors but the output file contains error messages indicating to certificate. pleas check below messages that was inside the output file and kindly guide me with fixing certificate issued with such URL. the URL is output of minemeld IOC.
```
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.9196650000000001,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:14.260Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.04099,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:15.387Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.021399,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:17.536Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.020947,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:19.661Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.030288,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:21.789Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.014396,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:23.904Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.018939,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:26.038Z"}
{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.01104,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:28.157Z"}
```
below is the logstash conf file .
```
input {
http_poller {
schedule => { "every" => "2s" }
urls => {
minemeld => {
method => GET
url => "http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json"
}
}
}
}
output {
file {
path => "/tmp/darpintel.json"
}
```
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 3, 2019, 11:12am UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/11 "2019-04-03T11:12:55Z")
@Badger - I generated a self signed certification for minemeld using below commands.
```auto
$ wget https://gist.githubusercontent.com/jtschichold/f0977e5c1ec09b3ec7d66bf80687d9da/raw/7ec994a3a731637ffa335365adddddbfd92004f2/generate-certificate.sh
$ chmod a+x generate-certificate.sh
$ sudo ./generate-certificate.sh
```
Now I have CA.crt created . when i tried converting the self signed certificate into valid jks format as per http\_poller document, I get error as **keytool error: java.lang.Exception: Input not an X.509 certificate**
I used below one liner as per the document
`openssl s_client -showcerts -connect http://192.168.56.30/feeds/DARP-output-feedgreen /dev/null|openssl x509 -outform PEM > downloaded_cert.pem; keytool -import -alias test -file downloaded_cert.pem -keystore downloaded_truststore.jks`
Kindly guide me how to fix the cert issue. I get below message in my json output file
`{"@version":"1","tags":["_http_request_failure"],"http_request_failure":{"backtrace":null,"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json","method":"get"},"runtime_seconds":0.9196650000000001,"name":"minemeld"},"@timestamp":"2019-04-03T09:41:14.260Z"}`
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 3, 2019, 1:59pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/12 "2019-04-03T13:59:21Z")
> [@HaranKumar](#):
>
> [http://192.168.56.30/](http://192.168.56.30/)
That is http, not https. There should not be any need for certificates unless it is redirecting.
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 3, 2019, 2:27pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/13 "2019-04-03T14:27:36Z")
I think it is redirecting to https. the output file is having message stating
`unable to find valid certification path to requested target","request":{"url":"http://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json`
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 3, 2019, 2:33pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/14 "2019-04-03T14:33:14Z")
If it is redirecting to https then you may need either a truststore or the cacert option on the input. Can you find what it redirects to and use that URL instead?
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 3, 2019, 2:44pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/15 "2019-04-03T14:44:10Z")
Please see below snippet of the URL in browser.
not sure if it is taking https or http . I created a self signed cert , used CAcert in the http\_poller input pipeline but same result.
Also i tried to create a truststore using the one liner in document, but i get error as **keytool error: java.lang.Exception: Input not an X.509 certificate**
I used below onliner command as per document for converting self signed cert to jks extention.
`
> openssl s\_client -showcerts -connect [https://192.168.56.30/feeds/DARP-output-feedgreen](https://192.168.56.30/feeds/DARP-output-feedgreen) \/dev/null|openssl x509 -outform PEM \> downloaded\_cert.pem; keytool -import -alias test -file downloaded\_cert.pem -keystore downloaded\_truststore.jks
`
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 3, 2019, 2:46pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/16 "2019-04-03T14:46:18Z")
Clearly in the browser it has redirected to https. So change your http\_poller input to use https. Then ask a new question about how to configure SSL on an http\_poller input.
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 3, 2019, 2:50pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/17 "2019-04-03T14:50:01Z")
Thanks @Badger - I changed my http\_poller input with https and also used CAcert path . stil I get http\_request failure error.
`"http_request_failure":{"runtime_seconds":0.015593000000000001,"name":"minemeld","request":{"cacert":"/opt/CA.cert","method":"get","url":"https://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json"},"error":"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target","backtrace":null},"@version":"1","@timestamp":"2019-04-03T14:36:30.591Z","tags":["_http_request_failure"]}`
my conf file
> input {
> http\_poller {
> schedule =\> { "every" =\> "2s" }
> urls =\> {
> minemeld =\> {
> method =\> GET
> url =\> "[https://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json](https://192.168.56.30/feeds/DARP-output-feedgreen?tr=1&v=json)"
> cacert =\> "/opt/CA.cert"
> }
> }
> }
> }
> output {
> file {
> path =\> "/tmp/darpintel.json"
> }
> }
kindly let me know if i need to create new discussion on certificate error. the original query created initially was addressed . Thank you. 🙂 now encountering with this cert error
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 3, 2019, 2:56pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/18 "2019-04-03T14:56:43Z")
Yes, start a new thread.
---
### Author:  [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)
#### Post date: [April 3, 2019, 2:57pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/19 "2019-04-03T14:57:56Z")
Thanks and appreciate your effort in narrowing down the issue.
Regards,
Haran
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 1, 2019, 2:57pm UTC](https://discuss.elastic.co/t/http-poller-input-plugin-error/174979/20 "2019-05-01T14:57:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.