# How to set up input for https curl and header key? **URL:** \ **Category:** Logstash\ **Created:** [July 30, 2018, 5:54pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231 "2018-07-30T17:54:42Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [July 30, 2018, 5:54pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/1 "2018-07-30T17:54:42Z")
I need to invoke a streaming endpoint by executing the following: > curl -X GET "[https://feed.source.com/1.0/json/A9575EEEE9F53398FD237049](https://feed.source.com/1.0/json/A9575EEEE9F53398FD237049)" -H "api\_key:zBA7goakSWiE1A7aHA" I tried this > input { > http { > host =\> "[https://feed.source.com/1.0/json/A9575EEEE9F44635332E9928FD237049](https://feed.source.com/1.0/json/A9575EEEE9F44635332E9928FD237049)" > port =\> 443 > response\_headers =\> "api\_key:zBA7goakSW19w1A7aHA" > } > } The error I got is: > Invalid setting for http input plugin: > input { > https { > # This setting must be a hash > # This field must contain an even number of items, got 1 > response\_header =\> "api\_key:zbas7goasafdlsakjdf" > ... > } > } How would you handle this? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 30, 2018, 6:14pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/2 "2018-07-30T18:14:49Z")
> [@emanresu](#): > > response\_headers =\> "api\_key:zBA7goakSW19w1A7aHA" ``` response_headers => { "api_key" => "zBA7goakSW19w1A7aHA" } ``` ---
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [July 30, 2018, 7:29pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/3 "2018-07-30T19:29:10Z")
Thank you for that Badger; my first error is resolved now. I made the change and now I get a 'name or service not known' error: > Preformatted text Sending Logstash's logs to C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logs which is now configured via log4j2.properties > [2018-07-30T14:28:08,686][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified > [2018-07-30T14:28:09,325][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.3.1"} > [2018-07-30T14:28:13,554][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50} > [2018-07-30T14:28:13,667][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"\"[https://feed.source.com/1.0/json/A9575E946](https://feed.source.com/1.0/json/A9575E946) > 35332E9928FD237049", port=\>443, response\_headers=\>{"api\_key"=\>"zBA7goakSY9w1A7aHA"}, id=\>"a13502ffdee5a1ea8db80433cbaccfcd084dfcccdfa2ebc00dcd448d12", enable\_metric=\>true, codec=\>\ Stash::Codecs::Plain id=\>"plain\_878fd503-b46f-4ba5-86e4-5f7b35627690", enable\_metric=\>true, charset=\>"UTF-8"\>, threads=\>4, ssl=\>false, verify\_mode=\>"none", additional\_codecs=\>{"application/json > "=\>"json"}\>", :error=\>"initialize: name or service not known", :thread=\>"#\"} > [2018-07-30T14:28:13,727][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\, :backtrace=\>["org/jr > uby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/l > ib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add\_tcp\_listener'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/vendor/bundle/jruby/2.3.0/gems/logstash-input-http-3.0.10/lib/l > ogstash/inputs/http.rb:119:in `register'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:340:in `register\_plugin'", "C:/Users/.../Documents/logstas > h-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:351:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/l > ogstash-core/lib/logstash/pipeline.rb:351:in `register_plugins'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:498:in `start\_inputs'", "C:/Users/.../ > Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:392:in `start\_workers'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline. > rb:288:in `run'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=\>"#\"} > [2018-07-30T14:28:13,764][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAct > ion::Create, action\_result: false", :backtrace=\>nil} > [2018-07-30T14:28:14,092][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} When I execute the cURL command in a shell, it works and the response looks like this: > {"TIMESTAMP\_UTC":"2018-07-30 18:30:48.769","RP\_STORY\_ID":"07C09A1D59151060EDABFB98AA2C1CE9","RP\_ENTITY\_ID":"B65303","ENTITY\_TYPE":"COMP","ENTITY\_NAME":"AstraZeneca PLC","COUNTRY\_CODE":"GB","RELEVANCE":3,"EVENT\_SENTIMENT\_SCORE":null,"EVENT\_RELEVANCE":null,"EVENT\_SIMILARITY\_KEY":null,"EVENT\_SIMILARITY\_DAYS":null,"TOPIC":null,"GROUP":null,"TYPE":null,"SUB\_TYPE":null,"PROPERTY":null,"FACT\_LEVEL":null,"RP\_POSITION\_ID":null,"POSITION\_NAME":null,"EVALUATION\_METHOD":null,"MATURITY":null,"EARNINGS\_TYPE":null,"EVENT\_START\_DATE\_UTC":null,"EVENT\_END\_DATE\_UTC":null,"REPORTING\_PERIOD":null,"REPORTING\_START\_DATE\_UTC":null,"REPORTING\_END\_DATE\_UTC":null,"RELATED\_ENTITY":null,"RELATIONSHIP":null,"CATEGORY":null,"EVENT\_TEXT":null,"NEWS\_TYPE":"PRESS-RELEASE","RP\_SOURCE\_ID":"5A5702","SOURCE\_NAME":"Benzinga","CSS":0.02,"NIP":-0.40,"PEQ":0,"BEE":1,"BMQ":1,"BAM":0,"BCA":0,"BER":0,"ANL\_CHG":0,"MCQ":0,"RP\_STORY\_EVENT\_INDEX":10,"RP\_STORY\_EVENT\_COUNT":11,"PRODUCT\_KEY":"RPA","PROVIDER\_ID":"BZG","PROVIDER\_STORY\_ID":"12100958:15513604","HEADLINE":"AMCP Partnership Forum Examines Non-Traditional Payment and Benefit Models for High-Cost Pharmaceuticals"} How would you handle this? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 30, 2018, 7:43pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/4 "2018-07-30T19:43:30Z")
If you do ``` nslookup feed.source.com ``` then what is the result? Do you have a proxy configured in your .curlrc? ---
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [July 31, 2018, 3:10pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/5 "2018-07-31T15:10:51Z")
You bring up a great point. My logstash is running on my Windows PC. I ran the cURL on a Linux server to check that [feed.source.com](http://feed.source.com) is running; and confirmed that it is. To answer your question, I did `> nslookup feed.source.com` gives me ``` Server: my_proxyserver_name.com Address: 167.12.21.22 Non-authoritative answer: Name: feed.source.com Address: 54.87.179.29 ``` I can't ping [feed.source.com](http://feed.source.com) but I can ping its IP address. So I think I need to provide the IP address to logstash, don't I? So to do that I tried the resolve filter after input like this: ``` filter { dns { resolve => ["source_host", "54.87.179.29"] } } ``` But it gives me the same error as before. How would you handle this? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 31, 2018, 3:31pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/6 "2018-07-31T15:31:56Z")
> [@emanresu](#): > > I can't ping [feed.source.com](http://feed.source.com) but I can ping its IP address. Is that referring to the machine where logstash runs? If so, try ``` host => "https://54.87.179.29/1.0/json/A9575EEEE9F44635332E9928FD237049" ``` ---
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [July 31, 2018, 3:41pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/7 "2018-07-31T15:41:09Z")
Thank you for your feedback Badger. Yes, I am referring to the machine where logstash is running. I still get the 'name or service not known' error. Here is my config: ``` input { http { host => "https://54.87.179.29/1.0/json/A9575EEEE9F446928FD237049" port => 443 response_headers => { "api_key" => "zBA7goakSWiEY19w1A7aHA" } } } output { stdout { codec => rubydebug } } ``` What can I try now? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 31, 2018, 4:02pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/8 "2018-07-31T16:02:54Z")
Wait up, notice that exception is getting raised in add\_tcp\_listener. The http input creates an network listener, and you do not have the IP address 54.87.179.29 on your server, so it cannot bind to it. I think you want an http\_poller input. ---
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [July 31, 2018, 7:02pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/9 "2018-07-31T19:02:33Z")
Thank you for pointing this out. I followed your tip and I am making progress. My config is this: ``` input { http_poller { urls => { test1 => { method => get url => "https://54.87.179.29:443/1.0/json/A9575EEEE9F44635332E7049" headers => { api_key => "zBA7goakSWiEY19w1A7aHA" } } } request_timeout => 60 schedule => { cron => "* * * * * UTC"} codec => "json" metadata_target => "http_poller_metadata" } } output { stdout { codec => rubydebug } } ``` My screen shows updates every 60 seconds with the following that shows an 'http\_request\_failure ``` ... "error"←[0;37m => ←[0m←[0;33m"Host name '54.87.179.29' does not match the certificate subject provided by the peer (OU=COMODO EV Multi-Domain SSL, O=RAVENPACK INTERNATIONAL SL, STREE T=URBANIZACION VILLA PARRA (CRTA. CADIZ KM. 176, L=Marbella, ST=Malaga, OID.2.5.4.17=29602, C=ES, OID.2.5.4.15=Private Organization, OID.1.3.6.1.4.1.311.60.2.1.3=ES, SERIALNUMBER=B92439181)"←[0m, ... ``` For format I am referring to: [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http\_poller.html](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html). What can I try now? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 31, 2018, 7:07pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/10 "2018-07-31T19:07:11Z")
> [@emanresu](#): > > Host name '54.87.179.29' does not match the certificate subject provided by the peer OK, try going back to a domain name instead of an IP address. ---
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [July 31, 2018, 9:03pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/11 "2018-07-31T21:03:00Z")
Going back to domain name instead of IP address didn't throw any errors but just sat there. I removed the port after the domain name which caused 'could not read from stream : Read time out ' error. Then, I tried ``` input { http_poller { urls => { rv_url => { method => get url => "https://feed.source.com:443/1.0/json/A9575EEEE9F44635332E9928FD237049/server-status?auto" headers => { api_key => "zBA7goakSWiEY19w1A7aHA" } } } # request_timeout => 60 schedule => { cron => "* * * * * UTC"} codec => "json" # metadata_target => "http_poller_metadata" } } output { stdout { codec => rubydebug } } ``` from: [https://www.elastic.co/blog/introducing-logstash-http-poller](https://www.elastic.co/blog/introducing-logstash-http-poller), which gave me this error: ``` { "@version"←[0;37m => ←[0m←[0;33m"1"←[0m, "@timestamp"←[0;37m => ←[0m2018-07-31T20:56:00.240Z, "message"←[0;37m => ←[0m←[0;33m"Not authorized"←[0m, "tags"←[0;37m => ←[0m[ ←[1;37m[0] ←[0m←[0;33m"_jsonparsefailure"←[0m ] } ``` What can I try now? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 31, 2018, 9:12pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/12 "2018-07-31T21:12:44Z")
I don't know what to suggest. It's not really a logstash question at this point. The http\_poller is connecting and issuing a request and getting back a response. Not the response you want, but in terms of the logstash configuration it is working as expected. ---
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [August 1, 2018, 1:39pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/13 "2018-08-01T13:39:48Z")
I found that I can run cURL like this: ``` input { exec { command => "curl -X GET \"https://feed.source.com/1.0/json/A9575EEEE9F49\" -H \"api_key:zBA7gHA\"" interval => 600000000 } } ``` But how can I handle the quotes? The above gives this error: 'is not recognized as an internal or external command, program, or batch file.' I tried single quotes on the outside which gave the same error. If I don't escape the inner double quotes, I get a configError with line and position of curl command. What can I try next? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 1, 2018, 1:44pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/14 "2018-08-01T13:44:47Z")
``` command => 'curl -X GET "https://feed.source.com/1.0/json/A9575EEEE9F49" -H "api_key:zBA7gHA"' ``` should work. ---
**Author:** ![emanresu](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Post date:** [August 2, 2018, 3:45pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/15 "2018-08-02T15:45:05Z")
It works now! Thank you Badger! My output is: ``` output { stdout { codec => rubydebug } } ``` What I see is > [2018-08-02T15:41:15,820][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"} > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0[2018-08-02T15:41:15,919][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} > [2018-08-02T15:41:16,183][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} > 100 17625 0 17625 0 0 623 0 --:--:-- 0:00:28 --:--:-- 887 I can't see the data on my PC's dos shell, until I hit Ctrl^C. Is this normal? How can I see the output streaming? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 2, 2018, 4:34pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/16 "2018-08-02T16:34:24Z")
> [@emanresu](#): > > Is this normal? I do not think so. If I run a configuration like ``` input { exec { interval => 10 command => 'echo foo' } } output { stdout { codec => rubydebug } } ``` I would expect to get this every 10 seconds ``` { "command" => "echo foo", "@version" => "1", "@timestamp" => 2018-08-02T16:33:33.716Z, "host" => "...", "message" => "foo\n" } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2018, 4:34pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231/17 "2018-08-30T16:34:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.