How to set up input for https curl and header key?

Elastic Stack Logstash
1

I need to invoke a streaming endpoint by executing the following:

curl -X GET "https://feed.source.com/1.0/json/A9575EEEE9F53398FD237049" -H "api_key:zBA7goakSWiE1A7aHA"

I tried this

input {
http {
host => "https://feed.source.com/1.0/json/A9575EEEE9F44635332E9928FD237049"
port => 443
response_headers => "api_key:zBA7goakSW19w1A7aHA"
}
}

The error I got is:

Invalid setting for http input plugin:
input {
https {
# This setting must be a hash
# This field must contain an even number of items, got 1
response_header => "api_key:zbas7goasafdlsakjdf"
...
}
}

How would you handle this?

2 
response_headers => { "api_key" => "zBA7goakSW19w1A7aHA" }
3

Thank you for that Badger; my first error is resolved now. I made the change and now I get a 'name or service not known' error:

Preformatted text Sending Logstash's logs to C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logs which is now configured via log4j2.properties
[2018-07-30T14:28:08,686][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-30T14:28:09,325][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.1"}
[2018-07-30T14:28:13,554][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-07-30T14:28:13,667][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Http host=>"https://feed.source.com/1.0/json/A9575E946
35332E9928FD237049", port=>443, response_headers=>{"api_key"=>"zBA7goakSY9w1A7aHA"}, id=>"a13502ffdee5a1ea8db80433cbaccfcd084dfcccdfa2ebc00dcd448d12", enable_metric=>true, codec=><Log
Stash::Codecs::Plain id=>"plain_878fd503-b46f-4ba5-86e4-5f7b35627690", enable_metric=>true, charset=>"UTF-8">, threads=>4, ssl=>false, verify_mode=>"none", additional_codecs=>{"application/json
"=>"json"}>", :error=>"initialize: name or service not known", :thread=>"#<Thread:0x5cbf7850 run>"}
[2018-07-30T14:28:13,727][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<SocketError: initialize: name or service not known>, :backtrace=>["org/jr
uby/ext/socket/RubyTCPServer.java:124:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/l
ib/puma/binder.rb:234:in `add_tcp_listener'", "(eval):2:in `add_tcp_listener'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/vendor/bundle/jruby/2.3.0/gems/logstash-input-http-3.0.10/lib/l
ogstash/inputs/http.rb:119:in `register'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:340:in `register_plugin'", "C:/Users/.../Documents/logstas
h-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:351:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/l
ogstash-core/lib/logstash/pipeline.rb:351:in `register_plugins'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:498:in `start_inputs'", "C:/Users/.../
Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:392:in `start_workers'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.
rb:288:in `run'", "C:/Users/.../Documents/logstash-6.3.1/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=>"#<Thread:0x5cbf7850 run>"}
[2018-07-30T14:28:13,764][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAct
ion::Create, action_result: false", :backtrace=>nil}
[2018-07-30T14:28:14,092][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

When I execute the cURL command in a shell, it works and the response looks like this:

{"TIMESTAMP_UTC":"2018-07-30 18:30:48.769","RP_STORY_ID":"07C09A1D59151060EDABFB98AA2C1CE9","RP_ENTITY_ID":"B65303","ENTITY_TYPE":"COMP","ENTITY_NAME":"AstraZeneca PLC","COUNTRY_CODE":"GB","RELEVANCE":3,"EVENT_SENTIMENT_SCORE":null,"EVENT_RELEVANCE":null,"EVENT_SIMILARITY_KEY":null,"EVENT_SIMILARITY_DAYS":null,"TOPIC":null,"GROUP":null,"TYPE":null,"SUB_TYPE":null,"PROPERTY":null,"FACT_LEVEL":null,"RP_POSITION_ID":null,"POSITION_NAME":null,"EVALUATION_METHOD":null,"MATURITY":null,"EARNINGS_TYPE":null,"EVENT_START_DATE_UTC":null,"EVENT_END_DATE_UTC":null,"REPORTING_PERIOD":null,"REPORTING_START_DATE_UTC":null,"REPORTING_END_DATE_UTC":null,"RELATED_ENTITY":null,"RELATIONSHIP":null,"CATEGORY":null,"EVENT_TEXT":null,"NEWS_TYPE":"PRESS-RELEASE","RP_SOURCE_ID":"5A5702","SOURCE_NAME":"Benzinga","CSS":0.02,"NIP":-0.40,"PEQ":0,"BEE":1,"BMQ":1,"BAM":0,"BCA":0,"BER":0,"ANL_CHG":0,"MCQ":0,"RP_STORY_EVENT_INDEX":10,"RP_STORY_EVENT_COUNT":11,"PRODUCT_KEY":"RPA","PROVIDER_ID":"BZG","PROVIDER_STORY_ID":"12100958:15513604","HEADLINE":"AMCP Partnership Forum Examines Non-Traditional Payment and Benefit Models for High-Cost Pharmaceuticals"}

How would you handle this?

4

If you do

nslookup feed.source.com

then what is the result? Do you have a proxy configured in your .curlrc?

5

You bring up a great point. My logstash is running on my Windows PC. I ran the cURL on a Linux server to check that feed.source.com is running; and confirmed that it is. To answer your question, I did

> nslookup feed.source.com

gives me

Server: my_proxyserver_name.com
Address: 167.12.21.22

Non-authoritative answer:
Name: feed.source.com
Address: 54.87.179.29

I can't ping feed.source.com but I can ping its IP address. So I think I need to provide the IP address to logstash, don't I? So to do that I tried the resolve filter after input like this:

filter {
  dns {
    resolve => ["source_host", "54.87.179.29"]
  }
}

But it gives me the same error as before. How would you handle this?

6

Is that referring to the machine where logstash runs? If so, try

host => "https://54.87.179.29/1.0/json/A9575EEEE9F44635332E9928FD237049"
7

Thank you for your feedback Badger. Yes, I am referring to the machine where logstash is running. I still get the 'name or service not known' error. Here is my config:

input {
  http {
    host => "https://54.87.179.29/1.0/json/A9575EEEE9F446928FD237049"
    port => 443 
	response_headers => { "api_key" => "zBA7goakSWiEY19w1A7aHA" }
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

What can I try now?

8

Wait up, notice that exception is getting raised in add_tcp_listener. The http input creates an network listener, and you do not have the IP address 54.87.179.29 on your server, so it cannot bind to it.

I think you want an http_poller input.

9

Thank you for pointing this out. I followed your tip and I am making progress. My config is this:

input {
  http_poller {
    urls => {
	  test1 => {
        method => get
		  url => "https://54.87.179.29:443/1.0/json/A9575EEEE9F44635332E7049"
		  headers => { 
		    api_key => "zBA7goakSWiEY19w1A7aHA" 
			}
		}
	}
	request_timeout => 60
	schedule => { cron => "* * * * * UTC"}
	codec => "json"
	metadata_target => "http_poller_metadata"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

My screen shows updates every 60 seconds with the following that shows an 'http_request_failure

... "error"←[0;37m => ←[0m←[0;33m"Host name '54.87.179.29' does not match the certificate subject provided by the peer (OU=COMODO EV Multi-Domain SSL, O=RAVENPACK INTERNATIONAL SL, STREE
T=URBANIZACION VILLA PARRA (CRTA. CADIZ KM. 176, L=Marbella, ST=Malaga, OID.2.5.4.17=29602, C=ES, OID.2.5.4.15=Private Organization, OID.1.3.6.1.4.1.311.60.2.1.3=ES, SERIALNUMBER=B92439181)"←[0m, ...

For format I am referring to: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html. What can I try now?

10

OK, try going back to a domain name instead of an IP address.

11

Going back to domain name instead of IP address didn't throw any errors but just sat there. I removed the port after the domain name which caused 'could not read from stream : Read time out
' error. Then, I tried

input {
  http_poller {
    urls => {
	  rv_url => {
        method => get
		  url => "https://feed.source.com:443/1.0/json/A9575EEEE9F44635332E9928FD237049/server-status?auto"
		  headers => { 
		    api_key => "zBA7goakSWiEY19w1A7aHA" 
			}
		}
	}
#	request_timeout => 60
	schedule => { cron => "* * * * * UTC"}
	codec => "json"
#	metadata_target => "http_poller_metadata"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

from: https://www.elastic.co/blog/introducing-logstash-http-poller, which gave me this error:

{
      "@version"←[0;37m => ←[0m←[0;33m"1"←[0m,
    "@timestamp"←[0;37m => ←[0m2018-07-31T20:56:00.240Z,
       "message"←[0;37m => ←[0m←[0;33m"<head><title>Not authorized</title></head>"←[0m,
          "tags"←[0;37m => ←[0m[
        ←[1;37m[0] ←[0m←[0;33m"_jsonparsefailure"←[0m
    ]
}

What can I try now?

12

I don't know what to suggest. It's not really a logstash question at this point. The http_poller is connecting and issuing a request and getting back a response. Not the response you want, but in terms of the logstash configuration it is working as expected.

13

I found that I can run cURL like this:

input {
   exec { 
      command => "curl -X GET \"https://feed.source.com/1.0/json/A9575EEEE9F49\" -H \"api_key:zBA7gHA\""
      interval => 600000000 
   }
}

But how can I handle the quotes? The above gives this error: 'is not recognized as an internal or external command, program, or batch file.' I tried single quotes on the outside which gave the same error. If I don't escape the inner double quotes, I get a configError with line and position of curl command.

What can I try next?

14 
command => 'curl -X GET "https://feed.source.com/1.0/json/A9575EEEE9F49" -H "api_key:zBA7gHA"'

should work.

15

It works now! Thank you Badger! My output is:

output {
  stdout {
    codec => rubydebug
  }
}

What I see is

[2018-08-02T15:41:15,820][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x70562d6c run>"}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0[2018-08-02T15:41:15,919][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2018-08-02T15:41:16,183][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
100 17625 0 17625 0 0 623 0 --:--:-- 0:00:28 --:--:-- 887

I can't see the data on my PC's dos shell, until I hit Ctrl^C. Is this normal? How can I see the output streaming?

16

I do not think so. If I run a configuration like

input { exec { interval => 10 command => 'echo foo' } }
output { stdout { codec => rubydebug } }

I would expect to get this every 10 seconds

{
       "command" => "echo foo",
      "@version" => "1",
    "@timestamp" => 2018-08-02T16:33:33.716Z,
          "host" => "...",
       "message" => "foo\n"
}
1 Like
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.