Help with formatting a logstash output exec using curl

Elastic Stack Logstash
1

Hi - I am really struggling with what I assume to be the correct escaping of the JSON payload of this curl command in a logstash output exec.

Any tips or guidance or examples would be most welcome!

Thanks

John

I curently have:

command => "curl -X POST -d '{\"objectID\": 0}' 'https://api.ontraport.com/1/objects'     --header 'Content-Type: application/json'     --header 'Api-Key: 1234567'     --header 'Api-Appid: ABCDEFG"

and I get the following errors in logstash debug logging:

[2019-05-15T18:08:37,439][DEBUG][logstash.outputs.exec    ] running exec command {:command=>"curl -X POST -d '{\\\"objectID\\\": 0}' 'https://api.ontraport.com/1/objects'     --header 'Content-Type: application/json'     --header 'Api-Key: 1234567'     --header 'Api-Appid: ABCDEFG'"}
[2019-05-15T18:08:39,202][DEBUG][logstash.outputs.exec    ] debugging command {:stdout=>"Invalid request. Could not parse JSON.", :stderr=>"  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100    17    0     0  100    17      0     10  0:00:01  0:00:01 --:--:--    10\r100    55    0    38  100    17     22      9  0:00:01  0:00:01 --:--:--    22"}

If I remove the escape () characters from around the objectID in the json then logstash rejects the config file with:

`Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Expected one of #, {, } at line 296, column 32 (byte 8355) after output {\n  elasticsearch { \n\t\thosts => [\"localhost:9200\"]\n\t\tindex => [\"pnm-%{+YYYY.MM.dd.H}\"]\n#\t\tssl => true\n#\t\tssl_certificate_verification => false\n#\t\tcacert => './bin/ca.crt'\n#\t\tuser =>'elastic'\n#\t\tpassword => 'password'\n\t\t }\n\nif  \"AGENTSTARTING\" in [tags] {\n    exec {\ncommand => \"curl -X POST -d '{\"", :backtrace=>["/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/pipeline_action/reload.rb:34:in `execute'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:105:in `block in execute'", "/Users/johnwood/TSIElastic/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in `interval'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/agent.rb:94:in `execute'", "/Users/johnwood/TSIElastic/logstash-6.2.4/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/Users/johnwood/TSIElastic/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}`
2

You need to use the right markup for your post. You currently have

Indent the 'command =>' etc with 4 spaces. It will then appear as

and the escapes, asterisks, underscores et al will be preserved.

3

I’ve done that

4

It appears to me that it has sent the escapes along with the double quotes. Can you try reversing the use of single and double quotes?

command => 'curl -X POST -d \'{"objectID": 0}\' ...
5

Okay, I tried what I think you were after:

command => 'curl -X POST -d \'{"objectID": 0}\' \'https://api.ontraport.com/1/objects\'     --header \'Content-Type: application/json\'     --header \'Api-Key: ABCDEFG\'     --header \'Api-Appid: 1234567\''

and I got this in the debug logs:

[2019-05-16T07:32:41,769][DEBUG][logstash.outputs.exec    ] running exec command {:command=>"curl -X POST -d \\'{\"objectID\": 0}\\' \\'https://api.ontraport.com/1/objects\\'     --header \\'Content-Type: application/json\\'     --header \\'Api-Key: ABCDEFG\\'     --header \\'Api-Appid: 1234567\\'"}

[2019-05-16T07:32:41,840][DEBUG][logstash.outputs.exec    ] debugging command {:stdout=>"", :stderr=>"curl: (3) [globbing] unmatched close brace/bracket in column 2\ncurl: (1) Protocol \"'https\" not supported or disabled in libcurl\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: application\ncurl: (6) Could not resolve host: ABCDEFG'\ncurl: (6) Could not resolve host: 1234567'"}

FYI You are welcome to execute this curl on your end - the API and API Key are bogus so you won't do any harm....

Thanks......

6

I don't have enough memory on my machine to run an exec filter, since that forks the JVM.

Can you just remove the single quotes around the URL? They are needed for the JSON, but I don't think they are needed for the URL.

7

Nope that didn’t work either / unless there is someone here with the ability to exec a curl and see the issue then suggest the way forward based on seeing the issue and then fixing the issue, I will just put in place a shell script to work around it and see how to raise a formal ticket on elastic.co to get them to take a look

I appreciate your trying to help mr/ms badger but if you can’t see the issue, I fell it will take days or weeks of ‘try this’ before we can find a solution......thanks for trying to help though....

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.