Error in using EXEC filter for curl

[Gauti (Gautham)]

Hi All,

We have a scenario where through curl we are getting some output, when tried the curl command directly we are getting the output.

curl --location --request POST 'https://demo.cloud.ie/api/v2/analytics/conversations/aggregates/query'
 --header 'Authorization: Bearer Ejd22Vdsfergvvsdrgar' --header 'Cont
ent-Type: application/json'  --data '{  "interval": "2021-08-09T18:30:00.000Z/2021-08-10T18:30:00.000Z",  "metrics": [ "nOffered", "tAn
swered", "tAbandon" ] } '

And this is the output

{"results":[{"group":{"mediaType":"chat"},"data":[{"interval":"2021-08-09T18:30:00.000Z/2021-08-10T18:30:00.000Z","metrics":[{"metric":
"nOffered","stats":{"count":5}},{"metric":"tAnswered","stats":{"max":8500505,"min":2966,"count":5,"sum":17101288}}]}]},{"group":{"media
Type":"email"},"data":[{"interval":"2021-08-09T18:30:00.000Z/2021-08-10T18:30:00.000Z","metrics":[{"metric":"nOffered","stats":{"count"
:8}},{"metric":"tAnswered","stats":{"max":7619543,"min":6446,"count":8,"sum":11067681}}]}]},{"group":{"mediaType":"voice"},"data":[{"in
terval":"2021-08-09T18:30:00.000Z/2021-08-10T18:30:00.000Z","metrics":[{"metric":"nOffered","stats":{"count":30}},{"metric":"tAbandon",
"stats":{"max":143658,"min":1460,"count":15,"sum":457995}},{"metric":"tAnswered","stats":{"max":40646,"min":3107,"count":14,"sum":14216
0}}]}]}]}

When i try putting the same into logstash using exec filter i am getting error, here is my logstash input filter

input {
    exec {
command => " curl --location --request POST 'https://demo.cloud.ie/api/v2/analytics/conversations/aggregates/query'
 --header 'Authorization: Bearer Ejd22Vdsfergvvsdrgar' --header 'Cont
ent-Type: application/json'  --data '{  "interval": "2021-08-09T18:30:00.000Z/2021-08-10T18:30:00.000Z",  "metrics": [ "nOffered", "tAn
swered", "tAbandon" ] } ' "
schedule => "* * * * *"
}
}

While running this i am getting error like below

[2021-08-25T08:01:03,931][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.4", "jruby.version"=>"jruby 9.
2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-08-25T08:01:04,376][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line o
ptions are specified
[2021-08-25T08:01:05,382][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
[2021-08-25T08:01:05,700][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipelin
e_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\" at line 3, column 3
04 (byte 325) after input {\r\n    exec {\r\n        command => \"curl --location --request POST 'https://api.mypurecloud.ie/api/v2/ana
lytics/conversations/aggregates/query' --header 'Authorization: Bearer Ejd22Vlgr-KPzRFUlF7YomzTxbIeCgiFtMJbOtj-ev4PMQCgX5wwuI3KGbMEgRYe
kFmGgZiugSgdpL6kNmKT7Q' --header 'Content-Type: application/json'  --data '{  \"", :backtrace=>["/usr/share/logstash/logstash-core/lib/
logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/
execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initial
ize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/l
ib/logstash/agent.rb:389:in `block in converge_state'"]}
[2021-08-25T08:01:05,792][INFO ][logstash.runner          ] Logstash shut down.
[2021-08-25T08:01:05,800][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

Any advice, where i'm going wrong?

Thanks
Gautham

[Cad]

Hi,

You have to escape all double quotes of the command.

[Gauti (Gautham)]

Hi @Cad

We tried but our source system is not accepting the query without double quotes.

{"message":"The request could not be understood by the server due to malformed syntax.","code":"bad.request","status":400,"contextId":"

Any way where we can keep the double quotes and run the curl in logstash?

Thanks
Gautham

[Cad]

On this previous topic https://discuss.elastic.co/t/logstash-http-poller-body-escape-double-and-single-quotation-marks/282039/2?u=cad someone suggest to use unicode. It's on a different plugin but you should give a try. The unicode for quotes is \u0022

Cad.

[Gauti (Gautham)]

@Cad Tried it, but no luck

Thanks
Gautham

[Cad]

According to this source the following code should work.

' curl --location --request POST "https://demo.cloud.ie/api/v2/analytics/conversations/aggregates/query" --header "Authorization: Bearer Ejd22Vdsfergvvsdrgar" --header "Content-Type: application/json"  --data "{ \"interval\": \"2021-08-09T18:30:00.000Z/2021-08-10T18:30:00.000Z\",  \"metrics\": [ \"nOffered\", \"tAnswered\", \"tAbandon\" ] } " '