Logstash calling a curl through exec quotations not allowed

[katara]

Im trying to send my logstash output to an API url with the exec output plugin.

> output {
> exec {
> command => "curl -u username:'password' -k https://url.com/zdmd/Events/evconsole_router -d '{"action":"EventsRouter","method":"add_event","data":[{"summary":%{Message},"device":%{hostname},"component":%{appname},"severity":5,"evclasskey":"Test","evclass":"/Status/Ping","monitor":"localhost"}],"type":"rpc","tid":1}' -H "Content-Type: application/json""
> }
> }

But since the command has to be under a double quote, the internal double quotes are causing an error in configutation. I tried changing all the double quotes to single quotes inside, but even if there is no error in logstash, the data isnt reaching the server api.

when i try the curl directly (single quoted) it gives me the below error:

> ## Site Error
> 
>  <p>An error was encountered while publishing this resource.
>   </p>
>   <p><strong><class 'AccessControl.unauthorized.Unauthorized'></strong></p>
> 
>   Sorry, a site error occurred.<p><p>Traceback (innermost last):</p>
> <ul>
> 
> <li>  Module ZPublisher.Publish, line 237, in publish_module_standard</li>
> 
> <li>  Module ZPublisher.Publish, line 165, in publish</li>
> 
> <li>  Module Zope2.App.startup, line 230, in __call__</li>
> 
> <li>  Module Products.ZenUI3.browser, line 104, in __call__</li>
> 
> <li>  Module Products.Five.browser.pagetemplatefile, line 59, in __call__</li>
> 
> <li>  Module zope.pagetemplate.pagetemplate, line 113, in pt_render</li>
> 
> <li>  Module zope.tal.talinterpreter, line 271, in __call__</li>
> 
> <li>  Module zope.tal.talinterpreter, line 343, in interpret</li>
> 
> <li>  Module zope.tal.talinterpreter, line 858, in do_defineMacro</li>
> 
> <li>  Module zope.tal.talinterpreter, line 343, in interpret</li>
> 
> <li>  Module zope.tal.talinterpreter, line 533, in do_optTag_tal</li>
> 
> <li>  Module zope.tal.talinterpreter, line 518, in do_optTag</li>
> 
> <li>  Module zope.tal.talinterpreter, line 513, in no_tag</li>
> 
> <li>  Module zope.tal.talinterpreter, line 343, in interpret</li>
> 
> <li>  Module zope.tal.talinterpreter, line 742, in do_insertStructure_tal</li>
> 
> <li>  Module Products.PageTemplates.Expressions, line 218, in evaluateStructure</li>
> 
> <li>  Module zope.tales.tales, line 696, in evaluate<br />
> <b>URL: /opt/zenoss/Products/ZenUI3/browser/error_message.pt</b><br />
> <b>Line 14, Column 8</b><br />
> <b>Expression: &lt;PathExpr standard:u'view/headExtra'&gt;</b><br />
> <b>Names:</b><pre>{'args': (),
> 'container': AttributeError('evconsole_router',),
> 'context': AttributeError('evconsole_router',),
> 'default': &lt;object object at 0x7fda721c4580&gt;,
> 'here': AttributeError('evconsole_router',),
> 'loop': {},
> 'nothing': None,
> 'options': {},
> 'repeat': &lt;Products.PageTemplates.Expressions.SafeMapping object at 0x7fda4b5ae940&gt;,
> 'request': &lt;HTTPRequest, URL=https://url.com/zport/dmd/evconsole_router&gt;,
> 'root': None,
> 'template': &lt;Products.Five.browser.pagetemplatefile.ViewPageTemplateFile object at 0x7fda34910850&gt;,
> 'traverse_subpath': [],
> 'user': &lt;SpecialUser 'Anonymous User'&gt;,
> 'view': &lt;Products.Five.metaclass.ErrorMessage object at 0x7fda443e6f90&gt;,
> 'views': &lt;Products.Five.browser.pagetemplatefile.ViewMapper object at 0x7fda44500390&gt;}</pre></li>
> 
> <li>  Module zope.tales.expressions, line 217, in __call__</li>
> 
> <li>  Module Products.PageTemplates.Expressions, line 147, in _eval</li>
> 
> <li>  Module zope.tales.expressions, line 124, in _eval</li>
> 
> <li>  Module Products.PageTemplates.Expressions, line 97, in trustedBoboAwareZopeTraverse</li>
> 
> <li>  Module zope.traversing.adapters, line 136, in traversePathElement<br />
> __traceback_info__: (&lt;Products.Five.metaclass.ErrorMessage object at 0x7fda443e6f90&gt;, 'headExtra')</li>
> 
> <li>  Module zope.traversing.adapters, line 42, in traverse<br />
> __traceback_info__: (&lt;Products.Five.metaclass.ErrorMessage object at 0x7fda443e6f90&gt;, 'headExtra', [])</li>
> 
> <li>  Module Products.ZenUI3.browser, line 66, in headExtra</li>
> 
> <li>  Module Products.ZenUI3.browser, line 42, in _get_viewlets</li>
> 
> </ul><p>Unauthorized: Unauthorized()
> </p></p>
>   <hr noshade="noshade"/>
> 
>   <p>Troubleshooting Suggestions</p>
> 
>   <ul>
>   <li>The URL may be incorrect.</li>
>   <li>The parameters passed to this resource may be incorrect.</li>
>   <li>A resource that this resource relies on may be
>       encountering an error.</li>
>   </ul>
> 
>   <p>For more detailed information about the error, please
>   refer to the error log.
>   </p>
> 
>   <p>If the error persists please contact the site maintainer.
>   Thank you for your patience.

I also tried with

> command => "curl -u username:password -XPUT -k https://url.com/zdmd/Events/evconsole_router -d \"{\"action\":\"Eventrouter\", method:\"add_event\", \"data\": [{\"summary\":%{Message}, \"device\":%{host},\"message\":%{ApplicationName},\"componnet\":%{host},\"severity\":%{Severity},\"evclasskey\":\"nxlog\", \"evclass\":\"/nxlog/perf\",\"monitor\":\"localhost\"}],\"type\":\"rpc\",\"tid\":2}\" -H \"Content-Type: application/json\""

which wasnt working too.

How would this work?
Kindly help me out!

[ptamba]

i guess it's due to mixing of quotes in your command. since it's an http endpoint, why not use the http output plugin?

[katara]

I did try to @ptamba, but the cacert not working in the http output plugin is still an open issue. It seems to be a problem which is open from 2015. I thought this will work as a good alternative. But unfortunately, I'm not producing it with the right quotations.

[ptamba]

apologies, we were discussing the cacert option, didn’t we?

well this might be a long shot but have you tried building the http_body portion as a field then pass the field content to curl ?

so your curl will look like :
curl -u username:'password' -k https://url.com/zdmd/Events/evconsole_router -d %{request_body}

with request_body is an event field you build using logstash filter?

[katara]

Hi @ptamba, Thank you for bringing it up, I'm trying to see if this can be done, I do have issues with the filters being brought to array.
I've added a new topic here for the same.

If there are other options from any other members, I would very much like to try it out!

Thanks!