I am trying to use the HTTP poller to automate a curl command that I am able to run successfully in my environment. I am trying to run a query, put the results through a pipeline and then send the output to elasticsearch. Right now I am relying on bash scripts. I am trying to make use of the HTTP poller, but I am having issues with formatting the query.
The following is the curl:
curl -X POST -k --header "x-apikey: accesskey=xxxx; secretkey=xxx" https://xxxx/rest/analysis/download -d'
{
"query":{
"id":156002
},
"sourceType":"cumulative",
"sortField":"severity",
"startOffset":0,
"endOffset":100,
"type":"vuln",
"columns":[
{"name":"pluginID"},
{"name":"pluginName"},
{"name":"familyPlugin"},
{"name":"severity"},
{"name":"ip"},
{"name":"protocol"},
{"name":"port"},
{"name":"exploitAvailable"},
{"name":"repositoryID"},
{"name":"macAddress"},
{"name":"dnsName"},
{"name":"netbiosName"},
{"name":"pluginText"},
{"name":"firstSeen"},
{"name":"lastSeen"},
{"name":"exploitFrameworks"},
{"name":"synopsis"},
{"name":"description"},
{"name":"solution"},
{"name":"seeAlso"},
{"name":"riskFactor"},
{"name":"stigSeverity"},
{"name":"baseScore"},
{"name":"temporalScore"},
{"name":"cvssVector"},
{"name":"cpe"},
{"name":"cve"},
{"name":"bid"},
{"name":"xref"},
{"name":"vulnPubDate"},
{"name":"patchPubDate"},
{"name":"pluginPubDate"},
{"name":"PluginModDate"},
{"name":"exploitEase"},
{"name":"checkType"},
{"name":"version"}
]
}'
This is the query I am attempting with the HTTP Poller, but I am having trouble with formatting. I am unable to get the body of the query to run without throwing an error:
input {
http_poller {
urls => {
acas => {
method => POST
url => "https://xxxxxx/rest/analysis/download"
body => '{
"query": {
"id":156002
}
{
"sourceType":"cumulative",
"sortField":"severity",
"startOffset":0,
"endOffset":100,
"type":"vuln",
"columns": []
}
}'
headers => {
Accept => "application/json"
"x-apikey" => "accesskey=xxxxx; secretkey=xxxxx"
}
}
}
metadata_target => "http_poller_metadata"
schedule => { "every" => "1m" }
cacert => "/etc/logstash/certs/public.crt"
}
}
#filter {
# mutate {
# gsub => [ "[http_poller_metadata][request][headers][x-apikey]", ".*", "xxxxxx"]
# }
#}
output {
stdout {
codec => rubydebug
}
}
Finally, the output that I am getting when I run the above:
{
"@timestamp" => 2023-03-02T11:47:36.703Z,
"error_code" => 13,
"warnings" => [],
"type" => "regular",
"http_poller_metadata" => {
"response_headers" => {
"strict-transport-security" => "max-age=31536000; includeSubDomains",
"x-content-type-options" => "nosniff",
"content-security-policy" => "default-src 'self'; script-src 'self' pendo-io-static.storage.googleapis.com app.pendo.io cdn.pendo.io pendo-static-6165929460760576.storage.googleapis.com data.pendo.io cdn.metarouter.io e.metarouter.io api.amplitude.com cdn.amplitude.com *.cloudfront.net analytics.cloud.coveo.com platform.cloud.coveo api.tenable.com; connect-src 'self' app.pendo.io data.pendo.io pendo-static-6165929460760576.storage.googleapis.com cdn.metarouter.io e.metarouter.io api.amplitude.com cdn.amplitude.com *.cloudfront.net analytics.cloud.coveo.com platform.cloud.coveo api.tenable.com; img-src 'self' data: cdn.pendo.io app.pendo.io pendo-static-6165929460760576.storage.googleapis.com data.pendo.io; style-src 'self' app.pendo.io cdn.pendo.io pendo-static-6165929460760576.storage.googleapis.com; frame-ancestors 'self' app.pendo.io; form-action 'self'; block-all-mixed-content; upgrade-insecure-requests; object-src 'none'",
"content-length" => "139",
"x-xss-protection" => "1; mode=block",
"connection" => "close",
"vary" => "x-apikey",
"content-type" => "text/html; charset=UTF-8",
"date" => "Thu, 02 Mar 2023 11:47:36 GMT",
"server" => "Apache",
"cache-control" => "no-cache, no-store",
"expires" => "Thu, 19 Nov 1981 08:52:00 GMT",
"expect-ct" => "max-age=31536000",
"x-frame-options" => "DENY",
"pragma" => "no-cache",
"set-cookie" => "TNS_SESSIONID=5f9ef37e5c5b131375d9be674677c07a; path=/; secure; HttpOnly; SameSite=Strict"
},
"host" => "xxxxx",
"name" => "acas",
"runtime_seconds" => 0.554463,
"response_message" => "Bad Request",
"code" => 400,
"times_retried" => 0,
"request" => {
"method" => "post",
"headers" => {
"x-apikey" => "accesskey=xxxx; secretkey=xxxxx",
"Accept" => "application/json"
},
"url" => "https://xxxx/rest/analysis/download",
"body" => "{\n \"query\": {\n \"id\":156002\n }\n {\n \"sourceType\":\"cumulative\",\n \"sortField\":\"severity\",\n \"startOffset\":0,\n \"endOffset\":100,\n \"type\":\"vuln\",\n \"columns\": []\n }\n }"
}
},
"response" => "",
"error_msg" => "This request is not properly formatted.",
"@version" => "1",
"timestamp" => 1677757656
}
Thanks!
)