Passing Current time in Logstash input plugin 'exec' for executing API in curl

Elastic Stack Logstash
1

I want to execute API in Logstash & schedule it every 30 mins and store the data in elastic search. I'm using exec logstash plugin to do, In the curl command there is start and end query param where i need to specify epoch timestamp. Here i want to use something which give me current time in curl command which i'm not able to figure it out. I need to parse start and end so it will take current time. start=currenttime() - 1917 and end=currenttime() . Basically it will give me data for last 30 mins.

start=1596659630-1917=1596657713
end=1596659630
below is my code
input {
exec {
    command => 'curl -X POST "http://localhost:9080/events/query?start=1422823420000&end=1423687476000" -H"Content-type: application/vnd.appd.events+json;v=2" -d "SELECT pagename AS ErrorPage, referrer AS GeneratedPage, count(*) AS ErrorTraffic FROM browser_records WHERE appkey = 'AAA-AAB-AUB'
    schedule => "*/30 * * * *"
    }
  }
output {
stdout { codec => json_lines }
  }

Kindly help

2

You can reference environment variables in an input, but you that would require setting them externally to logstash and using something like cron to run logstash every 30 minutes.

I would suggest using exec with a schedule and command => '/bin/true' and then use ruby to calculate the start/end, and an http filter to send the request.

3

Thanks @Badger for the suggestion. so far i have done this

input {
exec {
    command => 'bin/true'
    tags => "myaccount_test"
    schedule => "* * * * *"
    }
  }

filter {
if "myaccount_test" in [tags] {
    ruby { code => "event.set('end', event.get('@timestamp').to_i)" }
    ruby { code => "event.set('start', event.get('@timestamp').to_i - 1917)" }
    http {
  body_format => "json"
  follow_redirects => false
  body => {
    "query" => "SELECT pagename AS ErrorPage, referrer AS GeneratedPage, count(*) AS ErrorTraffic FROM browser_records WHERE appkey = 'AAA-AAB-AUB'"
    "start" => "%{start}"
    "end" => "%{end}"
  }
  url => "http://<domainname>:<port>/events/query?limit=200"
  verb => "POST"
  headers => {
    "X-Events-API-AccountName" => "custxxxxx_xxxxxxxxxxxx"
    "X-Events-API-Key" => "xxxxxxx-xxxxxxx-xxxx"
    "Content-type" => "application/vnd.appd.events+json;v=2"
  }
}
}
}

output {
if "myaccount_test" in [tags] {
 stdout { codec => json_lines }
  }
}

But i'm getting error for http filter (Bad request in accessing /events/query

[2020-08-07T16:15:00,165][ERROR][logstash.filters.http    ][main][e70ac31237ba1ca2ac387603953776f0cb06c3aecbd113db0915a1404334ad85] error during HTTP request {:url=>"http://<domainname>:<port>/events/query?limit=200", :code=>400, :response=>"<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=utf-8\"/>\n<title>Error 400 Bad Request</title>\n</head>\n<body><h2>HTTP ERROR 400</h2>\n<p>Problem accessing /events/query. Reason:\n<pre>    Bad Request</pre></p>\n</body>\n</html>\n"}
{"@timestamp":"2020-08-07T12:15:00.041Z","end":1596802500,"start":1596800583,"message":"","@version":"1","command":"bin/true","host":"hostname","tags":["myaccount_test","_httprequestfailure"]}

below is the http request example which im trying to access

POST http://api.example.com/events/query?limit=100 HTTP/1.1
X-Events-API-AccountName:<global_account_name>
X-Events-API-Key:<api_key>
Content-Type: application/vnd.appd.events+json;v=2
Accept: application/vnd.appd.events+json;v=2
[
    {
      "query": "SELECT * FROM county WHERE population>50000",
      "limit": 10,
      "start": "2017-02-23T0:0:0Z",
      "end": "2017-03-1T0:0:0Z"
    },
]

Kindly help

4

Does the request body need to be an array? It does not appear to be valid JSON to me.

5

Dear Badger,

This is direct API request, when i tried in curl it is working

command => 'curl -X POST "http://localhost:9080/events/query?start=1422823420000&end=1423687476000" -H"Content-type: application/vnd.appd.events+json;v=2" -d "SELECT pagename AS ErrorPage, referrer AS GeneratedPage, count(*) AS ErrorTraffic FROM browser_records'

i want to convert this in format as of http filter. below is the sample example post from document which i want to achieve

POST http://analytics.api.example.com/events/query?start=1422823420000&end=1423687476000&limit=20000 HTTP/1.1
X-Events-API-AccountName:<global_account_name>
X-Events-API-Key:<api_key>
Content-Type: application/vnd.appd.events+text;v=2
Accept: application/vnd.appd.events+json;v=2
  
SELECT * FROM county WHERE size>=30 AND population>20000

Can you please help how i can achieve above in logstash

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.