Good day.
I try to achieve this pattern using Logstash and pipelines.
# Auth
- pipeline.id: api-getToken
path.config: "api-getToken.conf"
# Count of pages
- pipeline.id: api-getPages
path.config: "api-getPages.conf"
# Fetch data for a specified page
- pipeline.id: api-getData
path.config: "api-getData.conf"
Here is my auth config:
input {
exec {
command => "app-auth"
schedule => "" # Every k days
}
}
output {
pipeline { send_to => tokenEvt }
}
And my getPages config :
input {
pipeline {
address => tokenEvt
http_poller => { ... }
}
}
output {
pipeline { send_to => dataEvt }
}
This is a wrong pattern, probably because I try to use pipeline logic as an event trigger. Moreover, somes issues will probably occure by the necessity to declare the "schedule" property for the http_poller plugin.
All in all, I would know if what I'm trying to do is possible using logstash. If not, what is the best way to achieve that ? Generate all my data files from another process and then use logstash to push them in ES ? Or even generate all my data from a batch or powershell script used as an Logstash exec input and then push the output in ES ?
Thanks in advance for your suggestions.