Multiple pipeline pattern for REST API

Good day.
I try to achieve this pattern using Logstash and pipelines.

# Auth
- pipeline.id: api-getToken
  path.config: "api-getToken.conf"

# Count of pages
- pipeline.id: api-getPages
  path.config: "api-getPages.conf"

# Fetch data for a specified page
- pipeline.id: api-getData
  path.config: "api-getData.conf"

Here is my auth config:

input {
    exec {
        command => "app-auth"
        schedule => "" # Every k days
    }
}

output {
    pipeline { send_to => tokenEvt }
}

And my getPages config :

input {
    pipeline {
        address => tokenEvt
        http_poller => { ... }
    }
}

output {
    pipeline { send_to => dataEvt }
}

This is a wrong pattern, probably because I try to use pipeline logic as an event trigger. Moreover, somes issues will probably occure by the necessity to declare the "schedule" property for the http_poller plugin.

All in all, I would know if what I'm trying to do is possible using logstash. If not, what is the best way to achieve that ? Generate all my data files from another process and then use logstash to push them in ES ? Or even generate all my data from a batch or powershell script used as an Logstash exec input and then push the output in ES ?

Thanks in advance for your suggestions.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.