# Pipeline sending to itself
**URL:** https://discuss.elastic.co/t/pipeline-sending-to-itself/260413
**Category:** Logstash
**Created:** [January 7, 2021, 7:41am UTC](https://discuss.elastic.co/t/pipeline-sending-to-itself/260413 "2021-01-07T07:41:48Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)
#### Post date: [January 7, 2021, 7:41am UTC](https://discuss.elastic.co/t/pipeline-sending-to-itself/260413/1 "2021-01-07T07:41:48Z")
Hello all,
We have the following usecase:
We want to read data from a service provider that requires paging when polling data(they also support pushing data but we are not accessible from the internet). I found that there are a few threads about http\_poller and paging but no perfect solution.
At first, we made a straightforward try of using the http\_poller to read the data but only returned a partial result because the provider requires paging for larger results(only 100 entries per page) and a token for the next page.
My next try was the logic into 2 pipelines:
One pipeline `request-init`:
- uses the heartbeat input to regularly initiate the workflow
- uses the lumberjack output to pipeline `request-execution`.
Pipeline `request-execution`:
- has the Beats input(as the documentation of the lumberjack inputs recommends to use Beats input instead) to get the trigger
- uses the http filter to do the polling of the REST API. If the request contained a nextPageToken this will be sent with the request as well.
- does a bit of processing(like splitting the response)
- uses elasticsearch output to send the data to Elasticsearch
- uses the lumberjack output to pipeline `request-execution` with the nextPageToken to trigger reading the next page.
Unfortunately, I cannot start pipeline `request-execution` because of an error:
`[2021-01-07T08:06:41,580][ERROR][logstash.outputs.lumberjack] All hosts unavailable, sleeping ...`
So it looks like the output is opened before the input is open. Is there a way to workaround that?
If possible I would like to use LogStash only without calling an external program to handle the paging.
Best regards
Wolfram
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 7, 2021, 3:19pm UTC](https://discuss.elastic.co/t/pipeline-sending-to-itself/260413/2 "2021-01-07T15:19:28Z")
> [@Wolfram\_Haussig](#):
>
> Unfortunately, I cannot start pipeline `request-execution` because of an error:
> `[2021-01-07T08:06:41,580][ERROR][logstash.outputs.lumberjack] All hosts unavailable, sleeping ...`
That would be request-init, not request-execution, right?
Why not just wait ten seconds and let it retry?
---
### Author:  [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)
#### Post date: [January 7, 2021, 5:09pm UTC](https://discuss.elastic.co/t/pipeline-sending-to-itself/260413/3 "2021-01-07T17:09:09Z")
> [@Badger](#):
>
> That would be request-init, not request-execution, right?
No, `request-init` also does not start but this is fine because the pipeline `request-execution` is indeed not yet running. No, I have thought that the pipeline `request-execution` could call itself to support the paging feature which seems to be my problem as the input plugin is not yet running when the output is started - or so it seems.
Best regards
Wolfram
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 7, 2021, 5:27pm UTC](https://discuss.elastic.co/t/pipeline-sending-to-itself/260413/4 "2021-01-07T17:27:52Z")
Got it. So the request-execution tries to start the output which connects to its own input before it starts the input, so it never completes startup.
You could use a different input/output pair where the startup order does not matter. A file input and output would work, but would never clean up the old entries. A kafka input and output would also work. There may be others.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 4, 2021, 5:28pm UTC](https://discuss.elastic.co/t/pipeline-sending-to-itself/260413/5 "2021-02-04T17:28:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.