# Logstash http poller input & filter flow not working properly

**URL:** <https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183>\
**Category:** Logstash\
**Created:** [February 28, 2025, 3:16am UTC](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183 "2025-02-28T03:16:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 28, 2025, 3:16am UTC](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183/1 "2025-02-28T03:16:31Z")

</div>

Hello,

Below is my requirement:

**First** http call with user & password will return bearer token, token expires every 30mins. So need to call the URL every 30mins for updated token.

**Second** : Get token from above and poll Prometheus metrics URL every min with that bearer token.

Below is my implementation:

**First scenario:**

```auto
#Get bearer token 
input {
     http_poller {
     urls => 
		{
         gettoken => 
		 {
 			url => "http://test.dev.com:40021/oauth/token?grant_type=client_credentials"
			user => "46ad390f-aa20a253cae3"
			password => "9a9b-c27a3a7114d7"
		}
      }
      keepalive => true
      automatic_retries => 1
      codec => json
      schedule => { cron => "*/30 * * * * UTC"}
   }   
}

#Set bearer token 
filter
{
	ruby 
	{
		code => "
		event.set('token',event.get('access_token'))
		"
	}
}

```

**Second Scenario:**

```auto
#Prometheus Metrics Poller
input 
{
	http_poller 
	{
	 urls => 
		{
		 rps => 
		 {
			url => "http://test.dev.com:11000/green/service/actuator/prometheus?tId=7667977e-6ddd-4788-8dcf-578a746b8812"
			headers => 
			{
				"Authorization" => "Bearer %{token}"
			}
		}
		}
		keepalive => true
		automatic_retries => 1
		codec => line
		schedule => { cron => "* * * * * UTC"}
		add_field => { "index" => "hc-prometheus" }
		add_field => { "hc_type" => "prometheus-metrics" }
	}
}

#Prometheus Metrics Parsing Filter
filter
{
 #some filters....
}

```

Able to get token from first scenario and save to "token" field. But not able to send that to 2nd HTTP poller as i am getting unauthorised response.

**Pls let me know the issue?** @Badger or anyone else

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 28, 2025, 5:02am UTC](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183/2 "2025-02-28T05:02:32Z")

</div>

After going through other post where http\_poller will get initiated before any events, changed above code as below:

```auto
#Get bearer token 
input {
     http_poller {
     urls => 
		{
         gettoken => 
		 {
 			url => "http://test.dev.com:40021/oauth/token?grant_type=client_credentials"
			user => "46ad390f-f56a-aa20a253cae3"
			password => "8e057748-c27a3a7114d7"
		}
      }
	  keepalive => true
      automatic_retries => 1
	  codec => json
      schedule => { cron => "* * * * * UTC"}
	  add_field => { "index" => "hc-prometheus" }
	  add_field => { "hc_type" => "prometheus-metrics" }
   }   
}

#Save bearer token 
filter
{
	ruby 
	{
		code => "
		event.set('token',event.get('access_token'))
		"
	}
}

filter
{	
	http {
		url => "http://test.dev.com:11000/green/service/actuator/prometheus?tId=7667977e-6ddd-578a746b8812"
		verb => "GET"
		headers => 
		{
			"Authorization" => "Bearer %{token}"
		}
		target_body => "prometheusdataset"
	}
}

```

finally able to get **prometheusdataset** filed as below:

```auto
# HELP disk_free_bytes Usable space for path\n# TYPE disk_free_bytes gauge\ndisk_free_bytes{CUST=\"green\",ENV=\"QA\",PROCESS_NAME=\"trade_singleton_1\",path=\"/test/TRADE/.\"} 1.12633286656E11\n# HELP disk_total_bytes Total space for path\n

```

Earlier we have added a codec=\> line on http\_poller which was splitting above output to different lines. But now i cannot use line codec for http filter. **Pls let me know how can i split above output to multiline events.**

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 28, 2025, 1:44pm UTC](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183/3 "2025-02-28T13:44:04Z")

</div>

Use a [split](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html#_description_159) filter. The documentation calls out the use case of splitting the multiline output of a command into multiple events.

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 28, 2025, 4:28pm UTC](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183/4 "2025-02-28T16:28:16Z")

</div>

> [@Bhanu\_Praveen](#):
>
> ```auto
> http {
> url => "http://test.dev.com:11000/green/service/actuator/prometheus?tId=7667977e-6ddd-578a746b8812"
> verb => "GET"
> headers => 
> {
> "Authorization" => "Bearer %{token}"
> }
> target_body => "prometheusdataset"
> }
> 
> ```

Split is working, Thanks. Pls let me know is there anyway we can add another HTTP section inside filter dynamically from env variable? I do not want to update logstash config file manually for every new end point addition.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 28, 2025, 4:41pm UTC](https://discuss.elastic.co/t/logstash-http-poller-input-filter-flow-not-working-properly/375183/5 "2025-02-28T16:41:51Z")

</div>

> [@Bhanu\_Praveen](#):
>
> Pls let me know is there anyway we can add another HTTP section inside filter dynamically from env variable?

Not that I can think of.