# Logstash multiple conf files configuration is not working when passing fields between conf files

**URL:** <https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299>\
**Category:** Logstash\
**Created:** [March 3, 2025, 7:48am UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299 "2025-03-03T07:48:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [March 3, 2025, 7:48am UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299/1 "2025-03-03T07:48:06Z")

</div>

Hello,

Currently a I have a single file as below which will do the following requirements:

Step1: Get bearer token every minute  
Step2: Use bearer token and get Prometheus data  
stpe3: parsing Prometheus data

Below single file working fine.

```auto
#Get bearer token which is same for every DC
input {
     http_poller {
     urls => 
		{
     gettoken => 
		 {
 			url => "http://test.dev.com:40021/oauth/token?grant_type=client_credentials"
			user => "46ad390f-4b8f-8b36-aa20a253cae3"
			password => "8e057748-4dc1-9a9b-c27a3a7114d7"
		}
      }
	  keepalive => true
      automatic_retries => 1
	  codec => json
      schedule => { cron => "* * * * * UTC"}
	  add_field => { "index" => "hc-prometheus" }
	  add_field => { "hc_type" => "prometheus-metrics" }
   }   
}

#Save bearer token 
filter
{
	ruby 
	{
		code => "
		event.set('token',event.get('access_token'))
		"
	}
}

#Get prometheus dataset
filter
{	
	http {
		url => "http://test.dev.com:11000/green/service/actuator/prometheus?tId=7667977e-8dcf-578a746b8812"
		verb => "GET"
		headers => 
		{
			"Authorization" => "Bearer %{token}"
		}
		target_body => "prometheusdataset"
	}
}

filter
{	
 split {
   field => "prometheusdataset"
 }
}

#Prometheus Metrics Parsing Filter
filter
{
    
		mutate { add_tag => "a" }
		
		if [prometheusdataset] =~ /^#/ { drop{ } }
	 
		dissect { 
				mapping => 
				{ 
					"prometheusdataset" => "%{[@metadata][type]}{%{[@metadata][stuff]}} %{[@metadata][value]}" 
				} 
			}
	   
		kv { 
			source => "[@metadata][stuff]" target => "[@metadata][data]" field_split => "," trim_value => '\\"' 
			}
		
		mutate { rename => { "[@metadata][value]" => "[@metadata][data][value]" } }
		
		mutate { rename => { "[@metadata][data]" => "%{[@metadata][type]}" } }
}

```

Now I need to split the above single file into multiple conf files, so that i can change only Step2 file which will reload automatically for every new environment edition.

New split requirement:

Step1: Create new input\_file name "111-gettoken.conf"  
Step2: Create a new filter file name "112-getprometheusdata.conf"  
Step3: Create a new filter file name "113-parseprometheusdata.conf"

**111-gettoken.conf will have below code:**

```auto
#Get bearer token which is same for every DC
input {
     http_poller {
     urls => 
		{
     gettoken => 
		 {
 			url => "http://test.dev.com:40021/oauth/token?grant_type=client_credentials"
			user => "46ad390f-f56a-4b8f-8b36-aa20a253cae3"
			password => "8e057748-a09a-4dc1-9a9b-c27a3a7114d7"
		}
      }
	  keepalive => true
      automatic_retries => 1
	  codec => json
      schedule => { cron => "* * * * * UTC"}
	  add_field => { "index" => "hc-prometheus" }
	  add_field => { "hc_type" => "prometheus-metrics" }
   }   
}

#Save bearer token 
filter
{
	ruby 
	{
		code => "
		event.set('token',event.get('access_token'))
		"
	}
}

```

**112-getprometheusdata.conf will have below code:**

```auto
#Get prometheus dataset
filter
{	
	http {
		url => "http://in7lin075.dev.e2open.com:11000/green/service/actuator/prometheus?tId=7667977e-6ddd-4788-8dcf-578a746b8812"
		verb => "GET"
		headers => 
		{
			"Authorization" => "Bearer %{token}"
		}
		target_body => "prometheusdataset"
	}
}

```

**113-parseprometheusdata.conf will have below code:**

```auto

filter
{	
 split {
   field => "prometheusdataset"
 }
}

#Prometheus Metrics Parsing Filter
filter
{
    
		mutate { add_tag => "a" }
		
		if [prometheusdataset] =~ /^#/ { drop{ } }
	 
		dissect { 
				mapping => 
				{ 
					"prometheusdataset" => "%{[@metadata][type]}{%{[@metadata][stuff]}} %{[@metadata][value]}" 
				} 
			}
	   
		kv { 
			source => "[@metadata][stuff]" target => "[@metadata][data]" field_split => "," trim_value => '\\"' 
			}
		
		mutate { rename => { "[@metadata][value]" => "[@metadata][data][value]" } }
		
		mutate { rename => { "[@metadata][data]" => "%{[@metadata][type]}" } }
}

```

When i split to multiple config's as above, pipeline is not working. I have a docket ELK stack which will pick up all conf files and working for other scenarios where i am not passing variables between the files as shown pipeline.conf below. The problem is Step1 saved " **bearer token"** is not passing in Step2 filter HTTP section and same Step2 " **prometheusdataset**" is not parsed in Step3 conf file. Request to let me know the issue?

```auto
- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline/*.conf"
  pipeline.workers: 5

```

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [March 3, 2025, 2:03pm UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299/2 "2025-03-03T14:03:01Z")

</div>

@leandrojmp @Badger Request to check this

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [March 3, 2025, 2:15pm UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299/3 "2025-03-03T14:15:03Z")

</div>

Can you please share the log from the logstash server when this is "not working" as well as your logstash.yml

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [March 3, 2025, 2:46pm UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299/4 "2025-03-03T14:46:09Z")

</div>

No errors on logstash pipeline. Step 2 http url is returning 401 unauthorized/connection error due to bearer token is not there.

```auto
still the same 
[ERROR][logstash.filters.http][main][24d3c8cea4b1de3f69c8df3d30b2cd69825af6a1fbc352b9824d28edc4e61074] error during HTTP request {:url=>"http://test.dev.com:11000/green/service/actuator/prometheus?tId=7667977e-6ddd-4788-8dcf-578a746b8812", :body=>nil, :client_error=>"Connect to test.dev.com:11000 [test.dev.com/10.12.14.78] failed: Connection refused"}

```

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [March 3, 2025, 3:08pm UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299/5 "2025-03-03T15:08:23Z")

</div>

Please share the entire log file from Logstash, not just one line. If you can enable debug logging first that would be great too.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 3, 2025, 3:46pm UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299/6 "2025-03-03T15:46:42Z")

</div>

What exactly is not working? It is not clear.

There is no difference for Logstash if the configuration is just one file or multiple files as multiple files will be merged.

Please share **all files** you are using, you didn't share nothing with an output for example.

> [@Bhanu\_Praveen](#):
>
> I have a docket ELK stack which will pick up all conf files and working for other scenarios where i am not passing variables between the files as shown pipeline.conf below.

Please share how you are starting Logstash as well, are you using Docker compose? Did you stopped the container and started it again before changing from one conf file to multiple conf files?

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [March 3, 2025, 5:02pm UTC](https://discuss.elastic.co/t/logstash-multiple-conf-files-configuration-is-not-working-when-passing-fields-between-conf-files/375299/7 "2025-03-03T17:02:37Z")

</div>

Thanks for your time into this. It is working fine. My bad for not configured properly.