# Pipeline conf file processes wrong log

**URL:** <https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159>\
**Category:** Logstash\
**Created:** [May 16, 2018, 2:53pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159 "2018-05-16T14:53:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![shine](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@shine](https://discuss.elastic.co/u/shine)\
**Post date:** [May 16, 2018, 2:53pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/1 "2018-05-16T14:53:21Z")

</div>

My logs are not being processed by the correct conf file.

Customer B logs are being processed by "cust\_log\_a.conf" instead of "cust\_log\_b.conf"

**pipeline.yml**  
pipeline.id: customer\_logs\_a  
path.config: "/etc/logstash/conf.d/cust\_log\_a.conf"  
pipeline.id: customer\_logs\_b  
path.config: "/etc/logstash/conf.d/cust\_log\_b.conf"

**cust\_log\_a.conf**

```
input {
   beats {
                 port => 5044
                 type => "log"
             }
}
filter
{
   mutate { gsub => ["message", "[\n]", "" ]
   }
          if [fields][customer_name] == 'cust_apple_log'
         {
              grok 
                      {
                       ......
                      }
        }		
}
output {
     elasticsearch {
          hosts => "10.222.3.44:9200"
          manage_template => false
         index => "cust_apple"
}
stdout {}
}

```

**cust\_log\_b.conf**

```
input {
   beats {
                 port => 5045
                 type => "log"
             }
}
filter
{
   mutate { gsub => ["message", "[\n]", "" ]
   }
          if [fields][customer_name] == 'cust_baseball_log'
         {
              grok 
                      {
                       ......
                      }
        }		
}
output {
     elasticsearch {
          hosts => "10.222.3.44:9200"
          manage_template => false
         index => "cust_baseball"
}
stdout {}
}

```

**filebeats.yml**

```
type: log
     paths:
     /usr/local/glassfish4/domains/customer_a/logs/events.log
     fields:
         customer_name: cust_apple_log

type: log
     paths:
     /usr/local/glassfish4/domains/customer_B/logs/events.log
     fields:
         customer_name: cust_baseball_log

output.logstash:
hosts: ["10.222.3.44:5044", "10.222.3.44:5045"]
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 16, 2018, 3:52pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/2 "2018-05-16T15:52:36Z")

</div>

> [@shine](#):
>
> output.logstash:  
> hosts: ["10.222.3.44:5044", "10.222.3.44:5045"]

If load balancing is disabled, but multiple hosts are configured, one host is selected randomly (there is no precedence). See [here](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html).

---

<div class="post-metadata">

**Author:** ![shine](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@shine](https://discuss.elastic.co/u/shine)\
**Post date:** [May 16, 2018, 5:30pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/3 "2018-05-16T17:30:47Z")

</div>

Thanks Badger,

What do you recommend I do to process multiple logs with different patterns?

We have 12 server(Ubuntu 14.04) with 6 customer websites hosted on each one.  
We need to track 3 custom logs for each customer's website.  
12 servers  
-- 6 customers/server  
--- 3 logs\_files/customer

Should I create one xxx.conf file with 216 if..else statements?

```
if ['cust_01_log_apple','cust_02_log_apple', ... 'cust_216_log_apple'] in [fields][customer_name]
{
	grok 
	{
  	  .... pattern 1
	  add_field => { "cust_log_type" == "apple" }
	}
} else {
if ['cust_01_log_baseball','cust_02_log_baseball', ... 'cust_216_log_baseball'] in [fields][customer_name]
	{
		grok 
		{
		  .... pattern 2
		  add_field => { "cust_log_type" == "baseball" }
		}
	}
}
} else {
if ['cust_01_log_football','cust_02_log_football', ... 'cust_216_log_football'] in [fields][customer_name]
	{
		grok 
		{
		  .... pattern 3
		  add_field => { "cust_log_type" == "football" }
		}
	}
}

output {
   elasticsearch {
   hosts => "11.222.3.44:9200"
   manage_template => false
	index => "%{[fields][customer_name]}_{cust_log_type}"
  }
  stdout {}
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 16, 2018, 5:43pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/4 "2018-05-16T17:43:04Z")

</div>

Why would you need to have one conditional _per server_? If you have 6 customers and they all have 3 unique log file formats you'll probably need 18 conditionals, unless you can use a single grok filter that lists 18 expressions (which Logstash will try in order and bail out as soon as there's a match).

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 16, 2018, 5:48pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/5 "2018-05-16T17:48:25Z")

</div>

> [@shine](#):
>
> What do you recommend I do to process multiple logs with different patterns?

I would configure filebeat with 18 different prospectors that tag the events with the customer id and the log format. You have 3 different log formats, not 18, right? Then in logstash execute the groks conditionally based on the log format.

---

<div class="post-metadata">

**Author:** ![shine](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@shine](https://discuss.elastic.co/u/shine)\
**Post date:** [May 16, 2018, 6:34pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/6 "2018-05-16T18:34:52Z")

</div>

Hi Badger,

_12 servers x 6 customer = 72 customers_  
Each customer has an apple.log + baseball.log + football.log

Is this what you mean?

```
if ['cust_1_apple_log','cust_2_apple_log', ... 'cust_72_apple_log'] in [fields][customer_name]
	{
		grok 
		{
		  .... pattern 1
		  add_field => { "cust_log_type" == "apple" }
		}
	}
}else
if ['cust_1_baseball_log','cust_2_baseball_log', ... 'cust_72_baseball_log'] in [fields][customer_name]
	{
		grok 
		{
		  .... pattern 2
		  add_field => { "cust_log_type" == "baseball" }
		}
	}
}
}else
if ['cust_1_football_log','cust_2_football_log', ... 'cust_72_football_log'] in [fields][customer_name]
	{
		grok 
		{
		  .... pattern 3
		  add_field => { "cust_log_type" == "football" }
		}
	}
}
}

```

**filebeats.yml**

```
prospectors:
##################### Customer 1 ###############################
type: log
     paths:
     /usr/local/glassfish4/domains/customer_A/logs/apple.log
     fields:
         customer_name: cust_1_apple_log

type: log
     paths:
     /usr/local/glassfish4/domains/customer_A/logs/baseball.log
     fields:
         customer_name: cust_1_baseball_log

type: log
     paths:
     /usr/local/glassfish4/domains/customer_A/logs/football.log
     fields:
         customer_name: cust_1_football_log
		 
##################### Customer 2 ###############################
type: log
     paths:
     /usr/local/glassfish4/domains/customer_2/logs/apple.log
     fields:
         customer_name: cust_2_apple_log

type: log
     paths:
     /usr/local/glassfish4/domains/customer_2/logs/baseball.log
     fields:
         customer_name: cust_2_baseball_log

type: log
     paths:
     /usr/local/glassfish4/domains/customer_2/logs/football.log
     fields:
         customer_name: cust_2_football_log
		 
##################### Customer 3,4,5... ###############
```

---

<div class="post-metadata">

**Author:** ![shine](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@shine](https://discuss.elastic.co/u/shine)\
**Post date:** [May 16, 2018, 6:59pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/7 "2018-05-16T18:59:14Z")

</div>

Hi MagnusBaeck,

_My mistake on 216 customers_  
(12 servers) x (6 customer/per server) = 72 customers  
Only 3 grok patterns needed

I made this change but I noticed apple.log will match correctly but baseball.log gets a  
tags: beats\_input\_codec\_plain\_applied, \_grokparsefailure

But when I run one baseball.log entry and filter in [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) it filters correctly.

```
filter
{
	grok 
		{
                        #### apple.log Grok
			match => {"message" => "\[%{TIMESTAMP_ISO8601:timestamp_utc_jvm}\] (\[(%{WORD:server_name} %{NUMBER:server_ver})\]) \[%{WORD:severity}\] \[(?:%{DATA:glassfish_code}|)\] \[(?:%{JAVACLASS:java_pkg}|)\] \[(?<Thread_Name>[^\]]+)\] \[(?<timeMillis>[^\]]+)\] \[(?<levelValue>[^\]]+)\] (?<StackTrace>(?m:.*))"}
                        #### baseball.log Grok
			match => {"message" => "\[%{TIMESTAMP_ISO8601:timestamp_utc_jvm}\] %{WORD:severity} %{JAVACLASS:java_pkg} \[(?<Thread_Name>[^\]]+)\] (?<log_msg>(?m:.*))"}
		}
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 16, 2018, 7:08pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/8 "2018-05-16T19:08:45Z")

</div>

I was thinking of something like:

```
type: log
   paths:
   /usr/local/glassfish4/domains/customer_A/logs/apple.log
   fields:
       customer_name: cust_1
       filetype: apple_log
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 13, 2018, 7:19pm UTC](https://discuss.elastic.co/t/pipeline-conf-file-processes-wrong-log/132159/9 "2018-06-13T19:19:54Z")

</div>

Closed on June 13, 2018, 7:19pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.