"}
[2019-12-10T10:30:05,337][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=\>"amproxy"}
[2019-12-10T10:30:05,337][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=\>"beslproxy"}
[2019-12-10T10:30:05,344][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=\>"cisproxy"}
**[2019-12-10T10:30:05,361][INFO][logstash.inputs.tcp] Starting tcp input listener {:address=\>"0.0.0.0:5542", :ssl\_enable=\>"false"}**
**[2019-12-10T10:30:05,362][INFO][logstash.inputs.tcp] Starting tcp input listener {:address=\>"0.0.0.0:5544", :ssl\_enable=\>"false"}**
**[2019-12-10T10:30:05,389][INFO][logstash.inputs.tcp] Starting tcp input listener {:address=\>"0.0.0.0:5543", :ssl\_enable=\>"false"}**
[2019-12-10T10:30:05,436][INFO][logstash.agent] Pipelines running {:count=\>3, :running\_pipelines=\>[:beslproxy, :amproxy, :cisproxy], :non\_running\_pipelines=\>}
[2019-12-10T10:30:05,631][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
Pipeline Settings in /etc/logstash/logstash.yml
# pipeline.id: main
# pipeline.workers: 2
# pipeline.batch.size: 125
# path.config:
/etc/logstash/pipelines.yml
- pipeline.id: beslproxy
path.config: "/appdata/ELK/logstash/conf/logstash\_BESL.conf"
- pipeline.id: amproxy
path.config: "/appdata/ELK/logstash/conf/logstash\_AM.conf"
- pipeline.id: cisproxy
path.config: "/appdata/ELK/logstash/conf/logstash\_CIS.conf"
* * *
Here are my logstash conf files (The ownership and permissions are same for all 3 conf file)
logstash\_BESL.conf
input {
tcp {
port =\> 5544
type =\> syslog
tags =\> ["BESLPROXY"]
#Therotically this tags really not required, but trying with that too
}
}
filter {
if "BESLPROXY" in [tags] {
grok {
match =\> [REMOVED]
remove\_field =\> ["port", "\_id",]
}
mutate {
convert =\> {
"statusCode" =\> "integer"
"systemTimestamp" =\> "integer"
"targetResponseTimeInMillis" =\> "integer"
"requestProcessingTimeInMillis" =\> "integer"
}
}
```
date {
match => ["timestamp", "ISO8601"]
target => "@timestamp"
}
geoip {
source => "sourceIP"
}
```
}
}
output {
if "BESLPROXY" in [tags] {
if [apiproxy] == "VALUEX" or [apiproxy] == "VALUEY" or [apiproxy] == "VALUEX" or [apiproxy] == "VALUEAB" {
elasticsearch {
hosts =\> ["SERVER:9200"]
user =\> "USERNAME"
password =\> "PASSWORD"
index =\> "logstash-besl-%{+YYYY.MM.dd}"
}
stdout { codec =\> rubydebug }
}
}
}
* * *
logstash\_AM.conf
input {
tcp {
port =\> 5543
type =\> syslog
tags =\> ["AppointmentManagerProxy"]
#Therotically this tags really not required, but trying with that too
}
}
filter {
if "AppointmentManagerProxy" in [tags] {
grok {
match =\> [REMOVED]
remove\_field =\> ["port", "\_id"]
}
mutate {
convert =\> {
"statusCode" =\> "integer"
"systemTimestamp" =\> "integer"
"targetResponseTimeInMillis" =\> "integer"
"requestProcessingTimeInMillis" =\> "integer"
}
}
```
date {
match => ["timestamp", "ISO8601"]
target => "@timestamp"
}
geoip {
source => "sourceIP"
}
```
}
}
output {
if "AppointmentManagerProxy" in [tags] {
if [apiproxy] == "AppointmentManager" {
elasticsearch {
hosts =\> ["SERVER:9200"]
user =\> "USERNAME"
password =\> "PASSWORD"
index =\> "logstash-am-%{+YYYY.MM.dd}"
}
stdout { codec =\> rubydebug }
}
}
}
* * *
logstash\_CIS.conf
input {
tcp {
port =\> 5542
type =\> syslog
tags =\> ["CISPROXY"]
#Therotically this tags really not required, but trying with that too
}
}
filter {
if "CISPROXY" in [tags] {
grok {
match =\> [REMOVED]
remove\_field =\> ["port", "\_id"]
}
mutate {
convert =\> {
"statusCode" =\> "integer"
"systemTimestamp" =\> "integer"
"targetResponseTimeInMillis" =\> "integer"
"requestProcessingTimeInMillis" =\> "integer"
"tin\_last4" =\> "integer"
}
}
date {
match =\> ["timestamp", "ISO8601"]
target =\> "@timestamp"
}
geoip {
source =\> "sourceIP"
}
}
}
output {
if "CISPROXY" in [tags]{
if [apiproxy] == "CIS" {
elasticsearch {
hosts =\> ["SERVER:9200"]
user =\> "USERNAME"
password =\> "PASSWORD"
index =\> "logstash-cis-%{+YYYY.MM.dd}"
}
stdout { codec =\> rubydebug }
}
}
}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 10, 2019, 7:39pm UTC](https://discuss.elastic.co/t/configured-logstash-multiple-pipelines-but-not-working-need-help/211361/2 "2019-12-10T19:39:48Z")
Your writes to elasticsearch are conditional upon the value of apiproxy. Maybe that does not have the value you think it does.
---
### Author:  [@shylendran](https://discuss.elastic.co/u/shylendran)
#### Post date: [December 11, 2019, 3:09pm UTC](https://discuss.elastic.co/t/configured-logstash-multiple-pipelines-but-not-working-need-help/211361/3 "2019-12-11T15:09:06Z")
> [@Badger](#):
>
> Your writes to elasticsearch are conditional upon the value of apiprocy. Maybe that does not have the value you think it does.
Hello,
All the conditions are satisfying, meaning the vales sending from the source to Logstash are correct. Confirmed this by running the individual Logstash conf file.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [December 11, 2019, 3:22pm UTC](https://discuss.elastic.co/t/configured-logstash-multiple-pipelines-but-not-working-need-help/211361/4 "2019-12-11T15:22:37Z")
Pipleline.yml file is always tricky.
even some space kills it. I had so much trouble in past with it.
I never figure out what it is. but create new file and type everything in it. do not cut-paste anything in that file.
hopefully that will fix the problem.
---
### Author:  [@ITIC](https://discuss.elastic.co/u/ITIC)
#### Post date: [December 12, 2019, 1:10pm UTC](https://discuss.elastic.co/t/configured-logstash-multiple-pipelines-but-not-working-need-help/211361/5 "2019-12-12T13:10:15Z")
Hi
Maybe you have already tried it, but I'll give it a shot.
I have a single pipeline with several `.yml` files. Each file has its own `input{}` with unique `tags` and unique `id`, and everything works like a charm. I see you don't have an `id`.
If you have them in separate pipelines you won't need either `tags` or `id`, since they don't "see" each other, but this setup seems to add more complexity, and there might be some error in your pipeline configuration, as already pointed out by others.
I don't use the `tcp` input, but I figure it must work just the same with the setup I described.
Hope this helps.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 9, 2020, 1:10pm UTC](https://discuss.elastic.co/t/configured-logstash-multiple-pipelines-but-not-working-need-help/211361/6 "2020-01-09T13:10:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.