# How to send different logs files to different logstash ports

**URL:** <https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 3, 2017, 9:22pm UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781 "2017-08-03T21:22:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nikhilpawar1985](https://avatars.discourse-cdn.com/v4/letter/n/e68b1a/32.png) [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\
**Post date:** [August 3, 2017, 9:22pm UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781/1 "2017-08-03T21:22:54Z")

</div>

I am trying to send different log files on different logstash ports on logstash ..like one on 5044 and other on 5043

Here is my filebeat config . can someone help

#=========================== Filebeat prospectors =============================

```
filebeat.prospectors:

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /etc/filebeat/download.2017-06-16-1125-1224.log
 # tail_files: true
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  document_type: download

- input_type: log
  paths:
    - /etc/filebeat/vus.2017-08-02.log
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  document_type: vus

#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["dfdevlogstash1.df.jabodo.com:5044"]
  loadbalance: true
  worker: 2
# filebeat.publish_async: true
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [August 3, 2017, 10:52pm UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781/2 "2017-08-03T22:52:21Z")

</div>

Beats are designed to deliver data to one output. That output can be multiple servers for the purposes of load balancing. But you cannot send specific logs to one port and other logs to another.

Why do you need to do this? What are you trying to accomplish?

---

<div class="post-metadata">

**Author:** ![Nikhilpawar1985](https://avatars.discourse-cdn.com/v4/letter/n/e68b1a/32.png) [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\
**Post date:** [August 4, 2017, 12:32am UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781/3 "2017-08-04T00:32:49Z")

</div>

Thanks for the info andrew . I am sending 2 log files ( different log structure) to to logstash cluster at port 5044 but its not able to identify properly and grok is unable to parse it . so i was wondering if sending it to different ports might help . Can you please suggest how can i achieve this ?

Also log files which i am sending are custom log files not generic like apache , tomcat etc.

Nikhil

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [August 4, 2017, 12:37am UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781/4 "2017-08-04T00:37:14Z")

</div>

You can apply tags to the events or use the `source` field to identify which logs came from which files.

Example using tags:

```auto
filebeat.prospectors:
- paths:
  - /path/to/log/type/A/*.log
  tags: ["log_type_a"]
- paths:
  - /path/to/log/type/B/*.json
  tags: ["log_type_b", json]

```

Then over in your Logstash config you can use a condition to apply whatever processing you would like.

```auto
filters {
  if "log_type_b" in [tags] {
     // do something fun like grok
  } // else if ...
}

```

---

<div class="post-metadata">

**Author:** ![Nikhilpawar1985](https://avatars.discourse-cdn.com/v4/letter/n/e68b1a/32.png) [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\
**Post date:** [August 4, 2017, 6:43pm UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781/5 "2017-08-04T18:43:54Z")

</div>

Hi Andrew , i tried as you said but i am only able to parse and send one log file to ES .

Below are my filebeat and logstash configs .

logstash.conf

```
   input {
        beats {
        client_inactivity_timeout => 86400
        port => 5044
        type => "log"
              }
          }
    filter {
        if "vus" in [tags] {
        mutate {
        gsub => [
          "message", "\t", " ",
          "message", "\n", " "
                ]
            }
        grok {
    match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}%{WORD:level}%{SPACE}%{JAVACLASS:coidkey}%{SPACE}%{USER:ident}%{SPACE}%{GREEDYDATA:Url}"}

      }
    }
     else if "download" in [tags] {
        mutate {
        gsub => [
          "message", "\t", " ",
          "message", "\n", " "
                ]
            }
        grok {
        match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}\:\|\:%{SPACE}%{WORD:level}%{SPACE}\:\|\:%{SPACE}%{USERNAME:host_name}%{SPACE}\:\|\:%{SPA
    CE}%{GREEDYDATA:coidkey}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:clientinfo}%{SPACE}\:\|\:%{SPACE}(%{IP:clientip})?%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:Url}%{SPACE}\:\|\:%
    {SPACE}%{JAVACLASS:class}%{SPACE}\:\|\:%{SPACE}%{USER:ident}%{SPACE}%{GREEDYDATA:msg}"}
       remove_field => ["ident","offset","name","version","host"]
             }
                     }

      }
     output {
        stdout { codec => rubydebug }

      if "_grokparsefailure" in [tags] {
        # write events that didn't match to a file
        file { "path" => "/tmp/grok_failures.txt" }
      }
    # if "vus" in [tags] {
    # elasticsearch {
    # hosts => "dfdevelasticp1.df.jabodo.com:9200"
    # user => "elastic"
    # password => "fd5dd89c"
    # index => "download-%{+YYYY.MM.dd}"
    # document_type => "log"
    # }
    # }
    #if [type] == "download"{
    else{
    elasticsearch {
           hosts => "dfsyselastic.df.jabodo.com:9200"
           user => "UN"
           password => "PW"
           index => "vicinio-%{+YYYY.MM.dd}"
           document_type => "log"
         }
       }
    }

```

filebeat.yml (1st host )

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

```
- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /opt/log/tomcat/vus.log
  tail_files: true
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["vus"]

#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["dfsyslogstash1.df.jabodo.com:5044"]
# hosts: ["dfsyslogstash1.df.jabodo.com:5044","dfsyslogstash2.df.jabodo.com:5044",
"dfsyslogstash3.df.jabodo.com:5044","dfsyslogstash4.df.jabodo.com:5044"]
  loadbalance: true
  worker: 2
# filebeat.publish_async: true

```

==============

filebeat.yml (2nd host )

```
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /archives/logs/tomcat7-8080/download.log
    - /archives/logs/tomcat7-8090/download.log
  tail_files: true
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["download"]
#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["dfsyslogstash1.df.jabodo.com:5044"]
# hosts: ["dfsyslogstash1.df.jabodo.com:5044","dfsyslogstash2.df.jabodo.com:5044",
"dfsyslogstash3.df.jabodo.com:5044","dfsyslogstash4.df.jabodo.com:5044"]
  loadbalance: true
  worker: 2
# filebeat.publish_async: true

```

Do i need to send it to different indexes on ES

---

<div class="post-metadata">

**Author:** ![Nikhilpawar1985](https://avatars.discourse-cdn.com/v4/letter/n/e68b1a/32.png) [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\
**Post date:** [August 5, 2017, 2:14pm UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781/6 "2017-08-05T14:14:05Z")

</div>

Andre ..can you please help on this .

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 2, 2017, 2:14pm UTC](https://discuss.elastic.co/t/how-to-send-different-logs-files-to-different-logstash-ports/95781/7 "2017-09-02T14:14:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.