# Beat and Logstash

**URL:** <https://discuss.elastic.co/t/beat-and-logstash/118200>\
**Category:** Logstash\
**Created:** [February 2, 2018, 9:55am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200 "2018-02-02T09:55:21Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [February 2, 2018, 9:55am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/1 "2018-02-02T09:55:21Z")

</div>

I am new to coding and logstash.  
I am receiving many files through filebeat on single port in my logstash.conf.

How can I differentiate the files basis on their naming conventions and then write a filter for them.

Anyone can share the sample of logstash.conf file.

Thanks  
Siddharth

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 2, 2018, 10:17am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/2 "2018-02-02T10:17:49Z")

</div>

Update your Filebeat configuration to set fields on events from each file. Those fields could e.g. describe the application name and the environment name. Based on that you can apply custom filtering on the Logstash side.

---

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [February 4, 2018, 3:46am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/3 "2018-02-04T03:46:57Z")

</div>

> [@magnusbaeck](#):
>
> fields

Thanks, I would really appreciate if you can share a way that i can do something to differentiate the files in logstash.conf. Is it possible? Would be a good practice?

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 4, 2018, 9:07pm UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/4 "2018-02-04T21:07:30Z")

</div>

> I would really appreciate if you can share a way that i can do something to differentiate the files in logstash.conf. Is it possible?

Probably. Do you have any examples?

> Would be a good practice?

I think it's a better idea to classify logs at the source (in this case Filebeat) where you know the truth instead of guessing afterwards.

---

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [February 8, 2018, 7:17am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/5 "2018-02-08T07:17:05Z")

</div>

Thanks as always, for every advise.  
You mean, i should tag the file in filebeat.conf and then use that tag in logstash.conf for filtering.

filebeat example 1:

filebeat.prospectors:

- paths: ["/var/log/smtp.log"]
- tags: ["smtp"]

logstash example1:

filter{  
if "smtp" in [tags] {  
processing.....  
}  
}

filebeat example2:

filebeat.prospectors:

- paths: ["/var/log/smtp.log"]
- document\_type: smtp
- input\_type: log

logstash example2:  
filter{  
if [type] == "smtp" {  
processing....  
}  
}

can i use either way to differentiate the file in filebeat.conf?  
both are the correct ways to differentiate the file in filebeat.conf?

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 8, 2018, 7:54am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/6 "2018-02-08T07:54:19Z")

</div>

> You mean, i should tag the file in filebeat.conf and then use that tag in logstash.conf for filtering.

Yes.

> can i use either way to differentiate the file in filebeat.conf?

Yes.

> both are the correct ways to differentiate the file in filebeat.conf?

Yes.

---

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [February 8, 2018, 9:04am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/7 "2018-02-08T09:04:03Z")

</div>

Thankyou so much Magnus.  
I honestly appreciate your help.

---

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [February 20, 2018, 8:51am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/8 "2018-02-20T08:51:30Z")

</div>

Dear Magnus,

Can you review it as i m getting an error in filebeat "Failed to Publish events: write: connection reset by peer" while pushing the file from Filebeat to Logstash.

filebeat.prospectors:

- type: log  
enabled: true  
paths:
  - /app/uce/uce.csv  
document\_type: spam

logstash.conf  
input {  
port =\> "10011"  
[type] =\> "spam"  
}  
filter {  
if [type] == "spam" {  
}  
}  
output {  
if [type] == "spam" {  
}  
}

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 9:48am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/9 "2018-02-20T09:48:29Z")

</div>

What does Filebeat's output configuration look like?

**Always** format configuration snippets as preformatted text.

---

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [February 26, 2018, 8:09am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/11 "2018-02-26T08:09:07Z")

</div>

here is my filebeat.yml configuration.

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so

# you can use different prospectors for various configurations.

# Below are the prospector specific configurations.

- type: log

#----------------------------- Logstash output --------------------------------  
output.logstash:

# The Logstash hosts

hosts: ["10.224.225.18:10001"]

# Optional SSL. By default is off.

# List of root certificates for HTTPS server verifications

#ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 26, 2018, 8:39am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/12 "2018-02-26T08:39:09Z")

</div>

I repeat: **Always** format configuration snippets as preformatted text. There's a toolbar button for it.

> input {  
> port =\> "10011"  
> [type] =\> "spam"  
> }

That's invalid syntax so I'm surprised Logstash even starts.

```plaintext
input {
  beats {
    port => 10011
    type => "spam"
  }
}

```

---

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [March 5, 2018, 4:52am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/13 "2018-03-05T04:52:10Z")

</div>

Thanks, It is now working.  
I can see the harvester is running and sending data to my logstash and logstash is tagging it as i want.

But I am getting something like below, actually its many.

at io.netty.channel.MultipthreadEvenLoopGroup.register  
at io.netty.bootstrap.AbstractBootstrap.initAndRegister  
at io.netty.bootstrap.AbstractBootstrap.doBind  
at io.netty.bootstrap.AbstractBootstrap.bind  
at sun.reflect.GeneratedMethodAccessor41  
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling  
at org.jruby.javasupport.JavaMethod.invokeDirect

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 5, 2018, 7:04am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/14 "2018-03-05T07:04:51Z")

</div>

Please post the full error message.

---

<div class="post-metadata">

**Author:** ![sid.sh](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@sid.sh](https://discuss.elastic.co/u/sid.sh)\
**Post date:** [March 15, 2018, 9:06am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/15 "2018-03-15T09:06:29Z")

</div>

Thanks Magnus for always be with me.  
🙂  
Everything is now placed and working.

V soon i will share my conf files, which i think will be helpful for others.

Thanks  
Siddharth

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2018, 9:06am UTC](https://discuss.elastic.co/t/beat-and-logstash/118200/16 "2018-04-12T09:06:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.