# Logstash with beats issue **URL:** \ **Category:** Logstash\ **Created:** [June 27, 2018, 7:55pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686 "2018-06-27T19:55:55Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 27, 2018, 7:55pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/1 "2018-06-27T19:55:55Z")
**Installed Logstash beats plug in** ./logstash-plugin install logstash-input-beats Validating logstash-input-beats Installing logstash-input-beats Installation successful When i run the logstash with configuration file which is going to read input from filebeats, I am getting below error. "=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} [2018-06-27T15:54:05,768][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[http://dhrg:9200/](http://dhrg:9200/)"]} [2018-06-27T15:54:06,668][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=\>"10.78.102.73:5044"} [2018-06-27T15:54:06,708][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"} [2018-06-27T15:54:06,879][INFO][org.logstash.beats.Server] Starting server on port: 5044 [2018-06-27T15:54:06,909][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>[]} [2018-06-27T15:54:07,344][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2018-06-27T15:54:13,237][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin. Pipeline\_id:main [2018-06-27T15:54:21,485][INFO][org.logstash.beats.Server] Starting server on port: 5044 [2018-06-27T15:54:27,705][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin. Pipeline\_id:main Plugin: \5044, host=\>"10.78.102.73", id=\>"6f44a9e3029c51148a1228b248e138fa8e73491a8b31870ddd1246d8f8f1d26c", enable\_metric=\>true, codec=\>\"plain\_0a34a7a5-b2eb-4e25-9a67-f16008df7496", enable\_metric=\>true, charset=\>"UTF-8"\>, ssl=\>false, ssl\_verify\_mode=\>"none", ssl\_peer\_metadata=\>false, include\_codec\_tag=\>true, ssl\_handshake\_timeout=\>10000, tls\_min\_version=\>1, tls\_max\_version=\>1.2, cipher\_suites=\>["TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_GCM\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA384", "TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA256", "TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_CBC\_SHA256"], client\_inactivity\_timeout=\>60, executor\_threads=\>16\> Error: Cannot assign requested address Exception: Java::JavaNet::BindException Stack: sun.nio.ch.Net.bind0(Native Method) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:433) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:425) sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:223) io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:128) io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:558) io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1283) io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:501) io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:486) io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:989) io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:254) io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:364) When i installed on windows, i am able to read data from the filebeat using logstash. Can you please help what it is missing. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 8:19pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/2 "2018-06-27T20:19:24Z")
Is 10.78.102.73 the address of the Logstash host? ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 27, 2018, 8:41pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/3 "2018-06-27T20:41:08Z")
It is file beat host address. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 27, 2018, 9:13pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/4 "2018-06-27T21:13:27Z")
Then you have misunderstood what the `host` option means. It's the network interface _on the Logstash host_ on which to listen. Just remove that option; you don't need it. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 28, 2018, 1:29pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/5 "2018-06-28T13:29:34Z")
Thank you Mag. I was thinking this is the IP address of filebeat host. I have removed. Now the error is gone but it is not creating any index by reading the log files. Is that normal to show "0.0.0.0" as a address of beats? [2018-06-28T09:22:10,384][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=\>"0.0.0.0:5044"} [2018-06-28T09:22:10,421][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"} [2018-06-28T09:22:10,545][INFO][org.logstash.beats.Server] Starting server on port: 5044 [2018-06-28T09:22:10,562][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>[]} [2018-06-28T09:22:10,973][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} Do you think the filebeat registry causing issue?. Please advise. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 28, 2018, 4:04pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/6 "2018-06-28T16:04:44Z")
I also tried to dispaly using rubydebug but nothing is displaying on the screen **Logstash.Conf entry** stdout{ codec=\> "rubydebug" } **Logstash while runinng** [2018-06-28T11:59:48,090][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=\>"0.0.0.0:5044"} [2018-06-28T11:59:48,123][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"} [2018-06-28T11:59:48,244][INFO][org.logstash.beats.Server] Starting server on port: 5044 [2018-06-28T11:59:48,244][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>[]} [2018-06-28T11:59:48,673][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} **Filebeat** I deleted filebeat registry before running logstash. When i run logstash, i can see data in the registry. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 28, 2018, 6:53pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/7 "2018-06-28T18:53:33Z")
Mag, Further more, i have also observed in the filebeat log file as below 2018-06-28T14:44:32.802-0400 ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 10.78.102.73:48056-\>10.78.102.106:5044: write: connection reset by peer 2018-06-28T14:44:33.802-0400 ERROR pipeline/output.go:92 Failed to publish events: write tcp 10.78.102.73:48056-\>10.78.102.106:5044: write: connection reset by peer ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 29, 2018, 6:07am UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/8 "2018-06-29T06:07:38Z")
> Is that normal to show "0.0.0.0" as a address of beats? Yes. That address means "listen on all network interfaces". ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 29, 2018, 6:08am UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/9 "2018-06-29T06:08:41Z")
> 2018-06-28T14:44:32.802-0400 ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 10.78.102.73:48056-\>10.78.102.106:5044: write: connection reset by peer > 2018-06-28T14:44:33.802-0400 ERROR pipeline/output.go:92 Failed to publish events: write tcp 10.78.102.73:48056-\>10.78.102.106:5044: write: connection reset by peer It looks like Logstash is closing the connection. Are you sure there aren't any log entries related to this? Make sure your SSL configuration is symmetrical in Logstash and Filebeat, i.e. it should either be on in both places or off in both places. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 29, 2018, 1:37pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/10 "2018-06-29T13:37:59Z")
Thank you Mag. Finally it worked by changing the logstash server name with IP address in the filebeat.yml. I observed two issues after filebeat is working. 1. It looks its reading only last lines of the file rather than all the files of the file. 2. I have two logstash config files for different delivery.Both use the same source files coming from the filebeat. While i am running logstash with 2nd config file, it says " Error: Address already in use". Can you please advise how to we run two logstash processes with different config files both are consuming same file beat data. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 29, 2018, 1:46pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/11 "2018-06-29T13:46:09Z")
> It looks its reading only last lines of the file rather than all the files of the file. If you want to reprocess an old file with Filebeat you need to delete its registry file from disk. > I have two logstash config files for different delivery.Both use the same source files coming from the filebeat. While i am running logstash with 2nd config file, it says " Error: Address already in use". Can you please advise how to we run two logstash processes with different config files both are consuming same file beat data. Do you really need to run multiple Logstash instances? It would be much easier if you just ran everything in a single instance. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 29, 2018, 3:30pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/12 "2018-06-29T15:30:23Z")
Mag, I removed registry however somehow its not reading full content of the file. In my scenario, i need to have two logstash instances which are going to have different grok logic inside logstash conf file. I should have two logstash instances in my scenario. Please do let me know what is the best way to work around. I also tried filebeat sending log data to server folder. From there i am using logstash to read the server folder but the thing is Filebeat is storing "prosperctor" header info along with log record. Is there anyway we can avoid storing filebeat "prospector" while filebeat is sending data to file. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [June 29, 2018, 3:53pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/13 "2018-06-29T15:53:25Z")
{"prospector":{"type":"log"},"host":{"name":"dhrg"},"@version":"1","beat":{"version":"6.3.0","hostname":"dsrc","name":"dsrc"},"message": "LOG ROW data" From the above, i would like to put only "LOG ROW data" in the target file from logstash output file plugin. The input is beats. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 2, 2018, 7:02am UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/14 "2018-07-02T07:02:52Z")
> Mag, I removed registry however somehow its not reading full content of the file. That's probably something you should ask the Filebeat group about. > In my scenario, i need to have two logstash instances which are going to have different grok logic inside logstash conf file. I should have two logstash instances in my scenario. To make a good suggestion of how to deal with the situation I need to understand why you need two instances. > I also tried filebeat sending log data to server folder. From there i am using logstash to read the server folder but the thing is Filebeat is storing "prosperctor" header info along with log record. Is there anyway we can avoid storing filebeat "prospector" while filebeat is sending data to file. I don't know, but you can always delete unwanted fields on the Logstash side. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [July 3, 2018, 2:32pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/15 "2018-07-03T14:32:28Z")
Sure Mag. I will post this to filebeat group. The reason i have multiple logstash instances is due to having different logic for different consumers. Different people write logstash logic to different consumer applications. Hope this gives you enough info. Even though i delete unwanted columns in the logstash, filebeat metadata is still showing in the output. Ex: {"input":{"type":"log"},"@timestamp":"2018-06-29T19:23:13.456Z","message": I am trying to remove the metadata fields and also wanted to see just actual message content in the logstash output file ie. i wanted to move filebeat data to file and use file as a source to logstash input. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 3, 2018, 5:44pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/16 "2018-07-03T17:44:27Z")
> Even though i delete unwanted columns in the logstash, filebeat metadata is still showing in the output. > > Ex: {"input":{"type":"log"},"@timestamp":"2018-06-29T19:23:13.456Z","message": So how, exactly, are you deleting the fields? > I am trying to remove the metadata fields and also wanted to see just actual message content in the logstash output file ie. i wanted to move filebeat data to file and use file as a source to logstash input. Make sure you're using the lines codec for the file output in your Logstash configuration. That codec allows you to configure the output format. I believe the default format includes the timestamp. ---
**Author:** ![HRG](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@HRG](https://discuss.elastic.co/u/HRG)\ **Post date:** [July 3, 2018, 7:22pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/17 "2018-07-03T19:22:08Z")
Thank you Mag. After adding codec line, it is able to send atleast raw log rows to output file. I observed its sending only first 500. Going through some posts to resolve this 500 limitation. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 31, 2018, 7:22pm UTC](https://discuss.elastic.co/t/logstash-with-beats-issue/137686/18 "2018-07-31T19:22:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.