# Filebeat service doesn't send log

**URL:** <https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 6, 2020, 9:37am UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430 "2020-03-06T09:37:55Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Roms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roms/32/63867_2.png) [@Roms](https://discuss.elastic.co/u/Roms)\
**Post date:** [March 6, 2020, 9:37am UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/1 "2020-03-06T09:37:55Z")

</div>

Hello everyone,

I'm configuring filebeat to read logs from an xml file.

When I do tests with the exe I get the logs in logstasch/elasticsearch and I can work on it in Kibana but when I run Filebeat as a service I don't get any more logs.

This is my current configuration file.

```
filebeat.inputs:
- type: log
  enabled: true
  paths:
    \\mynetworkpathing
  reload.enabled: true
  reload.period: 60s
 

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

output.logstash:
  hosts: ["myhost:5044"]

processors:
  - add_host_metadata: ~

```

And here my Logstash configuration

```
    #Taking data over beats
input {
      beats {
			port => 5044
			tags => ["mytag"]
    }
}

#filtring data
filter {
	if "checkmytag" in [tags] {
  	 xml { my parsing is made here }

}
#output in elastic search
output {
        if "cifs-auditing" in [tags] {
			elasticsearch {
				template_overwrite => true
					hosts => "localhost:9200"
					index => "myfilename-%{+YYYY.MM.dd}" 		                           
			}
        }
		stdout { codec => rubydebug }
}

```

What I would like to do is that the Filebeat can read the file every 5 minutes and pass the additions to logstash. I can't find the error I'm making and I've been at it for quite a while so I'm turning to you.

PS: I'm completely new to Elastic.

Thanking you in advance.  
Sincerely, Romain.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 6, 2020, 11:14am UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/2 "2020-03-06T11:14:03Z")

</div>

Hey @Roms, welcome to discuss 🙂

I think that the first thing to do would be to identify in what piece of the data chain your logs are being lost, you can try to [enable debug logging in filebeat](https://www.elastic.co/guide/en/beats/filebeat/7.6/configuration-logging.html) to see if it is reading the files and sending events to Logstash. If it is not doing it you will have to review the configuration.

For example I see that you are using backslashes in your paths, when these characters are used it is recommended to quote them with single quotes because they are used for escaping other characters and can lead to unexpected results. Also the paths should be configured as a list. So not sure of what kind of paths you want to collect files from, but I recommend you to add the paths like this:

```auto
  paths:
    - '\\mynetworkpathing'

```

Also, the lines about configuration reload are not expected at the input level, they are used when including external configuration files, for example:

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - '\\mynetworkpathing'

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 60s

```

But if you are not using modules you can fully remove this `filebeat.config.modules` section.

> [@Roms](#):
>
> When I do tests with the exe I get the logs in logstasch/elasticsearch and I can work on it in Kibana but when I run Filebeat as a service I don't get any more logs.

How are you using logstash in this scenario? Is "the exe" your application?

---

<div class="post-metadata">

**Author:** ![Roms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roms/32/63867_2.png) [@Roms](https://discuss.elastic.co/u/Roms)\
**Post date:** [March 6, 2020, 12:00pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/3 "2020-03-06T12:00:02Z")

</div>

Hello jsoriano,

I'm using the Filebeat.exe and that's send log but just one time on my Logstash.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cb73c074bcfc87941f373da5b9e2602132e5d051.png)

How can I configure Filebeat to check my logs every 5 minutes?

I catched also this strange error on my Logstash:

```auto
    [2020-03-06T10:45:00,422][INFO][org.logstash.beats.Server][main] Starting server on port: 5044
[2020-03-06T10:45:06,845][ERROR][logstash.javapipeline][main] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats port=>5044, tags=>["mytags"], id=>"c2dc84a0d528db334d1485e14900eb9be9707e471815f0e2881c1a912af179c6", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_5e08c25a-bc2c-498c-b322-628057dfe54f", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"], client_inactivity_timeout=>60, executor_threads=>2>
  Error: Address already in use
  Exception: Java::JavaNet::BindException
  Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:433)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:425)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:223)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:134)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:550)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:504)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:489)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:973)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:248)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:348)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:472)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:500)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:748)

```

edit1: This is my log from Filebeat

```auto
2020-03-06T13:55:18.607+0100 DEBUG [input] input/input.go:152 Run input
2020-03-06T13:55:18.607+0100 DEBUG [input] log/input.go:191 Start next scan
2020-03-06T13:55:18.611+0100 DEBUG [input] log/input.go:212 input states cleaned up. Before: 0, After: 0, Pending: 0

```

```auto
2020-03-06T13:57:58.601+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":109},"total":{"ticks":124,"value":124},"user":{"ticks":15}},"handles":{"open":226},"info":{"epheme
ral_id":"73e23dbb-2f76-4855-9858-4930689da186","uptime":{"ms":180119}},"memstats":{"gc_next":8748816,"memory_alloc":4493008,"memory_total":13228952,"rss":12288},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"modul
e":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2020-03-06T13:58:00.898+0100 DEBUG [input] input/input.go:152 Run input
2020-03-06T13:58:02.369+0100 DEBUG [input] log/input.go:191 Start next scan
2020-03-06T13:58:02.369+0100 DEBUG [input] log/input.go:212 input states cleaned up. Before: 0, After: 0, Pending: 0

```

What I understand after this logs, it's that's doesn't read anymore the file. The first time that I run my Filebeat it's work find and send data but after he doesn't send anymore data and I'm sure that file is growing over time because this xml catch all the things that are made on the file server.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 6, 2020, 3:37pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/4 "2020-03-06T15:37:25Z")

</div>

> [@Roms](#):
>
> I'm using the Filebeat.exe and that's send log but just one time on my Logstash.

Filebeat continuously monitors the configured paths, so if any new log is added to the file, it should be collected.

Filebeat is though for files that are sequentially written, and possibly rotated after some time (as usual log files are). How is the content of this file being written?

The error in logstash indicates that there is already a service listening on port 5044, is it possible that you are

---

<div class="post-metadata">

**Author:** ![Roms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roms/32/63867_2.png) [@Roms](https://discuss.elastic.co/u/Roms)\
**Post date:** [March 6, 2020, 3:48pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/5 "2020-03-06T15:48:27Z")

</div>

@jsoriano  
My file look like

```auto
<EventMain >
<Event><Someinfo></Someinfo></Event>
<Event><Someinfo></Someinfo></Event>
</EventMain>

```

He's allways writing in. It's a CIFS-Auditing file from Netapp.  
It's why I doesn't understant why I got log only on start.  
Filebeat doesn't work for somthing like that ?

And of course he got rotate every 2 hours by the log system.

Thank you for your reply.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 6, 2020, 4:06pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/6 "2020-03-06T16:06:07Z")

</div>

Is the file truncated every time an XML is written? Or the file contains multiple XML objects one after the other?

---

<div class="post-metadata">

**Author:** ![Roms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roms/32/63867_2.png) [@Roms](https://discuss.elastic.co/u/Roms)\
**Post date:** [March 9, 2020, 8:23am UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/7 "2020-03-09T08:23:09Z")

</div>

It is a file in which XML lines are continuously written. When the service starts it reads the data and sends it to logstash and elasticsearch indexes it without any problem. I see them in Kibana.  
But then the file is indicated as if there is no modification while it is continuously fed since every action is transcribed in the file.

That's where I'm lost, how can I get Filebeat to detect the new lines? Do I have to implement some kind of tag on the last line he read? Do I need to change the Filebeat configuration? Do I have to use another product than Filebeat?

Thank you for the support.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 9, 2020, 12:03pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/8 "2020-03-09T12:03:58Z")

</div>

> [@Roms](#):
>
> It is a file in which XML lines are continuously written.

But are the lines appended? or they replace the content in the file?

Filebeat keeps track of the last line read of every file it reads. If new lines are appended, they are automatically collected, no special options are needed for that, this is the normal Filebeat behaviour. If written XML lines replace the existing content in the file, it may be a problem, because Filebeat only detects the changes after the last line read. If content in the file is being continuously replaced then a different strategy may be needed.

---

<div class="post-metadata">

**Author:** ![Roms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roms/32/63867_2.png) [@Roms](https://discuss.elastic.co/u/Roms)\
**Post date:** [March 9, 2020, 12:12pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/9 "2020-03-09T12:12:10Z")

</div>

The file is written continuously and is saved under a different name after a certain period of time.

The logs are contained in the file for one day before being saved under another name.

(There is also the problem that Filebeat only reads the logs at startup I see in Kibana that the reading time remains the same and it no longer feeds the logstash. )

So logically I can't operate filebeat in these conditions and I have to turn to another tool. Which one do you think would do the job well?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [March 9, 2020, 1:02pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/10 "2020-03-09T13:02:03Z")

</div>

It would be good to know what is special in this file, or in this deployment, you may have similar problems with any other tool if you only replace filebeat.

To try to isolate things and identify where is the problem, you might try to run filebeat with console output and check in the logs if it is collecting event from this file.

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - '\\mynetworkpathing'

output.console:
  pretty: true

```

---

<div class="post-metadata">

**Author:** ![Roms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roms/32/63867_2.png) [@Roms](https://discuss.elastic.co/u/Roms)\
**Post date:** [March 9, 2020, 1:26pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/11 "2020-03-09T13:26:56Z")

</div>

At the start I get my full file log and then I get this on a loop:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9d4837555549318b2a59b20b23670aaccbfc0fba.png)

Edit1: I also performed actions that are logged in this file

And this line pops up from time to time.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/d/fd1f88f8d653f6047678463affdbf37d3019a6d3.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2020, 3:26pm UTC](https://discuss.elastic.co/t/filebeat-service-doesnt-send-log/222430/12 "2020-04-06T15:26:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.