Filebeat service doesn't send log

Roms · March 6, 2020, 9:37am

Hello everyone,

I'm configuring filebeat to read logs from an xml file.

When I do tests with the exe I get the logs in logstasch/elasticsearch and I can work on it in Kibana but when I run Filebeat as a service I don't get any more logs.

This is my current configuration file.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    \\mynetworkpathing
  reload.enabled: true
  reload.period: 60s
 

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml



output.logstash:
  hosts: ["myhost:5044"]

processors:
  - add_host_metadata: ~

And here my Logstash configuration

    #Taking data over beats
input {
      beats {
			port => 5044
			tags => [ "mytag" ]
    }
}

#filtring data
filter {
	if "checkmytag" in [tags] {
  	 xml { my parsing is made here }

}
#output in elastic search
output {
        if "cifs-auditing" in [tags] {
			elasticsearch {
				template_overwrite => true
					hosts => "localhost:9200"
					index => "myfilename-%{+YYYY.MM.dd}" 		                           
			}
        }
		stdout { codec => rubydebug }
}

What I would like to do is that the Filebeat can read the file every 5 minutes and pass the additions to logstash. I can't find the error I'm making and I've been at it for quite a while so I'm turning to you.

PS: I'm completely new to Elastic.

Thanking you in advance.
Sincerely, Romain.

jsoriano · March 6, 2020, 11:14am

Hey @Roms, welcome to discuss

I think that the first thing to do would be to identify in what piece of the data chain your logs are being lost, you can try to enable debug logging in filebeat to see if it is reading the files and sending events to Logstash. If it is not doing it you will have to review the configuration.

For example I see that you are using backslashes in your paths, when these characters are used it is recommended to quote them with single quotes because they are used for escaping other characters and can lead to unexpected results. Also the paths should be configured as a list. So not sure of what kind of paths you want to collect files from, but I recommend you to add the paths like this:

  paths:
    - '\\mynetworkpathing'

Also, the lines about configuration reload are not expected at the input level, they are used when including external configuration files, for example:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - '\\mynetworkpathing'

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 60s

But if you are not using modules you can fully remove this filebeat.config.modules section.

How are you using logstash in this scenario? Is "the exe" your application?

Roms · March 6, 2020, 12:00pm

Hello jsoriano,

I'm using the Filebeat.exe and that's send log but just one time on my Logstash.

How can I configure Filebeat to check my logs every 5 minutes?

I catched also this strange error on my Logstash:

    [2020-03-06T10:45:00,422][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
[2020-03-06T10:45:06,845][ERROR][logstash.javapipeline    ][main] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats port=>5044, tags=>["mytags"], id=>"c2dc84a0d528db334d1485e14900eb9be9707e471815f0e2881c1a912af179c6", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_5e08c25a-bc2c-498c-b322-628057dfe54f", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"], client_inactivity_timeout=>60, executor_threads=>2>
  Error: Address already in use
  Exception: Java::JavaNet::BindException
  Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:433)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:425)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:223)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:134)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:550)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:504)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:489)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:973)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:248)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:348)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:472)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:500)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:748)

edit1: This is my log from Filebeat

2020-03-06T13:55:18.607+0100    DEBUG   [input] input/input.go:152      Run input
2020-03-06T13:55:18.607+0100    DEBUG   [input] log/input.go:191        Start next scan
2020-03-06T13:55:18.611+0100    DEBUG   [input] log/input.go:212        input states cleaned up. Before: 0, After: 0, Pending: 0

2020-03-06T13:57:58.601+0100    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":109},"total":{"ticks":124,"value":124},"user":{"ticks":15}},"handles":{"open":226},"info":{"epheme
ral_id":"73e23dbb-2f76-4855-9858-4930689da186","uptime":{"ms":180119}},"memstats":{"gc_next":8748816,"memory_alloc":4493008,"memory_total":13228952,"rss":12288},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"modul
e":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2020-03-06T13:58:00.898+0100    DEBUG   [input] input/input.go:152      Run input
2020-03-06T13:58:02.369+0100    DEBUG   [input] log/input.go:191        Start next scan
2020-03-06T13:58:02.369+0100    DEBUG   [input] log/input.go:212        input states cleaned up. Before: 0, After: 0, Pending: 0

What I understand after this logs, it's that's doesn't read anymore the file. The first time that I run my Filebeat it's work find and send data but after he doesn't send anymore data and I'm sure that file is growing over time because this xml catch all the things that are made on the file server.

jsoriano · March 6, 2020, 3:37pm

Filebeat continuously monitors the configured paths, so if any new log is added to the file, it should be collected.

Filebeat is though for files that are sequentially written, and possibly rotated after some time (as usual log files are). How is the content of this file being written?

The error in logstash indicates that there is already a service listening on port 5044, is it possible that you are

Roms · March 6, 2020, 3:48pm

@jsoriano
My file look like

<EventMain >
<Event><Someinfo></Someinfo></Event>
<Event><Someinfo></Someinfo></Event>
</EventMain>

He's allways writing in. It's a CIFS-Auditing file from Netapp.
It's why I doesn't understant why I got log only on start.
Filebeat doesn't work for somthing like that ?

And of course he got rotate every 2 hours by the log system.

Thank you for your reply.

jsoriano · March 6, 2020, 4:06pm

Is the file truncated every time an XML is written? Or the file contains multiple XML objects one after the other?

Roms · March 9, 2020, 8:23am

It is a file in which XML lines are continuously written. When the service starts it reads the data and sends it to logstash and elasticsearch indexes it without any problem. I see them in Kibana.
But then the file is indicated as if there is no modification while it is continuously fed since every action is transcribed in the file.

That's where I'm lost, how can I get Filebeat to detect the new lines? Do I have to implement some kind of tag on the last line he read? Do I need to change the Filebeat configuration? Do I have to use another product than Filebeat?

Thank you for the support.

jsoriano · March 9, 2020, 12:03pm

But are the lines appended? or they replace the content in the file?

Filebeat keeps track of the last line read of every file it reads. If new lines are appended, they are automatically collected, no special options are needed for that, this is the normal Filebeat behaviour. If written XML lines replace the existing content in the file, it may be a problem, because Filebeat only detects the changes after the last line read. If content in the file is being continuously replaced then a different strategy may be needed.

Roms · March 9, 2020, 12:12pm

The file is written continuously and is saved under a different name after a certain period of time.

The logs are contained in the file for one day before being saved under another name.

(There is also the problem that Filebeat only reads the logs at startup I see in Kibana that the reading time remains the same and it no longer feeds the logstash. )

So logically I can't operate filebeat in these conditions and I have to turn to another tool. Which one do you think would do the job well?

jsoriano · March 9, 2020, 1:02pm

It would be good to know what is special in this file, or in this deployment, you may have similar problems with any other tool if you only replace filebeat.

To try to isolate things and identify where is the problem, you might try to run filebeat with console output and check in the logs if it is collecting event from this file.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - '\\mynetworkpathing'

output.console:
  pretty: true

Roms · March 9, 2020, 1:26pm

At the start I get my full file log and then I get this on a loop:

Edit1: I also performed actions that are logged in this file

And this line pops up from time to time.

system · April 6, 2020, 3:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.