Filebeat don't pick up overwritten log file

Elastic Stack Beats
1

Hi Everyone,

already looked here and Fielbeat documentation, but something isn't working :confused:

there is only 1 log file that i want to collect and this one get overwritten every 3 minutes.

i already added close_eof: true but that didn't help, any other clue?

EDIT:

adding logs and config

filebeat log

2018-07-18T08:00:06.176+0200	INFO	log/harvester.go:228	Harvester started for file: D:\BCP\carrental_table_counts.csv
2018-07-18T08:00:06.182+0200	INFO	log/harvester.go:251	End of file reached: D:\BCP\table_counts.csv. Closing because close_eof is enabled.
2018-07-18T08:00:06.223+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1875},"total":{"ticks":2734,"value":2734},"user":{"ticks":859}},"info":{"ephemeral_id":"26574d15-86f5-42bd-8aa2-84a22109808b","uptime":{"ms":52890202}},"memstats":{"gc_next":4319648,"memory_alloc":2269240,"memory_total":182437792,"rss":1581056}},"filebeat":{"events":{"active":719,"added":720,"done":1},"harvester":{"closed":1,"open_files":0,"running":0,"started":1},"input":{"log":{"files":{"truncated":1}}}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":718,"filtered":2,"published":718,"total":720}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}}}}}
2018-07-18T08:00:07.182+0200	ERROR	logstash/async.go:235	Failed to publish events caused by: write tcp y.y.y.y:61031->x.x.x.x:5044: wsasend: An existing connection was forcibly closed by the remote host.
2018-07-18T08:00:08.183+0200	ERROR	pipeline/output.go:92	Failed to publish events: write tcp y.y.y.y:61031->x.x.x.x:5044: wsasend: An existing connection was forcibly closed by the remote host.

at this snipped the time was 08:00, then it follows 08:03 and then 08:27, so he skipped quite a lot of 3minute intervals.

config:

filebeat.inputs:

- type: log
  paths:
    - D:\BCP\table_counts.csv
  close_eof: true
  close_inactive: 1m
  fields:
    logstash: carrental
  fields_under_root: true

logging:
  to_files: true
  files:
    path: C:\Program Files\filebeat\Logs
  level: info

#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["x.x.x.x:5044"]

thanks in advance

Cheers,
Dirk

2

I would recommend:

  • remove the close_eof and close_inactive options. Filebeat should be able to detect this use case without them
  • start filebeat in debug mode -d "*" (or level:debug in the config) then post the full log file

I also see an output error in the log, at 8:00. If Logstash is not available for a 3 minutes interval, and the files gets overwritten, then that interval is lost.

3

Hi tudor,

thanks for the info, i removed both and enabled debug, looks like filebeat is taking to long to read the .csv?

2018-07-19T09:54:11.459+0200	DEBUG	[publish]	pipeline/processor.go:291	Publish event: {
  "@timestamp": "2018-07-19T07:54:11.459Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.1"
  },
  "offset": 21350,
  "message": "717,dbo,YieldSortingType,16",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "logstash": "table_counts",
  "beat": {
    "name": "ServerY",
    "hostname": "ServerY",
    "version": "6.3.1"
  },
  "host": {
    "name": "ServerY"
  },
  "source": "D:\\BCP\\table_counts.csv"
}
2018-07-19T09:54:11.459+0200	DEBUG	[harvester]	log/log.go:85	End of file reached: D:\BCP\table_counts.csv; Backoff now.
2018-07-19T09:54:12.460+0200	DEBUG	[harvester]	log/log.go:85	End of file reached: D:\BCP\table_counts.csv; Backoff now.
2018-07-19T09:54:12.460+0200	DEBUG	[transport]	transport/client.go:201	handle error: write tcp 10.10.10.237:60742->10.104.7.27:5044: wsasend: An existing connection was forcibly closed by the remote host.
2018-07-19T09:54:12.460+0200	DEBUG	[logstash]	logstash/async.go:142	7 events out of 7 events sent to logstash host 10.104.7.27:5044. Continue sending
2018-07-19T09:54:12.461+0200	DEBUG	[logstash]	logstash/async.go:99	close connection
2018-07-19T09:54:12.461+0200	DEBUG	[transport]	transport/client.go:114	closing
2018-07-19T09:54:12.461+0200	ERROR	logstash/async.go:235	Failed to publish events caused by: write tcp 10.10.10.237:60742->10.104.7.27:5044: wsasend: An existing connection was forcibly closed by the remote host.
2018-07-19T09:54:12.461+0200	DEBUG	[logstash]	logstash/async.go:99	close connection
2018-07-19T09:54:13.463+0200	ERROR	pipeline/output.go:92	Failed to publish events: write tcp 10.10.10.237:60742->10.104.7.27:5044: wsasend: An existing connection was forcibly closed by the remote host.
2018-07-19T09:54:13.463+0200	DEBUG	[logstash]	logstash/async.go:94	connect
2018-07-19T09:54:13.465+0200	DEBUG	[logstash]	logstash/async.go:142	7 events out of 7 events sent to logstash host 10.104.7.27:5044. Continue sending
2018-07-19T09:54:13.466+0200	DEBUG	[memqueue]	memqueue/ackloop.go:143	ackloop: receive ack [1: 0, 7]
2018-07-19T09:54:13.479+0200	DEBUG	[memqueue]	memqueue/eventloop.go:518	broker ACK events: count=7, start-seq=713, end-seq=719

2018-07-19T09:54:13.479+0200	DEBUG	[memqueue]	memqueue/ackloop.go:111	ackloop: return ack to broker loop:7
2018-07-19T09:54:13.479+0200	DEBUG	[memqueue]	memqueue/ackloop.go:114	ackloop:  done send ack
2018-07-19T09:54:13.479+0200	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 7}
2018-07-19T09:54:13.479+0200	DEBUG	[registrar]	registrar/registrar.go:236	Processing 7 events
2018-07-19T09:54:13.479+0200	DEBUG	[registrar]	registrar/registrar.go:206	Registrar state updates processed. Count: 7
2018-07-19T09:54:13.479+0200	DEBUG	[registrar]	registrar/registrar.go:291	Write registry file: C:\ProgramData\filebeat\registry
2018-07-19T09:54:13.480+0200	DEBUG	[registrar]	registrar/registrar.go:284	Registry file updated. 1 states written.
2018-07-19T09:54:14.466+0200	DEBUG	[harvester]	log/log.go:85	End of file reached: D:\BCP\carrental_table_counts.csv; Backoff now.
2018-07-19T09:54:16.437+0200	DEBUG	[input]	input/input.go:125	Run input

any idea?

Thanks for the help.

Cheers,
Dirk

4

There seems to be some sort of network issue between Logstash and Filebeat, which can make Filebeat slow down on the reading part. Do you see any errors in the logstash logs? Is there any middle box between filebeat and logstash?

5

yea, there were some issues, i moved it to a different host, looks more stable from the connection, but i don't get filebeat to read each file new if overwritten.

so i just created a script to remove the old files, that a new will be created each time.

think that's fine for now, only have these issue with filebeat on windows :confused:

6

Ah, I missed that this happens on Windows. Yes, if that script works for you it sounds like a good solution. Summoning @ruflin in case he has an idea of what could go wrong here.

7

Can you share what you exactly mean by "overwritten"? Is it a new file or the existing file is truncated and the new content is added?

8

overwritten = truncated and new content added, with my script i remove the files and then its working, but without script would be better :smiley:

9

The problem here is the truncation I think. If the size of a file increases, FB assumes new content was added. If it decreases, FB assumes it's a new file and starts from scratch. In your case I assume old and new content are almost identical in size so Filebeat in some cases doesn't even know the content of the file changed.

Could you somehow change your log creation script that it does create a new file instead of truncating the original one?

10

yea, i changed the output to add the current timestamp, then i get every time a new file with a unique name.

now it's working, but was hoping for a wait without including an script to delete old files on the windows host.

thanks for the help and input, hope that others with the same problem can get help with the thread

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.