20:04:30.359 [qtp1453606810-20] INFO AuditInterceptor - [13pn9kdr5tl84stzkmaa8vmg] end /web/v1/project/fhp4clxfbu0w3ym2n7ee6ynh/statistics?executionName=bayes_poc_test GET, 187 ms
And, I have another question.
When I modify the log file,all records will be published to logstash, not the latest record.
There are two kinds of logs. one:
2018/01/12 02:00:54.002115 harvester.go:215: INFO Harvester started for file: /tmp/INFO_17_01_11.log
2018/01/12 02:00:54.002558 processor.go:275: DBG [publish] Publish event: {log content}
2018/01/12 02:00:54.002558 processor.go:275: DBG [publish] Publish event: {log content}
2018/01/12 02:00:54.002558 processor.go:275: DBG [publish] Publish event: {log content}
... 2018/01/12 02:00:55.009330 async.go:235: ERR Failed to publish events caused by: write tcp [::1]:61579->[::1]:5044: write: broken pipe 2018/01/12 02:00:56.014211 output.go:92: ERR Failed to publish events: write tcp [::1]:61579->[::1]:5044: write: broken pipe
2018/01/12 02:01:23.515368 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30001 beat.memstats.gc_next=5635296 beat.memstats.memory_alloc=2855728 beat.memstats.memory_total=10677992 filebeat.events.added=11 filebeat.events.done=11 filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.config.module.running=0 libbeat.output.events.acked=9 libbeat.output.events.batches=2 libbeat.output.events.failed=9 libbeat.output.events.total=18 libbeat.output.read.bytes=6 libbeat.output.write.bytes=596 libbeat.output.write.errors=1 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=0 libbeat.pipeline.events.filtered=2 libbeat.pipeline.events.published=9 libbeat.pipeline.events.retry=18 libbeat.pipeline.events.total=11 libbeat.pipeline.queue.acked=9 registrar.states.cleanup=1 registrar.states.current=1 registrar.states.update=11 registrar.writes=3
Hello, its because filebeat tracks inode changes to know if it needs to read the file or not. When you are editing with vim, every time that you save your file the inode change. Look at the number on the left in the following example.
ph@sashimi /tmp ls -li testing.log
4305672956 -rw-r--r-- 1 ph wheel 21 Jan 22 10:05 o.log
ph@sashimi /tmp vim o.log
ph@sashimi /tmp ls -li testing.log
4305672969 -rw-r--r-- 1 ph wheel 30 Jan 22 10:05 o.log
ph@sashimi /tmp
When you are using echo, the inode won't change, this is the normal behavior when you are logging, you are appending to a file.
ph@sashimi /tmp ls -li echo.log
4305673010 -rw-r--r-- 1 ph wheel 2 Jan 22 10:06 echo.log
ph@sashimi /tmp echo "no inode change" >> echo.log
ph@sashimi /tmp ls -li echo.log
4305673010 -rw-r--r-- 1 ph wheel 18 Jan 22 10:06 echo.log
I am new to ELK, I found the reason why this error happens.
You can add "client_inactivity_timeout => "1200"" to
input {
beats {
port => "5044"
}
}
The default value for this option is 60.
Hope it helps.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.