Filebeat is reading my logs, but logs with date time on name, stop incremet


(name less) #1

Hello folks!

I'm new on FileBeat, Logstash and Elastiscsearch. I'm trying to implement in my work, but I'm having problem with FileBeat.

When I start the service from FileBeat, my logs with date time on name, stop increment.

For example, logs like that:

App20151218.Log

I did searches on the internet, but I couldn't solve this issue.

Follow my FileBeat config:

filebeat:
prospectors:
-
paths:
- /var/log/pfswf01.log
- /var/log/pfswf02.log
- /var/log/pfswf03.log
- /var/log/pfswf04.log
document_type: firewall
-
paths:
- /var/log/syslog-ng.log
document_type: syslog
-
paths:
- "/mnt/helicon/Notif/.Log"
document_type: urlnotif
-
paths:
- "/mnt/florina_LogIIS/logs/LogFiles/W3SVC1/
.log"
- "/mnt/alpha_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/daribow_LogIIS/logs/LogFiles/W3SVC1/
.log"
- "/mnt/livia_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/cygni_LogIIS/logs/LogFiles/W3SVC1/
.log"
- "/mnt/euterpe_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/korell_LogIIS/logs/LogFiles/W3SVC1/
.log"
- "/mnt/nexon_LogIIS/logs/LogFiles/W3SVC1/.log"
document_type: iis
-
paths:
- "/mnt/livia/
.Log"
- "/mnt/rhea/.Log"
- "/mnt/cygni/
.Log"
- "/mnt/sarip/.Log"
- "/mnt/euterpe/
.Log"
- "/mnt/hesperos/.Log"
- "/mnt/daribow/
.Log"
- "/mnt/florina/.Log"
- "/mnt/bonde/
.Log"
- "/mnt/alpha/.Log"
- "/mnt/korell/
.Log"
- "/mnt/vega/.Log"
- "/mnt/cinna/
.Log"
- "/mnt/helicon/.Log"
- "/mnt/fomalhaut/
.Log"
- "/mnt/gamma/.Log"
- "/mnt/ifni/
.Log"
- "/mnt/nexon/*.Log"
document_type: operadoras

   spool_size: 1024
   idle_timeout: 5s
   input_type: log
   fields:
   level: debug
   review: 1
   tail_files: false   

output:

logstash:

  hosts: ["10.1.1.112:5044"]
  index: filebeat

file:

path: "/tmp/filebeaat"

filename: filebeat

rotate_every_kb: 10000

number_of_files: 7

logging:

to_syslog: false

to_files: true

files:

path: "/var/log"

name: filebeat.log

rotateeverybytes: 10485760 # = 10MB

keepfiles: 7

selectors: ["*"]
level: error

I appreciate the attemption.


(Steffen Siering) #2

In which directory if the file App20151218.Log located?

Are you working with network shares? E.g. file shared from windows system?


(name less) #3

Hello Steffens,

this file is located in my shared folders on Windows Servers, for example:

/mnt/nexon/App20151218.Log

Thanks for your reply.


(name less) #4

Hi steffens,

is it possible use something like that on FileBeat configuration?

/mnt/nexon/App{YYYYMMDD}.Log

I tried, but unsuccessfully.

I continue searching, but I did not find nothing.


(Steffen Siering) #5

No, this pattern will not work. Just using '/mnt/nexon/App*.Log' or '/mnt/nexon/*.Log' should do the trick.


(Steffen Siering) #6

Windows Server network share might be a problem. If possible try to have filebeat running on server directly.

  1. depending on log rotation strategy, rotation might fail due to open files 'blocking' renamings on windows. Consider setting force_close_files to true. This option was introduced for windows. But network shares might complicate matters.

  2. When using network shares between different kinds of systems, it might break file identification. e.g. samba might randomly generate file ids if systems are incompatible.

You can try inode being stable on linux using 'ls -i <filename>'. multiple times. If inode changes, check for alternative mount options.

All in all I would not try to use any log forwarders via windows/network shares if possible.


(name less) #7

Hi Steffens,

thanks a lot for your reply.

I will check with my team, it can be possible install FileBeat directly on ours Windows Servers.

Thanks for the tips.


(system) #8