Filebeat is reading my logs, but logs with date time on name, stop incremet

Elastic Stack Beats
1

Hello folks!

I'm new on FileBeat, Logstash and Elastiscsearch. I'm trying to implement in my work, but I'm having problem with FileBeat.

When I start the service from FileBeat, my logs with date time on name, stop increment.

For example, logs like that:

App20151218.Log

I did searches on the internet, but I couldn't solve this issue.

Follow my FileBeat config:

filebeat:
prospectors:
-
paths:
- /var/log/pfswf01.log
- /var/log/pfswf02.log
- /var/log/pfswf03.log
- /var/log/pfswf04.log
document_type: firewall
-
paths:
- /var/log/syslog-ng.log
document_type: syslog
-
paths:
- "/mnt/helicon/Notif/.Log"
document_type: urlnotif
-
paths:
- "/mnt/florina_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/alpha_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/daribow_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/livia_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/cygni_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/euterpe_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/korell_LogIIS/logs/LogFiles/W3SVC1/.log"
- "/mnt/nexon_LogIIS/logs/LogFiles/W3SVC1/.log"
document_type: iis
-
paths:
- "/mnt/livia/.Log"
- "/mnt/rhea/.Log"
- "/mnt/cygni/.Log"
- "/mnt/sarip/.Log"
- "/mnt/euterpe/.Log"
- "/mnt/hesperos/.Log"
- "/mnt/daribow/.Log"
- "/mnt/florina/.Log"
- "/mnt/bonde/.Log"
- "/mnt/alpha/.Log"
- "/mnt/korell/.Log"
- "/mnt/vega/.Log"
- "/mnt/cinna/.Log"
- "/mnt/helicon/.Log"
- "/mnt/fomalhaut/.Log"
- "/mnt/gamma/.Log"
- "/mnt/ifni/.Log"
- "/mnt/nexon/*.Log"
document_type: operadoras

   spool_size: 1024
   idle_timeout: 5s
   input_type: log
   fields:
   level: debug
   review: 1
   tail_files: false

output:

logstash:

  hosts: ["10.1.1.112:5044"]
  index: filebeat

file:

path: "/tmp/filebeaat"

filename: filebeat

rotate_every_kb: 10000

number_of_files: 7

logging:

to_syslog: false

to_files: true

files:

path: "/var/log"

name: filebeat.log

rotateeverybytes: 10485760 # = 10MB

keepfiles: 7

selectors: ["*"]
level: error

I appreciate the attemption.

2

In which directory if the file App20151218.Log located?

Are you working with network shares? E.g. file shared from windows system?

3

Hello Steffens,

this file is located in my shared folders on Windows Servers, for example:

/mnt/nexon/App20151218.Log

Thanks for your reply.

4

Hi steffens,

is it possible use something like that on FileBeat configuration?

/mnt/nexon/App{YYYYMMDD}.Log

I tried, but unsuccessfully.

I continue searching, but I did not find nothing.

5

No, this pattern will not work. Just using '/mnt/nexon/App*.Log' or '/mnt/nexon/*.Log' should do the trick.

6

Windows Server network share might be a problem. If possible try to have filebeat running on server directly.

  1. depending on log rotation strategy, rotation might fail due to open files 'blocking' renamings on windows. Consider setting force_close_files to true. This option was introduced for windows. But network shares might complicate matters.

  2. When using network shares between different kinds of systems, it might break file identification. e.g. samba might randomly generate file ids if systems are incompatible.

You can try inode being stable on linux using 'ls -i <filename>'. multiple times. If inode changes, check for alternative mount options.

All in all I would not try to use any log forwarders via windows/network shares if possible.

7

Hi Steffens,

thanks a lot for your reply.

I will check with my team, it can be possible install FileBeat directly on ours Windows Servers.

Thanks for the tips.