Filebeat didn't parse log file

Slop · April 6, 2018, 9:44am

I've created new log files recently, but I didn't succeed to having them harvest by filebeat.
Currently filebeat haverst two tomcat log files.

Here's my filebeat.yml :

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - \\ateur03\Tomcat03$\base\logs\tomcat.log
    - \\ateur04\Tomcat04$\base\logs\tomcat.log
    - D:/95-23_EU/07-Working-area/Statistiques/2018-04-05-UserDetails.log

I've added the path to the log file on "D:" repository.

It didn't work, so I've read filebeat and logstash logs, and my file isn't prospected. But it appear that filebeat is reading registry file on ProgramaData and not on the install repository.
I've change both registry to add my path (the one on programData and the second one on my install repository), and it still didn't work.

How can I change that ? Should I install a new filebeat service which'll use registry file from install repo ?

Thanks,
Kind regards.

ruflin · April 6, 2018, 12:58pm

I think we have several issues in this post, lets sort them out one by one. Lets start with the harvesting part.

To see if your logs are harvested or not it would be great if you could share log file from Filebeat. Then we should be able to see if the files are not found or perhaps something goes wrong during sending. Best enable debug level for the logging, then we have all the details.

Slop · April 6, 2018, 1:18pm

Hi, thanks for your help.

I've enable "debug" level and it speak a lot more

Here is the log of filebeat :

2018-04-06T15:14:50+02:00 DBG Disable stderr logging
2018-04-06T15:14:50+02:00 INFO Home path: [C:\Filebeat] Config path: [C:\Filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Filebeat\logs]
2018-04-06T15:14:50+02:00 INFO Setup Beat: filebeat; Version: 5.4.1
2018-04-06T15:14:50+02:00 DBG Processors:
2018-04-06T15:14:50+02:00 DBG Initializing output plugins
2018-04-06T15:14:50+02:00 INFO Loading template enabled. Reading template file: C:\Filebeat\filebeat.template.json
2018-04-06T15:14:50+02:00 INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\Filebeat\filebeat.template-es2x.json
2018-04-06T15:14:50+02:00 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: C:\Filebeat\filebeat.template-es6x.json
2018-04-06T15:14:50+02:00 INFO Elasticsearch url: http://localhost:9200
2018-04-06T15:14:50+02:00 INFO Activated elasticsearch as output plugin.
2018-04-06T15:14:50+02:00 INFO Max Retries set to: 3
2018-04-06T15:14:50+02:00 INFO Activated logstash as output plugin.
2018-04-06T15:14:50+02:00 DBG Create output worker
2018-04-06T15:14:50+02:00 DBG Create output worker
2018-04-06T15:14:50+02:00 DBG No output is defined to store the topology. The server fields might not be filled.
2018-04-06T15:14:50+02:00 INFO Publisher name: FREPPAU-AQTT156
2018-04-06T15:14:50+02:00 INFO Flush Interval set to: 1s
2018-04-06T15:14:50+02:00 INFO Max Bulk Size set to: 50
2018-04-06T15:14:50+02:00 DBG create bulk processing worker (interval=1s, bulk size=50)
2018-04-06T15:14:50+02:00 INFO Flush Interval set to: 1s
2018-04-06T15:14:50+02:00 INFO Max Bulk Size set to: 2048
2018-04-06T15:14:50+02:00 DBG create bulk processing worker (interval=1s, bulk size=2048)
2018-04-06T15:14:50+02:00 INFO filebeat start running.
2018-04-06T15:14:50+02:00 DBG Windows is interactive: false
2018-04-06T15:14:50+02:00 INFO Registry file set to: C:\Filebeat\data\registry
2018-04-06T15:14:50+02:00 INFO Loading registrar data from C:\Filebeat\data\registry
2018-04-06T15:14:50+02:00 INFO States Loaded from registrar: 6
2018-04-06T15:14:50+02:00 INFO Loading Prospectors: 1
2018-04-06T15:14:50+02:00 INFO Starting Registrar
2018-04-06T15:14:50+02:00 INFO Start sending events to output
2018-04-06T15:14:50+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-04-06T15:14:50+02:00 DBG File Configs: [\freppau-apeur03\Tomcat03$\base\logs\tomcat.log \freppau-apeur04\Tomcat04$\base\logs\tomcat.log W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/2018-04-05-UserDetails.log]
2018-04-06T15:14:50+02:00 DBG exclude_files:
2018-04-06T15:14:50+02:00 DBG New state added for \freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG New state added for W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/2018-04-05-UserDetails.log
2018-04-06T15:14:50+02:00 DBG New state added for \freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG New state added for \freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG New state added for \freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG New state added for \freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 INFO Prospector with previous states loaded: 6
2018-04-06T15:14:50+02:00 INFO Starting prospector of type: log; id: 17531284131367634480
2018-04-06T15:14:50+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018-04-06T15:14:50+02:00 DBG Start next scan
2018-04-06T15:14:50+02:00 DBG Check file for harvesting: \freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG Update existing file for harvesting: \freppau-apeur03\Tomcat03$\base\logs\tomcat.log, offset: 4963399
2018-04-06T15:14:50+02:00 DBG File didn't change: \freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG Check file for harvesting: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG Update existing file for harvesting: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log, offset: 29139231
2018-04-06T15:14:50+02:00 DBG Resuming harvesting of file: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log, offset: 29139231
2018-04-06T15:14:50+02:00 DBG Set previous offset for file: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log. Offset: 29139231
2018-04-06T15:14:50+02:00 DBG Setting offset for file: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log. Offset: 29139231
2018-04-06T15:14:50+02:00 DBG Prospector states cleaned up. Before: 6, After: 6
2018-04-06T15:14:50+02:00 INFO Harvester started for file: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-06T15:14:50+02:00 DBG Remove state for file as file removed: W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/2018-04-05-UserDetails.log
2018-04-06T15:14:50+02:00 DBG End of file reached: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log; Backoff now.
2018-04-06T15:14:51+02:00 DBG End of file reached: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log; Backoff now.
2018-04-06T15:14:53+02:00 DBG End of file reached: \freppau-apeur04\Tomcat04$\base\logs\tomcat.log; Backoff now.
2018-04-06T15:14:55+02:00 DBG Flushing spooler because of timeout. Events flushed: 26

ruflin · April 9, 2018, 7:21am

Can you share your full Filebeat config?

Now rereading your initial post, did you manually modify the registry file?

Does it work as expected for the tomcat files but not the file on directory D? The file seems to show up in the log file but with a very different path (W?) Is the file a symlink?

Slop · April 9, 2018, 8:23am

Yes I've modify te registry file.
It always work for tomcat files, but it didn't with the new one wich is in a different location (W:).

Here's my filebeat.yml :

> ###################### Filebeat Configuration Example #########################
> 
> # This file is an example configuration file highlighting only the most common
> # options. The filebeat.full.yml file from the same directory contains all the
> # supported options with more comments. You can use it as a reference.
> #
> # You can find the full configuration reference here:
> # https://www.elastic.co/guide/en/beats/filebeat/index.html
> 
> #=========================== Filebeat prospectors =============================
> filebeat.registry_file: C:\Filebeat\data\registry
> filebeat.prospectors:
> 
> # Each - is a prospector. Most options can be set at the prospector level, so
> # you can use different prospectors for various configurations.
> # Below are the prospector specific configurations.
> 
> - input_type: log
> 
>   # Paths that should be crawled and fetched. Glob based paths.
>   paths:
>     #- /var/log/*.log
>     - \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
>     - \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
>     - W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/2018-04-05-UserDetails.log
>   # Exclude lines. A list of regular expressions to match. It drops the lines that are
>   # matching any regular expression from the list.
>   #exclude_lines: ["^DBG"]
> 
>   # Include lines. A list of regular expressions to match. It exports the lines that are
>   # matching any regular expression from the list.
>   #include_lines: ["^ERR", "^WARN"]
> 
>   # Exclude files. A list of regular expressions to match. Filebeat drops the files that
>   # are matching any regular expression from the list. By default, no files are dropped.
>   #exclude_files: [".gz$"]
> 
>   # Optional additional fields. These field can be freely picked
>   # to add additional information to the crawled log files for filtering
>   #fields:
>   #  level: debug
>   #  review: 1
> 
>   ### Multiline options
> 
>   # Mutiline can be used for log messages spanning multiple lines. This is common
>   # for Java Stack Traces or C-Line Continuation
> 
>   # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
>   #multiline.pattern: ^\[
> 
>   # Defines if the pattern set under pattern should be negated or not. Default is false.
>   #multiline.negate: false
> 
>   # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
>   # that was (not) matched before or after or as long as a pattern is not matched based on negate.
>   # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
>   #multiline.match: after
> 
> 
> #================================ General =====================================
> 
> # The name of the shipper that publishes the network data. It can be used to group
> # all the transactions sent by a single shipper in the web interface.
> #name:
> 
> # The tags of the shipper are included in their own field with each
> # transaction published.
> #tags: ["service-X", "web-tier"]
> 
> # Optional fields that you can specify to add additional information to the
> # output.
> #fields:
> #  env: staging
> 
> #================================ Outputs =====================================
> 
> # Configure what outputs to use when sending the data collected by the beat.
> # Multiple outputs may be used.
> 
> #-------------------------- Elasticsearch output ------------------------------
> output.elasticsearch:
>   # Array of hosts to connect to.
>   hosts: ["localhost:9200"]
> 
>   # Optional protocol and basic auth credentials.
>   #protocol: "https"
>   #username: "elastic"
>   #password: "changeme"
> 
> #----------------------------- Logstash output --------------------------------
> output.logstash:
>   # The Logstash hosts
>   hosts: ["localhost:5044"]
>   template.name: "filebeat"
>   template.path: "filebeat.template.json"
>   template.overwrite: false
>   # Optional SSL. By default is off.
>   # List of root certificates for HTTPS server verifications
>   #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
> 
>   # Certificate for SSL client authentication
>   #ssl.certificate: "/etc/pki/client/cert.pem"
> 
>   # Client Certificate Key
>   #ssl.key: "/etc/pki/client/cert.key"
> 
> #================================ Logging =====================================
> 
> # Sets log level. The default log level is info.
> # Available log levels are: critical, error, warning, info, debug
> logging.level: debug
> 
> # At debug level, you can selectively enable logging only for some components.
> # To enable all selectors use ["*"]. Examples of other selectors are "beat",
> # "publish", "service".
> #logging.selectors: ["*"]

ruflin · April 10, 2018, 8:15am

I think we need to take one step back here. Can you explain why you modified the registry file manually? What did you exactly modify?

harshbajaj16 · April 10, 2018, 8:34am

Hi,

i observer in your config file you've enabled both elasticsearch and logstash o/p section.

we can use one section at a time try with this solution it may be help you.

Thanks,
Harsh Bajaj

Slop · April 10, 2018, 9:26am

I tried to modify it to add the path to my new file, but it didn"t work, here's the file actually :

[{"source":"\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log","offset":680894,"FileStateOS":{"idxhi":209518592,"idxlo":874,"vol":3859991179},"timestamp":"2018-04-06T15:14:55.1206099+02:00","ttl":-1},{"source":"\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log","offset":7884608,"FileStateOS":{"idxhi":75759616,"idxlo":14913,"vol":3859991179},"timestamp":"2018-04-06T15:14:55.1206099+02:00","ttl":-1},{"source":"\\freppau-apeur04\Tomcat04$\base\logs\tomcat.log","offset":14888719,"FileStateOS":{"idxhi":32440320,"idxlo":241926,"vol":1947368156},"timestamp":"2018-04-06T15:14:55.1206099+02:00","ttl":-1},{"source":"\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log","offset":7253130,"FileStateOS":{"idxhi":107085824,"idxlo":32139,"vol":3859991179},"timestamp":"2018-04-06T23:05:54.5133669+02:00","ttl":-1},{"source":"\\freppau-apeur04\Tomcat04$\base\logs\tomcat.log","offset":40848880,"FileStateOS":{"idxhi":657588224,"idxlo":24432,"vol":1947368156},"timestamp":"2018-04-06T20:53:23.1481041+02:00","ttl":-1},{"source":"\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log","offset":713684,"FileStateOS":{"idxhi":95813632,"idxlo":35159,"vol":3859991179},"timestamp":"2018-04-07T23:05:48.7151875+02:00","ttl":-1},{"source":"\\freppau-apeur04\Tomcat04$\base\logs\tomcat.log","offset":2984661,"FileStateOS":{"idxhi":218628096,"idxlo":35130,"vol":1947368156},"timestamp":"2018-04-07T15:32:24.1590115+02:00","ttl":-1},{"source":"\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log","offset":364613,"FileStateOS":{"idxhi":28114944,"idxlo":14990,"vol":3859991179},"timestamp":"2018-04-08T23:05:55.2584548+02:00","ttl":-1},{"source":"\\freppau-apeur04\Tomcat04$\base\logs\tomcat.log","offset":1177499,"FileStateOS":{"idxhi":1727332352,"idxlo":981,"vol":1947368156},"timestamp":"2018-04-08T17:31:56.6359215+02:00","ttl":-1},{"source":"\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log","offset":11973623,"FileStateOS":{"idxhi":127795200,"idxlo":14932,"vol":3859991179},"timestamp":"2018-04-09T23:09:45.7819686+02:00","ttl":-1},{"source":"\\freppau-apeur04\Tomcat04$\base\logs\tomcat.log","offset":52562737,"FileStateOS":{"idxhi":393871360,"idxlo":891,"vol":1947368156},"timestamp":"2018-04-09T14:38:14.8962142+02:00","ttl":-1},{"source":"\\freppau-apeur04\Tomcat04$\base\logs\tomcat.log","offset":43356221,"FileStateOS":{"idxhi":897646592,"idxlo":171933,"vol":1947368156},"timestamp":"2018-04-09T23:09:50.7828647+02:00","ttl":-1},{"source":"\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log","offset":4681874,"FileStateOS":{"idxhi":302383104,"idxlo":19461,"vol":3859991179},"timestamp":"2018-04-10T11:22:49.5180795+02:00","ttl":-1},{"source":"\\freppau-apeur04\Tomcat04$\base\logs\tomcat.log","offset":25646972,"FileStateOS":{"idxhi":1921384448,"idxlo":887,"vol":1947368156},"timestamp":"2018-04-10T11:17:49.9958407+02:00","ttl":-1}]

ruflin · April 11, 2018, 6:36am

Adding paths to Filebeat does not work through modifying the registry but through the config. In general the registry should never be touched.

What confuses me in this post that the log file paths keep changing. First it was \\ateur03* and D:* in the config, but in the logs we had \freppau-* and W:. Are you trying to migrate log files form one path to an other?

For the dual outputs mentioned above by @harshbajaj16, this is true in 6.x but still works in 5.x even though I would not encourage it.

Slop · April 11, 2018, 9:20am

No no it was just a mistake, path "\freppau- " and "W:" are correct.

I've disable elasticsearch output as mentionned by @harshbajaj16 but still not working.

Should I reset my registry file ? I've a backup file.

ruflin · April 13, 2018, 1:02pm

It would be great if you could start with a fresh non manually modified registry file and then share again the registry and the log output.

Slop · April 16, 2018, 8:04am

Hello,

Here's the registry without any modification :

[{"source":"\\\\freppau-apeur03\\Tomcat03$\\base\\logs\\tomcat.log","offset":680894,"FileStateOS":{"idxhi":209518592,"idxlo":874,"vol":3859991179},"timestamp":"2018-04-13T16:45:38.5274904+02:00","ttl":-1},{"source":"\\\\freppau-apeur04\\Tomcat04$\\base\\logs\\tomcat.log","offset":771892,"FileStateOS":{"idxhi":22872064,"idxlo":160425,"vol":1947368156},"timestamp":"2018-04-13T16:45:38.5274904+02:00","ttl":-1},{"source":"\\\\freppau-apeur03\\Tomcat03$\\base\\logs\\tomcat.log","offset":9216359,"FileStateOS":{"idxhi":75825152,"idxlo":14913,"vol":3859991179},"timestamp":"2018-04-13T23:05:55.2017307+02:00","ttl":-1},{"source":"\\\\freppau-apeur04\\Tomcat04$\\base\\logs\\tomcat.log","offset":18464318,"FileStateOS":{"idxhi":420020224,"idxlo":158653,"vol":1947368156},"timestamp":"2018-04-14T00:05:07.1131422+02:00","ttl":-1},{"source":"\\\\freppau-apeur03\\Tomcat03$\\base\\logs\\tomcat.log","offset":101035,"FileStateOS":{"idxhi":107151360,"idxlo":32139,"vol":3859991179},"timestamp":"2018-04-14T23:05:55.6798521+02:00","ttl":-1},{"source":"\\\\freppau-apeur04\\Tomcat04$\\base\\logs\\tomcat.log","offset":10937517,"FileStateOS":{"idxhi":657653760,"idxlo":24432,"vol":1947368156},"timestamp":"2018-04-15T00:05:06.2436805+02:00","ttl":-1},{"source":"\\\\freppau-apeur03\\Tomcat03$\\base\\logs\\tomcat.log","offset":368808,"FileStateOS":{"idxhi":95879168,"idxlo":35159,"vol":3859991179},"timestamp":"2018-04-15T23:05:45.8425067+02:00","ttl":-1},{"source":"\\\\freppau-apeur04\\Tomcat04$\\base\\logs\\tomcat.log","offset":15846250,"FileStateOS":{"idxhi":218693632,"idxlo":35130,"vol":1947368156},"timestamp":"2018-04-16T00:05:06.4338091+02:00","ttl":-1},{"source":"\\\\freppau-apeur03\\Tomcat03$\\base\\logs\\tomcat.log","offset":488862,"FileStateOS":{"idxhi":28180480,"idxlo":14990,"vol":3859991179},"timestamp":"2018-04-16T10:03:47.7000289+02:00","ttl":-1},{"source":"\\\\freppau-apeur04\\Tomcat04$\\base\\logs\\tomcat.log","offset":5274857,"FileStateOS":{"idxhi":1727397888,"idxlo":981,"vol":1947368156},"timestamp":"2018-04-16T10:03:42.7161448+02:00","ttl":-1}]

Slop · April 16, 2018, 8:06am

and the filbeat logs :

2018-04-16T10:05:41+02:00 INFO Metrics logging every 30s
2018-04-16T10:05:42+02:00 INFO Setup Beat: filebeat; Version: 5.4.1
2018-04-16T10:05:42+02:00 DBG  Processors: 
2018-04-16T10:05:42+02:00 DBG  Initializing output plugins
2018-04-16T10:05:42+02:00 INFO Max Retries set to: 3
2018-04-16T10:05:42+02:00 INFO Activated logstash as output plugin.
2018-04-16T10:05:42+02:00 DBG  Create output worker
2018-04-16T10:05:42+02:00 DBG  No output is defined to store the topology. The server fields might not be filled.
2018-04-16T10:05:42+02:00 INFO Publisher name: FREPPAU-AQTT156
2018-04-16T10:05:42+02:00 INFO Flush Interval set to: 1s
2018-04-16T10:05:42+02:00 INFO Max Bulk Size set to: 2048
2018-04-16T10:05:42+02:00 DBG  create bulk processing worker (interval=1s, bulk size=2048)
2018-04-16T10:05:42+02:00 INFO filebeat start running.
2018-04-16T10:05:42+02:00 DBG  Windows is interactive: false
2018-04-16T10:05:42+02:00 INFO Registry file set to: C:\Filebeat\data\registry
2018-04-16T10:05:42+02:00 INFO Loading registrar data from C:\Filebeat\data\registry
2018-04-16T10:05:42+02:00 INFO States Loaded from registrar: 10
2018-04-16T10:05:42+02:00 INFO Loading Prospectors: 1
2018-04-16T10:05:42+02:00 INFO Starting Registrar
2018-04-16T10:05:42+02:00 INFO Start sending events to output
2018-04-16T10:05:42+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-04-16T10:05:42+02:00 DBG  File Configs: [\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/2018-04-05-UserDetails.log]
2018-04-16T10:05:42+02:00 DBG  exclude_files: []
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 INFO Prospector with previous states loaded: 10
2018-04-16T10:05:42+02:00 INFO Starting prospector of type: log; id: 17531284131367634480 
2018-04-16T10:05:42+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018-04-16T10:05:42+02:00 DBG  Start next scan
2018-04-16T10:05:42+02:00 DBG  Check file for harvesting: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  Update existing file for harvesting: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log, offset: 499504
2018-04-16T10:05:42+02:00 DBG  File didn't change: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  Check file for harvesting: \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  Update existing file for harvesting: \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log, offset: 5325686
2018-04-16T10:05:42+02:00 DBG  File didn't change: \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-16T10:05:42+02:00 DBG  Prospector states cleaned up. Before: 10, After: 10

ruflin · April 19, 2018, 11:03am

Ok, so the file you are missing is W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/2018-04-05-UserDetails.log

Filebeat doesn't find the file so this should not be related to the registry in any way. I assume it's related to how the path is specified and could rely on backslash vs forwardslashes. What is the path shown if you change into the directory and use pwd. Can you try to replace forward with backward slashes as you have for the other files?

Slop · April 19, 2018, 1:09pm

I change it, now my paths :

 paths:
 #- /var/log/*.log
 - \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
 - \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
 - W:\Entity\EXPLO\IGEO\95_Prestations\95-23_EUREKA\07-Working-area\NMT\Statistiques\2018-04-05-UserDetails.log

But still the same problem, is it ok to use path with "W:" ?

I also tried with "W:\" but still the same.

Here's the filebeat log :

2018-04-19T15:07:41+02:00 INFO Metrics logging every 30s
2018-04-19T15:07:41+02:00 INFO Home path: [C:\Filebeat] Config path: [C:\Filebeat] Data path: [C:\\ProgramData\\filebeat] Logs path: [C:\Filebeat\logs]
2018-04-19T15:07:41+02:00 INFO Setup Beat: filebeat; Version: 5.4.1
2018-04-19T15:07:41+02:00 DBG  Processors: 
2018-04-19T15:07:41+02:00 DBG  Initializing output plugins
2018-04-19T15:07:41+02:00 INFO Max Retries set to: 3
2018-04-19T15:07:41+02:00 INFO Activated logstash as output plugin.
2018-04-19T15:07:41+02:00 DBG  Create output worker
2018-04-19T15:07:41+02:00 DBG  No output is defined to store the topology. The server fields might not be filled.
2018-04-19T15:07:41+02:00 INFO Publisher name: FREPPAU-AQTT156
2018-04-19T15:07:41+02:00 INFO Flush Interval set to: 1s
2018-04-19T15:07:41+02:00 INFO Max Bulk Size set to: 2048
2018-04-19T15:07:41+02:00 DBG  create bulk processing worker (interval=1s, bulk size=2048)
2018-04-19T15:07:41+02:00 INFO filebeat start running.
2018-04-19T15:07:41+02:00 DBG  Windows is interactive: false
2018-04-19T15:07:41+02:00 INFO Registry file set to: C:\Filebeat\data\registry
2018-04-19T15:07:41+02:00 INFO Loading registrar data from C:\Filebeat\data\registry
2018-04-19T15:07:41+02:00 INFO States Loaded from registrar: 16
2018-04-19T15:07:41+02:00 INFO Loading Prospectors: 1
2018-04-19T15:07:41+02:00 INFO Start sending events to output
2018-04-19T15:07:41+02:00 INFO Starting Registrar
2018-04-19T15:07:41+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-04-19T15:07:41+02:00 DBG  File Configs: [\\freppau-apeur03\Tomcat03$\base\logs\tomcat.log \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log W:\\Entity\EXPLO\IGEO\95_Prestations\95-23_EUREKA\07-Working-area\NMT\Statistiques\2018-04-05-UserDetails.log]
2018-04-19T15:07:41+02:00 DBG  exclude_files: []
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  New state added for \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 INFO Prospector with previous states loaded: 16
2018-04-19T15:07:41+02:00 INFO Starting prospector of type: log; id: 14070566624406119334 
2018-04-19T15:07:41+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018-04-19T15:07:41+02:00 DBG  Start next scan
2018-04-19T15:07:41+02:00 DBG  Check file for harvesting: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  Update existing file for harvesting: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log, offset: 4694402
2018-04-19T15:07:41+02:00 DBG  Resuming harvesting of file: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log, offset: 4694402
2018-04-19T15:07:41+02:00 DBG  Set previous offset for file: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log. Offset: 4694402 
2018-04-19T15:07:41+02:00 DBG  Setting offset for file: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log. Offset: 4694402 
2018-04-19T15:07:41+02:00 DBG  Check file for harvesting: \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 INFO Harvester started for file: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  Update existing file for harvesting: \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log, offset: 22566377
2018-04-19T15:07:41+02:00 DBG  File didn't change: \\freppau-apeur04\Tomcat04$\base\logs\tomcat.log
2018-04-19T15:07:41+02:00 DBG  End of file reached: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log; Backoff now.
2018-04-19T15:07:41+02:00 DBG  Prospector states cleaned up. Before: 16, After: 16
2018-04-19T15:07:42+02:00 DBG  End of file reached: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log; Backoff now.
2018-04-19T15:07:44+02:00 DBG  End of file reached: \\freppau-apeur03\Tomcat03$\base\logs\tomcat.log; Backoff now.
2018-04-19T15:07:46+02:00 DBG  Flushing spooler because of timeout. Events flushed: 37

ruflin · April 19, 2018, 1:58pm

What does windows tell you if you run pwd inside the directory?

Slop · April 19, 2018, 2:04pm

it tell : W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/

ruflin · April 20, 2018, 2:56pm

Hm, this is strange. Could you try "W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/" or 'W:/Entity/EXPLO/IGEO/95_Prestations/95-23_EUREKA/07-Working-area/NMT/Statistiques/' for the path definition?

Is W: any special device?

Slop · May 4, 2018, 9:27am

Sorry I was absent for a while.

There's no changes with " or ', it didn't work.

The W: is a network directory.

ruflin · May 7, 2018, 7:50am

Any chance you could install Filebeat directly on the machine and try it again instead of using a network file system? We do not recommend to use Filebeat with network drives.