Offset mssql logs

[GedeoN (Jonathan Kulitza)]

Hi,

I'm working on a solution to send mssql logs to Elasticsearch.
The global solution is working, but i don't have solution to only send the difference.

In thirst time, I copy and encoding mssql logs files in an other folder.
And I use filebeat to send these files to logstash.

filebeat configuration:

filebeat.inputs:
  
- type: log
  encoding: UTF-8
  enabled: true
  paths:
    - E:/sources/Log/FDLAUNCHERRORLOG*
  tags: ["FDLAUNCHERRORLOGPPD"]
  scan_frequency: 120s
  fields:
    log_type: fdlauncherrorlogppd
  close_inactive: 3600m
  
- type: log
  encoding: UTF-8
  enabled: true
  paths:
    - E:/sources/Log/SQLAGENT*
  tags: ["SQLAGENTPPD"]
  scan_frequency: 120s
  fields:
    log_type: sqlagentppd
  close_inactive: 3600m
  
- type: log
  encoding: UTF-8
  enabled: true
  paths:
    - E:/sources/Log/ERRORLOG*
  tags: ["ERRORLOGPPD"]
  scan_frequency: 120s
  fields:
    log_type: errorlogppd
  close_inactive: 3600m

My problem, it always copying all files, and i just would have the difference after copy.
Many files are archives, so i can send them manually for one time.
But ERRORLOG always evolved, and i would to send just the new lines on file and not all.

Is any solution exist to have an offset on file?
Sorry for my English, i hope it's understandable.

Best regards.
Jonathan

[kvch (Noémi Ványi)]

Are the log files you are reading located on a network share? What copies the files?

To exclude archives you can use the exclude_files option to filter out files with a specific name or extension. See more: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-exclude-files