.index .svclog


(Kunal Jha) #1

Hi
we've setup new elk server in our environment and its working well for text formatted logs.
but the blocker is we've few servers generating .svclog formatted logs and we can filter them even can ship then using filebeat from client to ELK server.
please help me to solve this
and the 2nd question is. is ELK works with this kind or (.svclog) log format.


(Mark Walkom) #2

What are .svclogs?


(Kunal Jha) #3

its a log format for web trace kind of logs ?
somewhere i found it belongs to .xml family but my elk works for xml. only svclog format is not working


(Kunal Jha) #4

can we expect help ??


(Christian Dahlqvist) #5

If you want help please show an example of what the logs look like and do not assume that this is a well known format. Also show what you have tried so far and the result of that.


(Kunal Jha) #6

i have tried to index it with normal .txt filters and also tried with .xml filter. in filebeat logs it show
in filebeat logs as
( 2017-06-04T16:15:51Z INFO Home path: [C:\Program Files\filebeat] Config path: [C:\Program Files\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Program Files\filebeat\logs]
2017-06-04T16:15:51Z INFO Setup Beat: filebeat; Version: 5.4.0
2017-06-04T16:15:51Z INFO Max Retries set to: 3
2017-06-04T16:15:51Z INFO Activated logstash as output plugin.
2017-06-04T16:15:51Z INFO Publisher name: WIN-CE7FJ4QU6SH
2017-06-04T16:15:51Z INFO Flush Interval set to: 1s
2017-06-04T16:15:51Z INFO Max Bulk Size set to: 2048
2017-06-04T16:15:51Z INFO filebeat start running.
2017-06-04T16:15:51Z INFO Registry file set to: C:\ProgramData\filebeat\registry
2017-06-04T16:15:51Z INFO Loading registrar data from C:\ProgramData\filebeat\registry
2017-06-04T16:15:51Z INFO States Loaded from registrar: 345
2017-06-04T16:15:51Z INFO Loading Prospectors: 1
2017-06-04T16:15:51Z INFO Starting Registrar
2017-06-04T16:15:51Z INFO Start sending events to output
2017-06-04T16:15:51Z INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-06-04T16:15:51Z INFO Prospector with previous states loaded: 338
2017-06-04T16:15:51Z INFO Starting prospector of type: log; id: 16986528915901883283
2017-06-04T16:15:51Z INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService11_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\CLIQPreparationService22_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService_trac.log
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\BusinessService22_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\SecurityTokenService_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService_trac.xml
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\CLIQPreparationService11_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\SecurityTokenService11_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService22_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\BusinessService11_trace.svclog
2017-06-04T16:15:52Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\SecurityTokenService22_trace.svclog
2017-06-04T16:15:52Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\BusinessService_trace.svclog
2017-06-04T16:15:52Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\CLIQPreparationService_trace.svclog
2017-06-04T16:16:21Z INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=14 filebeat.harvester.running=14 filebeat.harvester.started=14 publish.events=352 registrar.states.current=345 registrar.states.update=352 registrar.writes=1
2017-06-04T16:16:51Z INFO No non-zero metrics in the last 30s
2017-06-04T16:17:21Z INFO No non-zero metrics in the last 30s
2017-06-04T16:17:51Z INFO No non-zero metrics in the last 30s
2017-06-04T16:17:52Z INFO Stopping filebeat
2017-06-04T16:17:52Z INFO Prospector channel stopped because beat is stopping.
2017-06-04T16:17:52Z INFO Stopping Crawler
2017-06-04T16:17:52Z INFO Stopping 1 prospectors
2017-06-04T16:17:52Z INFO Prospector ticker stopped
2017-06-04T16:17:52Z INFO Stopping Prospector: 16986528915901883283
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\CLIQPreparationService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService_trac.xml. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\CLIQPreparationService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService_trac.log. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\BusinessService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\SecurityTokenService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\SecurityTokenService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\BusinessService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\SecurityTokenService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\CLIQPreparationService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\BusinessService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Crawler stopped
2017-06-04T16:17:52Z INFO Stopping spooler
2017-06-04T16:17:52Z INFO Stopping Registrar
2017-06-04T16:17:52Z INFO Ending Registrar
2017-06-04T16:17:52Z INFO Total non-zero values: filebeat.harvester.closed=14 filebeat.harvester.started=14 publish.events=352 registrar.states.current=345 registrar.states.update=352 registrar.writes=2
2017-06-04T16:17:52Z INFO Uptime: 2m1.0889532s
2017-06-04T16:17:52Z INFO filebeat stopped.)
but we can see any log on elk server.


(Christian Dahlqvist) #7

What does the logs you are trying to process look like?


(Kunal Jha) #8

is there any option to attach sample log. as i saw i only can upload jpg png files


(Christian Dahlqvist) #9

You can create a gist and link to it here. There are also other sites that allow you to upload and share text.


(Kunal Jha) #10