.index .svclog

Hi
we've setup new elk server in our environment and its working well for text formatted logs.
but the blocker is we've few servers generating .svclog formatted logs and we can filter them even can ship then using filebeat from client to ELK server.
please help me to solve this
and the 2nd question is. is ELK works with this kind or (.svclog) log format.

What are .svclogs?

its a log format for web trace kind of logs ?
somewhere i found it belongs to .xml family but my elk works for xml. only svclog format is not working

can we expect help ??

If you want help please show an example of what the logs look like and do not assume that this is a well known format. Also show what you have tried so far and the result of that.

i have tried to index it with normal .txt filters and also tried with .xml filter. in filebeat logs it show
in filebeat logs as
( 2017-06-04T16:15:51Z INFO Home path: [C:\Program Files\filebeat] Config path: [C:\Program Files\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\Program Files\filebeat\logs]
2017-06-04T16:15:51Z INFO Setup Beat: filebeat; Version: 5.4.0
2017-06-04T16:15:51Z INFO Max Retries set to: 3
2017-06-04T16:15:51Z INFO Activated logstash as output plugin.
2017-06-04T16:15:51Z INFO Publisher name: WIN-CE7FJ4QU6SH
2017-06-04T16:15:51Z INFO Flush Interval set to: 1s
2017-06-04T16:15:51Z INFO Max Bulk Size set to: 2048
2017-06-04T16:15:51Z INFO filebeat start running.
2017-06-04T16:15:51Z INFO Registry file set to: C:\ProgramData\filebeat\registry
2017-06-04T16:15:51Z INFO Loading registrar data from C:\ProgramData\filebeat\registry
2017-06-04T16:15:51Z INFO States Loaded from registrar: 345
2017-06-04T16:15:51Z INFO Loading Prospectors: 1
2017-06-04T16:15:51Z INFO Starting Registrar
2017-06-04T16:15:51Z INFO Start sending events to output
2017-06-04T16:15:51Z INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-06-04T16:15:51Z INFO Prospector with previous states loaded: 338
2017-06-04T16:15:51Z INFO Starting prospector of type: log; id: 16986528915901883283
2017-06-04T16:15:51Z INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService11_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\CLIQPreparationService22_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService_trac.log
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\BusinessService22_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\SecurityTokenService_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService_trac.xml
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\CLIQPreparationService11_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\SecurityTokenService11_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\AdministratorService22_trace.svclog
2017-06-04T16:15:51Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\BusinessService11_trace.svclog
2017-06-04T16:15:52Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\SecurityTokenService22_trace.svclog
2017-06-04T16:15:52Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\BusinessService_trace.svclog
2017-06-04T16:15:52Z INFO Harvester started for file: D:\Heimdal\HeimdalTraces\CLIQPreparationService_trace.svclog
2017-06-04T16:16:21Z INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=14 filebeat.harvester.running=14 filebeat.harvester.started=14 publish.events=352 registrar.states.current=345 registrar.states.update=352 registrar.writes=1
2017-06-04T16:16:51Z INFO No non-zero metrics in the last 30s
2017-06-04T16:17:21Z INFO No non-zero metrics in the last 30s
2017-06-04T16:17:51Z INFO No non-zero metrics in the last 30s
2017-06-04T16:17:52Z INFO Stopping filebeat
2017-06-04T16:17:52Z INFO Prospector channel stopped because beat is stopping.
2017-06-04T16:17:52Z INFO Stopping Crawler
2017-06-04T16:17:52Z INFO Stopping 1 prospectors
2017-06-04T16:17:52Z INFO Prospector ticker stopped
2017-06-04T16:17:52Z INFO Stopping Prospector: 16986528915901883283
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\CLIQPreparationService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService_trac.xml. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\CLIQPreparationService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService_trac.log. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\BusinessService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\SecurityTokenService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\SecurityTokenService11_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\BusinessService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\SecurityTokenService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\CLIQPreparationService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\BusinessService22_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Reader was closed: D:\Heimdal\HeimdalTraces\AdministratorService_trace.svclog. Closing.
2017-06-04T16:17:52Z INFO Crawler stopped
2017-06-04T16:17:52Z INFO Stopping spooler
2017-06-04T16:17:52Z INFO Stopping Registrar
2017-06-04T16:17:52Z INFO Ending Registrar
2017-06-04T16:17:52Z INFO Total non-zero values: filebeat.harvester.closed=14 filebeat.harvester.started=14 publish.events=352 registrar.states.current=345 registrar.states.update=352 registrar.writes=2
2017-06-04T16:17:52Z INFO Uptime: 2m1.0889532s
2017-06-04T16:17:52Z INFO filebeat stopped.)
but we can see any log on elk server.

What does the logs you are trying to process look like?

is there any option to attach sample log. as i saw i only can upload jpg png files

You can create a gist and link to it here. There are also other sites that allow you to upload and share text.


here is link

When looking at the log it seem like the XML events are not separated by a newline, which could make them difficult to parse without some form of preprocessing or custom codec. If you managed to separate out the events, you should however be able to parse them using the xml filter. I rarely parse XML data, so have very limited experience with this filter.

that is not working that's only works fine with .XML logs

Do we have any plugin available to convert log format.

If I separate out a single event from the file as follows it seems like the xml filter is able to parse it, so you should be able to use the xml filter to parse this, possibly together with some other filters depending on what you want the resulting events to look like.

<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent"><System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system"><EventID>0</EventID><Type>3</Type><SubType Name="Information">0</SubType><Level>8</Level><TimeCreated SystemTime="2017-05-02T14:16:53.4792899Z" /><Source Name="HeimdalAdministratorService.Traces" /><Correlation ActivityID="{684c237a-d5a2-4f18-91b1-0c2fe18dc3c6}" /><Execution ProcessName="w3wp" ProcessID="7072" ThreadID="19" /><Channel/><Computer>WIN-CE7FJ4QU6SH</Computer></System><ApplicationData>Service Version = 1.3.1.1744</ApplicationData></E2ETraceEvent>

I am not aware of any plugin that would allow you to read a full file though, as there does not seem to be any newline separating the events. You may therefore need to pre-process the file or perhaps even create a custom codec plugin.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.