Upload and parse exported .evtx files to Elasticsearch

I have a use case scenario where I have to manually upload and parse Windows logs to Elasticsearch by using exported .evtx files. Splunk handles this fine with the "oneshot" command and I was wondering if anyone in this forum found a similar solution with Winlogbeat or tools that is utilizing Winlogbeat?

I know that the latest version of Kibana supports upload of log files, however there is a limit to the size of files that can be uploaded, which in my case is not sufficient.

The best workround I found so far is using this tool https://dragos.com/blog/industry-news/evtxtoelk-a-python-module-to-load-windows-event-logs-into-elasticsearch/

The only issue I have with this tool, from testing so far, is that the field names are not the same as of those that Winlogbeat generates, making it inconsistent with existing dashboards etc.

I have pull request in the works. Please read the FAQ section added in the pull request to see if this would address your needs.

1 Like

Thank you, that sounds promising! I didn't think of checking out the pull requests on Github :sweat_smile:

Hi @andrewkroh,

This would be very helpful for our environment.
I have tried sending the logs to Elasticsearch as per your example in FAQ Section. But I'm getting the below error.

PS C:\Users\Sanjay\Downloads\softwares\winlogbeat> .\winlogbeat.exe -e -c .\winlogbeat-evtx.yml -E EVTX_FILE=D:/Hardware Events.evtx 2019-04-12T18:19:59.310+0530 INFO instance/beat.go:616 Home path: [C:\Users\Sanjay\Downloads\softwares\winlogbe at] Config path: [C:\Users\Sanjay\Downloads\softwares\winlogbeat] Data path: [C:\Users\Sanjay\Downloads\softwares\winlog beat\data] Logs path: [C:\Users\Sanjay\Downloads\softwares\winlogbeat\logs] 2019-04-12T18:19:59.315+0530 INFO instance/beat.go:623 Beat UUID: 42abafb0-d761-43c1-b73c-5edd3ed0c0e0 2019-04-12T18:19:59.316+0530 INFO [beat] instance/beat.go:936 Beat info {"system_info": {"beat": {"path" : {"config": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat", "data": "C:\\Users\\Sanjay\\Downloads\\softwares\\wi nlogbeat\\data", "home": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat", "logs": "C:\\Users\\Sanjay\\Downloads\\s oftwares\\winlogbeat\\logs"}, "type": "winlogbeat", "uuid": "42abafb0-d761-43c1-b73c-5edd3ed0c0e0"}}} 2019-04-12T18:19:59.320+0530 INFO [beat] instance/beat.go:945 Build info {"system_info": {"build": {"comm it": "2c385a0764bdc537b6dc078a1d9bf11bb6d7bd95", "libbeat": "6.6.0", "time": "2019-01-24T10:45:45.000Z", "version": "6.6 .0"}}} 2019-04-12T18:19:59.323+0530 INFO [beat] instance/beat.go:948 Go runtime info {"system_info": {"go": {"os":"wi ndows","arch":"amd64","max_procs":8,"version":"go1.10.8"}}} 2019-04-12T18:19:59.342+0530 INFO [beat] instance/beat.go:952 Host info {"system_info": {"host": {"archi tecture":"x86_64","boot_time":"2019-03-07T17:48:10.41+05:30","name":"DT-1895","ip":["","::1/128","127.0. 0.1/8","fe80::5efe:c0a8:6c8/128"],"kernel_version":"6.3.9600.19228 (winblue_ltsb.181208-0600)","mac":["64:00:6a:69:f2:9d ","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","ver sion":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19235"},"timezone":"IST","timezone_offset_sec":19800,"id":"741cb 13f-ff74-4cd0-95d1-0470e014726b"}}} 2019-04-12T18:19:59.351+0530 INFO [beat] instance/beat.go:981 Process info {"system_info": {"process": {"cw d": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat", "exe": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat\\ winlogbeat.exe", "name": "winlogbeat.exe", "pid": 22872, "ppid": 23244, "start_time": "2019-04-12T18:19:59.133+0530"}}} 2019-04-12T18:19:59.354+0530 INFO instance/beat.go:281 Setup Beat: winlogbeat; Version: 6.6.0 2019-04-12T18:19:59.356+0530 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200 2019-04-12T18:19:59.359+0530 INFO [publisher] pipeline/module.go:110 Beat name: DT-1895 2019-04-12T18:19:59.361+0530 INFO beater/winlogbeat.go:68 State will be read from and persisted to C:\Users\Sanjay \Downloads\softwares\winlogbeat\data\evtx-registry.yml 2019-04-12T18:19:59.363+0530 INFO instance/beat.go:403 winlogbeat start running. 2019-04-12T18:19:59.363+0530 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s 2019-04-12T18:19:59.433+0530 WARN beater/eventlogger.go:104 EventLog[D:/HardwareEvents.evtx] Open() error. N o events will be read from this source. The specified channel path is invalid. 2019-04-12T18:19:59.685+0530 INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics ": {"beat":{"cpu":{"system":{"ticks":109,"time":{"ms":109}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{ "ticks":62,"time":{"ms":62}}},"handles":{"open":186},"info":{"ephemeral_id":"e981ca91-3851-4d98-addd-0f335028e138","upti me":{"ms":212}},"memstats":{"gc_next":4194304,"memory_alloc":2015584,"memory_total":3541432,"rss":17444864}},"libbeat":{ "config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"sys tem":{"cpu":{"cores":8}}}}} 2019-04-12T18:19:59.691+0530 INFO [monitoring] log/log.go:153 Uptime: 468.5434ms 2019-04-12T18:19:59.692+0530 INFO [monitoring] log/log.go:130 Stopping metrics logging. 2019-04-12T18:19:59.693+0530 INFO instance/beat.go:413 winlogbeat stopped.

I'm not sure if I'm missing something... I have tried changing the paths but still no help... :frowning:

Please help me in resolving this issue..

Dharma Sanjay Reddy M.

Sort of, this is a work in progress so the feature hasn't been released yet.

Thanks for the quick response @andrewkroh

This is great work and we are eagerly waiting for this feature to be released :slight_smile:

So, is there any resolution for this type of error as of now?

Dharma Sanjay Reddy M.

You can download a snapshot build from here just to test it out. (It's not released yet.)


Direct link: https://storage.googleapis.com/beats-ci-artifacts/snapshots/winlogbeat/winlogbeat-oss-8.0.0-SNAPSHOT-windows-x86_64.zip

Thank you so much @andrewkroh
This worked like a charm :slight_smile:

Now am able to index the archived logs. Will work on it till the official release of this feature.

Dharma Sanjay Reddy M.

Thank you for the feedback.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.