Upload and parse exported .evtx files to Elasticsearch

Elastic Stack Beats
1

I have a use case scenario where I have to manually upload and parse Windows logs to Elasticsearch by using exported .evtx files. Splunk handles this fine with the "oneshot" command and I was wondering if anyone in this forum found a similar solution with Winlogbeat or tools that is utilizing Winlogbeat?

I know that the latest version of Kibana supports upload of log files, however there is a limit to the size of files that can be uploaded, which in my case is not sufficient.

The best workround I found so far is using this tool https://dragos.com/blog/industry-news/evtxtoelk-a-python-module-to-load-windows-event-logs-into-elasticsearch/

The only issue I have with this tool, from testing so far, is that the field names are not the same as of those that Winlogbeat generates, making it inconsistent with existing dashboards etc.

2

I have pull request in the works. Please read the FAQ section added in the pull request to see if this would address your needs.

1 Like
3

Thank you, that sounds promising! I didn't think of checking out the pull requests on Github :sweat_smile:

4

Hi @andrewkroh,

This would be very helpful for our environment.
I have tried sending the logs to Elasticsearch as per your example in FAQ Section. But I'm getting the below error.

PS C:\Users\Sanjay\Downloads\softwares\winlogbeat> .\winlogbeat.exe -e -c .\winlogbeat-evtx.yml -E EVTX_FILE=D:/Hardware Events.evtx 2019-04-12T18:19:59.310+0530 INFO instance/beat.go:616 Home path: [C:\Users\Sanjay\Downloads\softwares\winlogbe at] Config path: [C:\Users\Sanjay\Downloads\softwares\winlogbeat] Data path: [C:\Users\Sanjay\Downloads\softwares\winlog beat\data] Logs path: [C:\Users\Sanjay\Downloads\softwares\winlogbeat\logs] 2019-04-12T18:19:59.315+0530 INFO instance/beat.go:623 Beat UUID: 42abafb0-d761-43c1-b73c-5edd3ed0c0e0 2019-04-12T18:19:59.316+0530 INFO [beat] instance/beat.go:936 Beat info {"system_info": {"beat": {"path" : {"config": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat", "data": "C:\\Users\\Sanjay\\Downloads\\softwares\\wi nlogbeat\\data", "home": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat", "logs": "C:\\Users\\Sanjay\\Downloads\\s oftwares\\winlogbeat\\logs"}, "type": "winlogbeat", "uuid": "42abafb0-d761-43c1-b73c-5edd3ed0c0e0"}}} 2019-04-12T18:19:59.320+0530 INFO [beat] instance/beat.go:945 Build info {"system_info": {"build": {"comm it": "2c385a0764bdc537b6dc078a1d9bf11bb6d7bd95", "libbeat": "6.6.0", "time": "2019-01-24T10:45:45.000Z", "version": "6.6 .0"}}} 2019-04-12T18:19:59.323+0530 INFO [beat] instance/beat.go:948 Go runtime info {"system_info": {"go": {"os":"wi ndows","arch":"amd64","max_procs":8,"version":"go1.10.8"}}} 2019-04-12T18:19:59.342+0530 INFO [beat] instance/beat.go:952 Host info {"system_info": {"host": {"archi tecture":"x86_64","boot_time":"2019-03-07T17:48:10.41+05:30","name":"DT-1895","ip":["192.168.6.200/24","::1/128","127.0. 0.1/8","fe80::5efe:c0a8:6c8/128"],"kernel_version":"6.3.9600.19228 (winblue_ltsb.181208-0600)","mac":["64:00:6a:69:f2:9d ","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","ver sion":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19235"},"timezone":"IST","timezone_offset_sec":19800,"id":"741cb 13f-ff74-4cd0-95d1-0470e014726b"}}} 2019-04-12T18:19:59.351+0530 INFO [beat] instance/beat.go:981 Process info {"system_info": {"process": {"cw d": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat", "exe": "C:\\Users\\Sanjay\\Downloads\\softwares\\winlogbeat\\ winlogbeat.exe", "name": "winlogbeat.exe", "pid": 22872, "ppid": 23244, "start_time": "2019-04-12T18:19:59.133+0530"}}} 2019-04-12T18:19:59.354+0530 INFO instance/beat.go:281 Setup Beat: winlogbeat; Version: 6.6.0 2019-04-12T18:19:59.356+0530 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200 2019-04-12T18:19:59.359+0530 INFO [publisher] pipeline/module.go:110 Beat name: DT-1895 2019-04-12T18:19:59.361+0530 INFO beater/winlogbeat.go:68 State will be read from and persisted to C:\Users\Sanjay \Downloads\softwares\winlogbeat\data\evtx-registry.yml 2019-04-12T18:19:59.363+0530 INFO instance/beat.go:403 winlogbeat start running. 2019-04-12T18:19:59.363+0530 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s 2019-04-12T18:19:59.433+0530 WARN beater/eventlogger.go:104 EventLog[D:/HardwareEvents.evtx] Open() error. N o events will be read from this source. The specified channel path is invalid. 2019-04-12T18:19:59.685+0530 INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics ": {"beat":{"cpu":{"system":{"ticks":109,"time":{"ms":109}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{ "ticks":62,"time":{"ms":62}}},"handles":{"open":186},"info":{"ephemeral_id":"e981ca91-3851-4d98-addd-0f335028e138","upti me":{"ms":212}},"memstats":{"gc_next":4194304,"memory_alloc":2015584,"memory_total":3541432,"rss":17444864}},"libbeat":{ "config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"sys tem":{"cpu":{"cores":8}}}}} 2019-04-12T18:19:59.691+0530 INFO [monitoring] log/log.go:153 Uptime: 468.5434ms 2019-04-12T18:19:59.692+0530 INFO [monitoring] log/log.go:130 Stopping metrics logging. 2019-04-12T18:19:59.693+0530 INFO instance/beat.go:413 winlogbeat stopped.

I'm not sure if I'm missing something... I have tried changing the paths but still no help... :frowning:

Please help me in resolving this issue..

Thanks,
Dharma Sanjay Reddy M.

5

Sort of, this is a work in progress so the feature hasn't been released yet.

6

Thanks for the quick response @andrewkroh

This is great work and we are eagerly waiting for this feature to be released :slight_smile:

So, is there any resolution for this type of error as of now?

Thanks,
Dharma Sanjay Reddy M.

7

You can download a snapshot build from here just to test it out. (It's not released yet.)

https://beats-ci.elastic.co/job/elastic+beats+master+multijob-package-linux/lastSuccessfulBuild/gcsObjects/

Direct link: https://storage.googleapis.com/beats-ci-artifacts/snapshots/winlogbeat/winlogbeat-oss-8.0.0-SNAPSHOT-windows-x86_64.zip

8

Thank you so much @andrewkroh
This worked like a charm :slight_smile:

Now am able to index the archived logs. Will work on it till the official release of this feature.

Thanks,
Dharma Sanjay Reddy M.

9

Thank you for the feedback.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.