Manually upload Winlogbeat ndjson files to Elasticsearch

Greetings and salutations. I'm looking for a way to manually upload locally generated Winlogbeat (8.1.3) ndjson files to my Elastic stack as the beat has to work offline and cannot connect directly to either ES or Logstash.

I haven't found much in the way of recent scripts and what few I've tried unfortunately did not work.

The only thing that did work was a direct file upload through the Data Visualizer, but this method is limited to one file at a time and there is a 100MB limit per file.

Any tricks/tips?

Hello @fl33t

you can try with logstash for this case without beats also. please see the below syntax which would help.

SYNTAX:

input
{ 
file 
{
path =>"<path with filename>"
start_position =>"beginning"
}
}

output
{
elasticsearch
{
       hosts => "<your-es-url>"
       user => "<your-es-username>"
       password => "<your-es-pwd>"
       index => "<preferred-index-name>"
}
}

Keep Posted !!! Thanks !!! :slight_smile:

Thanks @sudhagar_ramesh !

I'll definitely try it and I do believe it might even fix another problem that crept up.

I ended up trying the very same thing with Filebeat. First I gathered the logs on the offline system with Winlogbeat, then shipped them to an ES-connected one with Filebeat installed and then off to my stack. The process worked inasmuch as all my logs got shipped to ES, but I didn't manage to send them to the right pipeline so they didn't get parsed correctly. To be more precise, they were simply stored whole with Filebeat meta attached. Either that, or I was doing it wrong.

Tl;dr, Logstash might just help steer the data in the right direction. I'll give it a go!

As for my Filebeat attempt, as i mentioned I was probably doing it wrong but in case it helps someone here's how I modified filebeat.yml -- everything else is stock.

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - c:\\path\\to\\collected_data\\*.ndjson

setup.template.settings:
  index.number_of_shards: 1
setup.template.name: "winlogbeat"
setup.template.pattern: "winlogbeat"
setup.template.override: false

output.elasticsearch:
  hosts: ["x.x.x.x:9200"]
  protocol: "https"
  ssl.enabled: true
  ssl.verification_mode: none
  username: "USERNAME"
  password: "PASSWORD"
  pipeline: winlogbeat-8.1.3-routing
  index: "my-winlogbeat-offline-data"

I figured that this way, the data would get routed to the proper pipelines further down the road.

Perhaps I should have sent it directly to Winlogbeat's security data pipeline instead (as I'd only collected from that specific log) or used "winlogbeat-8.1.3" for both the template pattern and the name as it already existed -- plus override was off.

We'll see what happens with logstash. :slight_smile:

1 Like

Unfortunately it looks like my experiment was not quite successful. Though Logstash seems to have had a healthy run, no data ended up in my stack.

Essentially, I downloaded/unzipped Logstash on the same system that previously hosted Filebeat, configured it for basic file upload and then pressed go. Here are my configs (everything else is stock).

pipelines.yml

- pipeline.id: main
  path.config: "C:\\path_to\logstash-8.1.3\\pipelines\\logstash.conf"

logstash.conf

input {
  file {
    path => "C:\path_to\offline_data\*"
	start_position => "beginning"
  }
}

output {
  elasticsearch {
    hosts => ["https://x.x.x.x:9200"]
	ssl => true
	ssl_certificate_verification => false
	user => "user"
    password => "password"
    index => "my-offline-data-index"
  }
}

And logstash-plain.log

[2022-06-03T11:20:57,386][INFO ][logstash.runner          ] Log4j configuration path used is: C:\path_to\logstash-8.1.3\config\log4j2.properties
[2022-06-03T11:20:57,397][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.1.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [mswin32-x86_64]"}
[2022-06-03T11:20:57,397][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-06-03T11:21:00,101][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-03T11:21:02,121][INFO ][org.reflections.Reflections] Reflections took 204 ms to scan 1 urls, producing 120 keys and 419 values 
[2022-06-03T11:21:04,611][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-06-03T11:21:04,845][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://x.x.x.x:9200"]}
[2022-06-03T11:21:04,954][WARN ][logstash.outputs.elasticsearch][main] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure remove `ssl_certificate_verification => false`
[2022-06-03T11:21:05,813][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://xxxxxx:xxxxxx@x.x.x.x:9200/]}}
[2022-06-03T11:21:06,871][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://xxxxxx:xxxxxx@x.x.x.x:9200/"}
[2022-06-03T11:21:06,902][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.1.3) {:es_version=>8}
[2022-06-03T11:21:06,918][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-06-03T11:21:07,108][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-03T11:21:07,108][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-06-03T11:21:07,514][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-06-03T11:21:07,561][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/path_to/logstash-8.1.3/pipelines/logstash.conf"], :thread=>"#<Thread:0x143cd477 run>"}
[2022-06-03T11:21:09,398][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.82}
[2022-06-03T11:21:10,244][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/path_to/logstash-8.1.3/data/plugins/inputs/file/.sincedb_ed318eee0ec05e040659bf5e0ca0db53", :path=>["C:\\path_to\\offline_data\\*"]}
[2022-06-03T11:21:10,291][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-03T11:21:10,556][INFO ][filewatch.observingtail  ][main][d722414a2dde756e112aae4fcfd629c45e7af6332013c004ab177bb0cf1994c0] START, creating Discoverer, Watch with file and sincedb collections
[2022-06-03T11:21:10,572][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-06-03T11:40:15,512][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2022-06-03T11:40:15,774][INFO ][filewatch.observingtail  ] QUIT - closing all files and shutting down.

I tried two types of upload.

  1. path => "C:\path_to\offline_data\*.ndjson"
    Where I only specified the ndjson files.

  2. path => "C:\path_to\offline_data\*" (the line you see in my logstash.conf and the log)
    Where I hoped that in addition to the ndjson files, Winlogbeat's original .winlogbeat.yml and meta.json might help.

Any ideas @sudhagar_ramesh ? Thanks again!

Hello @fl33t

i have tried the same way like yours and the data is not getting ingested.
Hence, please do the following

  1. Comment out or remove the pipelines.id and path.config we don't need to reference our "logstash.conf" in pipelines.yml

  2. just execute the logstash.conf file using the below command and its working fine.

Command:
logstash -f <file-path>

Example:
logstash -f G:\xxx\xxx\logstash.conf

code snippet

input
{
file 
{
path =>"G:/XXXX/XXX/sample_log.ndjson"
start_position =>"beginning"
}
}

output
{
stdout
{
codec => rubydebug
}

elasticsearch
{
    hosts => "localhost:9200"
	user => "<your-es-username>"
	password => "<your-es-pwd>"
	index => "ndjson_try"
}
}

Apologies for the delay and much thanks for the reply @sudhagar_ramesh !

Downloaded clean zip of logstash-oss 8.1.3, then created the custom logstash.conf file (no other files were modified, so pipelines.yml was entirely commented).

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  file {
    path => "C:\beats\raw_data\*.ndjson"
    start_position => "beginning" 
  }
}
output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["https://x.x.x.x:9200"]
    ssl => true
    ssl_certificate_verification => false
    user => "USER"
    password => "PASSWORD"
    index => "raw_data_test"
  }
}

Logstash seems to start correctly.

PS C:\beats\logstash-oss-8.1.3\bin> ./logstash.bat -f C:\beats\logstash-oss-8.1.3\config\logstash.conf
"Using bundled JDK: C:\beats\logstash-oss-8.1.3\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/beats/logstash-oss-8.1.3/logs which is now configured via log4j2.properties
[2022-06-10T14:57:29,314][INFO ][logstash.runner          ] Log4j configuration path used is: C:\beats\logstash-oss-8.1.3\config\log4j2.properties
[2022-06-10T14:57:29,332][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.1.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [mswin32-x86_64]"}
[2022-06-10T14:57:29,332][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-06-10T14:57:29,515][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-10T14:57:30,851][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-10T14:57:32,272][INFO ][org.reflections.Reflections] Reflections took 196 ms to scan 1 urls, producing 120 keys and 419 values
[2022-06-10T14:57:34,541][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-06-10T14:57:34,602][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://x.x.x.x:9200"]}
[2022-06-10T14:57:34,651][WARN ][logstash.outputs.elasticsearch][main] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure remove `ssl_certificate_verification => false`
[2022-06-10T14:57:35,071][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://xxxxxx:xxxxxx@x.x.x.x:9200/]}}
[2022-06-10T14:57:35,796][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://xxxxxx:xxxxxx@x.x.x.x:9200/"}
[2022-06-10T14:57:35,812][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.1.3) {:es_version=>8}
[2022-06-10T14:57:35,829][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-06-10T14:57:35,897][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-10T14:57:35,929][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-10T14:57:35,929][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-06-10T14:57:35,985][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-06-10T14:57:36,092][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/beats/logstash-oss-8.1.3/config/logstash.conf"], :thread=>"#<Thread:0x3c9e1919 run>"}
[2022-06-10T14:57:37,225][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.02}
[2022-06-10T14:57:37,664][INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/beats/logstash-oss-8.1.3/data/plugins/inputs/file/.sincedb_b07f85c6921adc7a0bc9cbc9d1bcb122", :path=>["C:\\beats\\raw_data\\*.ndjson"]}
[2022-06-10T14:57:37,693][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-10T14:57:37,782][INFO ][filewatch.observingtail  ][main][9f3442113a3d7f380db8afddc9aff06038282bebff5057d51a81f2b087b1dbe1] START, creating Discoverer, Watch with file and sincedb collections
[2022-06-10T14:57:37,814][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

And yet nothing seems to have been sent to my ES stack. To be sure, I even stopped the existing logstash service that was running on the server before the attempt, but no "raw_data_test" index anywhere.

Could it have something to do with these two lines perhaps? And yes, I am fishing a little. :slight_smile:

[2022-06-10T14:57:35,897][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-10T14:57:35,929][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`

In any case, I did learn something about logstash on Windows. Originally, I'd put my custom config file in "C:\beats\logstash-oss-8.1.3\bin\fileconf" in order to avoid having to write the config file's full path. Well, it seems that on Windows, the following command, run directly from the "bin" folder

./logstash.bat -f ./fileconf/logstash.conf

Resolves to

C:\beats\logstash-oss-8.1.3\fileconf\logstash.conf

Which does not contain the file. The more you know.

1 Like

Dear @fl33t

Thanks a lot for your reply.

I am glad that it works :smiley:

Have a great day!!! :slight_smile:

1 Like