Apologies for the delay and much thanks for the reply @sudhagar_ramesh !
Downloaded clean zip of logstash-oss 8.1.3, then created the custom logstash.conf file (no other files were modified, so pipelines.yml was entirely commented).
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
file {
path => "C:\beats\raw_data\*.ndjson"
start_position => "beginning"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["https://x.x.x.x:9200"]
ssl => true
ssl_certificate_verification => false
user => "USER"
password => "PASSWORD"
index => "raw_data_test"
}
}
Logstash seems to start correctly.
PS C:\beats\logstash-oss-8.1.3\bin> ./logstash.bat -f C:\beats\logstash-oss-8.1.3\config\logstash.conf
"Using bundled JDK: C:\beats\logstash-oss-8.1.3\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/beats/logstash-oss-8.1.3/logs which is now configured via log4j2.properties
[2022-06-10T14:57:29,314][INFO ][logstash.runner ] Log4j configuration path used is: C:\beats\logstash-oss-8.1.3\config\log4j2.properties
[2022-06-10T14:57:29,332][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.1.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [mswin32-x86_64]"}
[2022-06-10T14:57:29,332][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-06-10T14:57:29,515][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-10T14:57:30,851][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-06-10T14:57:32,272][INFO ][org.reflections.Reflections] Reflections took 196 ms to scan 1 urls, producing 120 keys and 419 values
[2022-06-10T14:57:34,541][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-06-10T14:57:34,602][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://x.x.x.x:9200"]}
[2022-06-10T14:57:34,651][WARN ][logstash.outputs.elasticsearch][main] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure remove `ssl_certificate_verification => false`
[2022-06-10T14:57:35,071][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://xxxxxx:xxxxxx@x.x.x.x:9200/]}}
[2022-06-10T14:57:35,796][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://xxxxxx:xxxxxx@x.x.x.x:9200/"}
[2022-06-10T14:57:35,812][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.1.3) {:es_version=>8}
[2022-06-10T14:57:35,829][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-06-10T14:57:35,897][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-10T14:57:35,929][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-10T14:57:35,929][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-06-10T14:57:35,985][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-06-10T14:57:36,092][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["C:/beats/logstash-oss-8.1.3/config/logstash.conf"], :thread=>"#<Thread:0x3c9e1919 run>"}
[2022-06-10T14:57:37,225][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.02}
[2022-06-10T14:57:37,664][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/beats/logstash-oss-8.1.3/data/plugins/inputs/file/.sincedb_b07f85c6921adc7a0bc9cbc9d1bcb122", :path=>["C:\\beats\\raw_data\\*.ndjson"]}
[2022-06-10T14:57:37,693][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-10T14:57:37,782][INFO ][filewatch.observingtail ][main][9f3442113a3d7f380db8afddc9aff06038282bebff5057d51a81f2b087b1dbe1] START, creating Discoverer, Watch with file and sincedb collections
[2022-06-10T14:57:37,814][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
And yet nothing seems to have been sent to my ES stack. To be sure, I even stopped the existing logstash service that was running on the server before the attempt, but no "raw_data_test" index anywhere.
Could it have something to do with these two lines perhaps? And yes, I am fishing a little. 
[2022-06-10T14:57:35,897][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-06-10T14:57:35,929][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
In any case, I did learn something about logstash on Windows. Originally, I'd put my custom config file in "C:\beats\logstash-oss-8.1.3\bin\fileconf" in order to avoid having to write the config file's full path. Well, it seems that on Windows, the following command, run directly from the "bin" folder
./logstash.bat -f ./fileconf/logstash.conf
Resolves to
C:\beats\logstash-oss-8.1.3\fileconf\logstash.conf
Which does not contain the file. The more you know.
