so brief recap:
logstash starts
logstash reads pipelines
logstash fails on output section using Elasticsearch connection
logstash runs on output section using file output
curl to Elasticsearch address on 9200 succeeds
as a quick test, i threw up a new elastic server on a non hardened ubuntu 20.0 with minimal settings, pointed this logstash server at it...
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,626][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.0.60.63:9200"]}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,628][DEBUG][logstash.outputs.elasticsearch][main] Normalizing http path {:path=>nil, :normalized=>nil}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,640][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.0.60.63:9200/]}}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,641][DEBUG][logstash.outputs.elasticsearch][main] Running health check to see if an ES connection is working {:url=>"http://10.0.60.63:9200/", :path=>"/"}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,648][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://10.0.60.63:9200/"}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,660][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.15.1) {:es_version=>7}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,664][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,746][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,847][DEBUG][logstash.outputs.elasticsearch][main] Attempting to install template {:template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,864][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,871][DEBUG][logstash.outputs.elasticsearch][main] Attempting to install template {:template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,935][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"logstash"}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,953][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"logstash"}
Nov 03 10:46:17 soctest001 logstash[50425]: [2021-11-03T10:46:17,102][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
Nov 03 10:46:17 soctest001 logstash[50425]: [2021-11-03T10:46:17,117][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
Nov 03 10:46:17 soctest001 logstash[50425]: [2021-11-03T10:46:17,126][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/test.conf"], :thread=>"#<Thread:0x5424236d run>"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,175][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional
Nov 03 10:46:18 soctest001 logstash[50425]: [if (event.getField('[type]')=='wds-metricbeat-input')]
Nov 03 10:46:18 soctest001 logstash[50425]: into
Nov 03 10:46:18 soctest001 logstash[50425]: org.logstash.config.ir.compiler.ComputeStepSyntaxElement@9fb449bc
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,177][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional
Nov 03 10:46:18 soctest001 logstash[50425]: [if (event.getField('[type]')=='wds-metricbeat-input')]
Nov 03 10:46:18 soctest001 logstash[50425]: into
Nov 03 10:46:18 soctest001 logstash[50425]: org.logstash.config.ir.compiler.ComputeStepSyntaxElement@9fb449bc
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,348][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]: P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"ecs-metricbeat-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:13:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]: hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: # user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: # password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]: index => "ecs-metricbeat-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]: }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]: into
Nov 03 10:46:18 soctest001 logstash[50425]: org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,403][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]: P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"ecs-metricbeat-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:13:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]: hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: # user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: # password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]: index => "ecs-metricbeat-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]: }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]: into
Nov 03 10:46:18 soctest001 logstash[50425]: org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,423][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]: P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:21:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]: hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: # user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: # password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]: index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]: }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]: into
Nov 03 10:46:18 soctest001 logstash[50425]: org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,431][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]: P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:21:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]: hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: # user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: # password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]: index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]: }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]: into
Nov 03 10:46:18 soctest001 logstash[50425]: org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,454][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.31}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,479][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:2598"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,504][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,512][DEBUG][logstash.javapipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5424236d run>"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,536][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.