Logstash output creating error

BeMoore · November 3, 2021, 10:47am

so brief recap:

logstash starts
logstash reads pipelines
logstash fails on output section using Elasticsearch connection
logstash runs on output section using file output
curl to Elasticsearch address on 9200 succeeds

as a quick test, i threw up a new elastic server on a non hardened ubuntu 20.0 with minimal settings, pointed this logstash server at it...


Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,626][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.0.60.63:9200"]}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,628][DEBUG][logstash.outputs.elasticsearch][main] Normalizing http path {:path=>nil, :normalized=>nil}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,640][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.0.60.63:9200/]}}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,641][DEBUG][logstash.outputs.elasticsearch][main] Running health check to see if an ES connection is working {:url=>"http://10.0.60.63:9200/", :path=>"/"}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,648][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://10.0.60.63:9200/"}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,660][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.15.1) {:es_version=>7}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,664][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,746][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,847][DEBUG][logstash.outputs.elasticsearch][main] Attempting to install template {:template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,864][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,871][DEBUG][logstash.outputs.elasticsearch][main] Attempting to install template {:template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,935][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"logstash"}
Nov 03 10:46:16 soctest001 logstash[50425]: [2021-11-03T10:46:16,953][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"logstash"}
Nov 03 10:46:17 soctest001 logstash[50425]: [2021-11-03T10:46:17,102][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
Nov 03 10:46:17 soctest001 logstash[50425]: [2021-11-03T10:46:17,117][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
Nov 03 10:46:17 soctest001 logstash[50425]: [2021-11-03T10:46:17,126][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/test.conf"], :thread=>"#<Thread:0x5424236d run>"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,175][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional
Nov 03 10:46:18 soctest001 logstash[50425]:  [if (event.getField('[type]')=='wds-metricbeat-input')]
Nov 03 10:46:18 soctest001 logstash[50425]:  into
Nov 03 10:46:18 soctest001 logstash[50425]:  org.logstash.config.ir.compiler.ComputeStepSyntaxElement@9fb449bc
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,177][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled conditional
Nov 03 10:46:18 soctest001 logstash[50425]:  [if (event.getField('[type]')=='wds-metricbeat-input')]
Nov 03 10:46:18 soctest001 logstash[50425]:  into
Nov 03 10:46:18 soctest001 logstash[50425]:  org.logstash.config.ir.compiler.ComputeStepSyntaxElement@9fb449bc
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,348][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]:  P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"ecs-metricbeat-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:13:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]:           hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: #          user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: #          password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]:           index => "ecs-metricbeat-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]:         }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]:  into
Nov 03 10:46:18 soctest001 logstash[50425]:  org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,403][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]:  P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"ecs-metricbeat-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:13:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]:           hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: #          user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: #          password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]:           index => "ecs-metricbeat-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]:         }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]:  into
Nov 03 10:46:18 soctest001 logstash[50425]:  org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,423][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]:  P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:21:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]:           hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: #          user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: #          password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]:           index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]:         }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]:  into
Nov 03 10:46:18 soctest001 logstash[50425]:  org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,431][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
Nov 03 10:46:18 soctest001 logstash[50425]:  P[output-elasticsearch{"hosts"=>"http://10.0.60.63:9200", "index"=>"%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"}|[file]/etc/logstash/conf.d/test.conf:21:9:```
Nov 03 10:46:18 soctest001 logstash[50425]: elasticsearch {
Nov 03 10:46:18 soctest001 logstash[50425]:           hosts => "http://10.0.60.63:9200"
Nov 03 10:46:18 soctest001 logstash[50425]: #          user => logstash_system
Nov 03 10:46:18 soctest001 logstash[50425]: #          password => 6EnArfBZ6OZtL2ncpkHQ
Nov 03 10:46:18 soctest001 logstash[50425]:           index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
Nov 03 10:46:18 soctest001 logstash[50425]:         }
Nov 03 10:46:18 soctest001 logstash[50425]: ```]
Nov 03 10:46:18 soctest001 logstash[50425]:  into
Nov 03 10:46:18 soctest001 logstash[50425]:  org.logstash.config.ir.compiler.ComputeStepSyntaxElement@7c6f4279
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,454][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.31}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,479][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:2598"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,504][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,512][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5424236d run>"}
Nov 03 10:46:18 soctest001 logstash[50425]: [2021-11-03T10:46:18,536][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.

BeMoore · November 3, 2021, 10:52am

as you can see.. the config works !!! logstash starts... connects to elastic... all good

So I'm back to square one. CIS hardening. As its a requirement in this environment to secure the server system before placing any application on it, something in CIS is stopping the connectivity portion working on the logstash server.

this is going to present a real problem/challenge in working this through as the logstash system has soooo many components that can be affected.

Has anyone come across a hardening of Linux OS requirement before place any ELK component on it?

leandrojmp · November 3, 2021, 2:36pm

Ok, this does not make much since, I'm curious to see what could be the cause.

Let's try to resume what you did, at least you could be able to open a bug report on Github.

You have one logstash server, named prodlst001, and a Elasticsearch server with the IP address 10.0.60.60, this Elasticsearch has security enabled, but when you try to run your pipeline it gives you a configuration error, right?

So, in your last test you created a new Elasticsearch server with the IP address 10.0.60.63 and changed the output, commenting the user and password lines and changed the IP address of the Elasticsearch output host and then your pipeline worked.

What is server soctest001 that has logstash logs? This is a different server from prodlst001? The logstash that worked was from this server?

If from the same server as before you can connect on one Elasticsearch without security, but when you enabled the user, password lines, logstash throws a configuration error, then there is something very weird.

Based on this I would say that you do not have any connection restrictions for using the Elasticsearch port, since it was able to talk to another server.

I have no idea what could be causing your issue, but here is some things I would try to troubleshoot this.

The first thing it would be to completely remove the output block, save the file, and them add it again editing the file directly on the server, this would make sure that there is no garbage character in the configuration file, which can happens some times.

If that does not work, I would remove the user and password lines, not comment, but completely remove them, and see if the pipeline starts and gives you an authentication error because of the missing credentials, then I would re-add the user and password again.

If it still not working, I would try setting the username and password in the logstash-keystore and try again, if nothing works then I would open a bug report.

BeMoore · November 3, 2021, 3:37pm

You have one logstash server, named prodlst001 , and a Elasticsearch server with the IP address 10.0.60.60 , this Elasticsearch has security enabled, but when you try to run your pipeline it gives you a configuration error, right?

Correct

So, in your last test you created a new Elasticsearch server with the IP address 10.0.60.63 and changed the output, commenting the user and password lines and changed the IP address of the Elasticsearch output host and then your pipeline worked.

Correct

What is server soctest001 that has logstash logs? This is a different server from prodlst001 ? The logstash that worked was from this server?

This is the 10.0.60.63 server. It is a ubuntu 20.04 unhardened with logstash and elasticsearch 7.15.1 installed

If from the same server as before you can connect on one Elasticsearch without security, but when you enabled the user , password lines, logstash throws a configuration error, then there is something very weird.

I can connect from 10.0.60.63 to 10.0.60.60 using the same config with username and password and pipeline runs
i can connect from 10.10.60.61to 10.0.60.63 without username and password and pipeline runs
i cannot connect from 10.0.60.61 to 10.0.60.60 with username and password
i can curl from 10.0.60.61 to 10.0.60.60 with username and password

Based on this I would say that you do not have any connection restrictions for using the Elasticsearch port, since it was able to talk to another server.

Correct

I have no idea what could be causing your issue, but here is some things I would try to troubleshoot this.

The first thing it would be to completely remove the output block, save the file, and them add it again editing the file directly on the server, this would make sure that there is no garbage character in the configuration file, which can happens some times.

Done. I have also changed output from elastic plugin to file output to a flat .txt file and the pipeline runs.

If that does not work, I would remove the user and password lines, not comment, but completely remove them, and see if the pipeline starts and gives you an authentication error because of the missing credentials, then I would re-add the user and password again.

Done. Tried removing .ie. blank.. tried wrong details to force error, nothing.. its like the plugin itself is not being started

If it still not working, I would try setting the username and password in the logstash-keystore and try again, if nothing works then I would open a bug report.

I have a bunch of Developers here who are going to pull Logstash apart and trace which components of the logstash elastic plugin are not working and why, as they have been working on similar issues under CIS hardening with Jira. if they can find, fix all good, if they can find the smoking gun, then i can open a bug report and let the logstash guys at it.

Badger · November 3, 2021, 4:20pm

No, the error message changed. You are missing a } to close the conditional in the filter section.

BeMoore · November 5, 2021, 9:39am

i had missed that, but on my running config i had # out the whole filter and still wasnt working.

what it turned out to be was the CIS hardening set a umask for the logstash service account of 0027.
Apparently when hardening is performed, all users/files/directories according to the spec of the CIS process are set to the most restrictive perms which allows for some activity, but just not all

setting it to 0022 allows the logstash service to start as normal AND process, access and utilize the elastic output plugin.

WHY?
Not sure, other plugins have not been tested except the file output plugin, i might redirect the output to another beats agent and see if this works under the 0027 umask. But it does leave the question as to why the file output plugin works under 0027 and not the elastic plugin unless umask is set to 0022, unless its some buried access requirement to some ruby/java sockets process ??

we're doing some more testing now to see what and if there's other perms lurking below the surface, but it seems the pipeline is active now.

system · December 3, 2021, 9:39am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Log stash Configuration Error Logstash	6	20117	February 22, 2018
ERR Failed to Perform Many Index Operations Beats filebeat	7	2647	July 5, 2017
Logstash errors out-Couldn't find any input plugin named 'beats' Logstash	2	1103	February 6, 2017
Filebeat with Logstash failed to create Index Beats filebeat	9	1305	May 28, 2021
Trouble sending logs to Logstash from Filebeat - Pipeline error recieved Beats filebeat	3	348	April 28, 2020

Logstash output creating error

Related topics