ETA on Support for ubuntu 18.04 Server?

Elastic Stack Logstash
1

Hi...

I've had a go at getting ELK up and running on ubuntu 18.04 Server (with Java 8).

I installed elasticsearch and logstash a day apart and got elasticsearch 6.4.1 and logstash 6.4.2 as a result. Both installed using apt-get.

I managed to get Elasticsearch working and logstash installed seemingly without incident. But, firing up logstash is a disappointment. It hangs. I'll happily paste in my logstash.conf file below.

The logstash server claims to start but then hangs, never progressing on successfully loading/filtering/and subsequently routing the data. Running logstash with the --debug flag creates a lot of spew, too much to post here. But I see this message a lot:

[DEBUG] 2018-10-03 08:57:29.975 [pool-2-thread-2] cgroup - Error, cannot retrieve cgroups information {:exception=>"NoMethodError", :message=>"undefined method `[]' for nil:NilClass"}

Below is my logstash.conf file. It's a pretty simple thing that just takes an access log file and then does some simple filtering/output:

stretch@stretch:/usr/share/logstash$ cat /etc/logstash/conf.d/logstash.conf 
input {
      file {
      	   path => "/home/stretch/access_log"
	   start_position => "beginning"
	   ignore_older => 0
      }
}
filter {
       grok {
       	    match => {"message" => "%{COMBINEDAPACHELOG}"}
       }
       date {
	    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
       }
}
output {
       elasticsearch {
       		     hosts => ["localhost:9200"]
       }
       stdout {
              codec => rubydebug
       }
}

Any thoughts on what's wrong?

Thanks!

2

Unless you are adding new data to the input file, you are likely running into sincedb problems.
Try setting to to /dev/null

3

Thanks Mark for your speedy reply. The file is just a static example file from an online course.

I modified my logstash.conf file to include a sincedb_path => /dev/null to the input file block. Logged out and logged back in. But things did not improve.

Here's the spew with the debugging turned off. This may be more instructive?

sudo bin/logstash -f /etc/logstash/conf.d/logstash.conf

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[WARN ] 2018-10-03 13:00:33.924 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified

[INFO ] 2018-10-03 13:00:34.955 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.4.1"}

[INFO ] 2018-10-03 13:00:41.328 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[INFO ] 2018-10-03 13:00:42.148 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}

[INFO ] 2018-10-03 13:00:42.159 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}

[WARN ] 2018-10-03 13:00:42.500 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}

[INFO ] 2018-10-03 13:00:42.829 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}

[WARN ] 2018-10-03 13:00:42.832 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}

[INFO ] 2018-10-03 13:00:42.886 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}

[INFO ] 2018-10-03 13:00:42.927 [Ruby-0-Thread-5: :1] elasticsearch - Using mapping template from {:path=>nil}

[INFO ] 2018-10-03 13:00:42.987 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}

[INFO ] 2018-10-03 13:00:43.716 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x27bcd905 run>"}

[INFO ] 2018-10-03 13:00:43.845 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

[INFO ] 2018-10-03 13:00:43.893 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections

[INFO ] 2018-10-03 13:00:44.525 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

For some reason, logstash was not finding my YAML files, in spite of the fact they were sitting where logstash claims is "the usual place." Invoking logstash this way

$ sudo bin/logstash -f /etc/logstash/conf.d/logstash.conf --path.settings=/etc/logstash

got rid of those messages but things still hang after logstash claims it has successfully started.

Are there quirks to ubuntu 18.04 server that are causing these problems?

Thanks!

4

Skip this and see if that makes a difference.

5

Thanks Christian,

I tried this. But, as I understand things, ignore_older => 0 tells logstash to look at all of the log data, even if from quite some time in the past (as it is with the log in question).

Here's the /var/log/logstash/logstash-plain.log file:

**stretch@stretch** : **/usr/share/logstash** $ more /var/log/logstash/logstash-plain.log 

[2018-10-03T13:12:02,247][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queu

e"}

[2018-10-03T13:12:02,263][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/l

ogstash/dead_letter_queue"}

[2018-10-03T13:12:02,948][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options a

re specified

[2018-10-03T13:12:03,016][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"f3febfd1-58fa-411a-860e

-fabb85233625", :path=>"/var/lib/logstash/uuid"}

[2018-10-03T13:12:04,106][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.1"}

[2018-10-03T13:12:10,171][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.siz

e"=>125, "pipeline.batch.delay"=>50}

[2018-10-03T13:12:10,908][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://loca

lhost:9200/]}}

[2018-10-03T13:12:10,925][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:health

check_url=>http://localhost:9200/, :path=>"/"}

[2018-10-03T13:12:11,255][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}

[2018-10-03T13:12:11,355][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}

[2018-10-03T13:12:11,371][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to dete

rmine the document _type {:es_version=>6}

[2018-10-03T13:12:11,416][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>

["//localhost:9200"]}

[2018-10-03T13:12:11,452][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}

[2018-10-03T13:12:11,498][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "

version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"

=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_typ

e"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"

@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"t

ype"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}

[2018-10-03T13:12:12,183][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x77c87f1e 

run>"}

[2018-10-03T13:12:12,316][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>

[]}

[2018-10-03T13:12:12,342][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections

[2018-10-03T13:12:12,943][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[2018-10-03T13:14:33,260][WARN ][logstash.runner ] SIGINT received. Shutting down.

[2018-10-03T13:14:33,479][INFO ][filewatch.observingtail ] QUIT - closing all files and shutting down.

[2018-10-03T13:14:33,794][FATAL][logstash.runner ] SIGINT received. Terminating immediately..

[2018-10-03T13:14:33,929][ERROR][org.logstash.Logstash ] org.jruby.exceptions.ThreadKill

[2018-10-03T13:15:20,304][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options a

re specified

[2018-10-03T13:15:26,277][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

[2018-10-03T13:47:27,977][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options a

re specified

[2018-10-03T13:47:33,707][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.