Help configuring logstash on Microsoft Azure

[LukeJB (Luke)]

Hi, I'm trying to get logstash configured correctly on Azure. I'm a bit less that a week old with this. Recently, when I try running logstash, I get the following warnings and errors:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-25 23:04:28.786 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[WARN ] 2020-06-25 19:57:55.334 [[main]-pipeline-manager] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[ERROR] 2020-06-25 19:55:34.151 [[main]<http] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
[WARN ] 2020-06-25 19:55:35.189 [[main]<http] AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xcc00ce86]
[ERROR] 2020-06-25 19:55:35.206 [[main]<http] rejectedExecution - Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated

The last error would then repeat until I kill the process. I've been trying to fix these errors starting from the top and have tried specifying the config path with --path.settings but that only results in another error.

ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/etc/logstash/, exception=Java::OrgLogstashSecretStore::SecretStoreException::LoadException, message=>Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore.

I try exporting a LOGSTASH_KEYSTORE_PASS on the off chance it simply is having trouble accessing the keystore file. No luck.

I'm low on options and have been at this for countless hours.

My logstash.conf file looks like this:

input {
    http {
        host => "12.34.567.231" # default: 0.0.0.0
        port =>  8080
        codec => json
    }
}

filter {
    split {
        field => "events"
    }
    ruby {
        code => "
        event.get('events').each do |k, v|
        event.set(k, v)
        end
        "
    }
    mutate {
        remove_field => [ "events" ]
    }
}


output {
    elasticsearch { hosts => ["12.21.435.245:9200"]
        index => "aol"
        template_overwrite => true
        user => default
        password => # excluded
    }
    stdout { codec => rubydebug }
}

My logstash.yml file looks like this:

node.name: "logstash-0"
path.settings: /etc/logstash
path.data: /var/lib/logstash
# path.config: /etc/logstash/conf.d/*.conf
path.logs: /var/log/logstash
log.level: error
xpack.monitoring.elasticsearch.hosts: ["${ELASTICSEARCH_URL}"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: "${LOGSTASH_SYSTEM_PASSWORD}"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://65.52.113.245:9200

Thanks.