I am currently trying to learn about Logstash. So I just download the zip file and put in some basic configuration and ran logstash.bat file.
But for some reason I get Proxy Authentication exception. This is console output:
"Using bundled JDK: C:\logstash-8.1.2\jdk\bin\java.exe"
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/logstash-8.1.2/logs which is now configured via log4j2.properties
[2022-05-10T16:23:37,085][INFO ][logstash.runner ] Log4j configuration path used is: C:\logstash-8.1.2\config\log4j2.properties
[2022-05-10T16:23:37,094][WARN ][logstash.runner ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-05-10T16:23:37,096][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.1.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [mswin32-x86_64]"}
[2022-05-10T16:23:37,097][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-05-10T16:23:38,732][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-05-10T16:23:39,400][INFO ][org.reflections.Reflections] Reflections took 57 ms to scan 1 urls, producing 120 keys and 419 values
[2022-05-10T16:23:40,669][INFO ][logstash.javapipeline ] Pipeline `test_pipeline` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-05-10T16:23:40,704][INFO ][logstash.outputs.elasticsearch][test_pipeline] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://135.75.116.157:9200"]}
[2022-05-10T16:23:40,953][INFO ][logstash.outputs.elasticsearch][test_pipeline] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://admin:xxxxxx@135.75.116.157:9200/]}}
[2022-05-10T16:23:47,163][WARN ][logstash.outputs.elasticsearch][test_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://admin:xxxxxx@135.75.116.157:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '407' contacting Elasticsearch at URL 'http://135.75.116.157:9200/'"}
[2022-05-10T16:23:47,190][INFO ][logstash.outputs.elasticsearch][test_pipeline] Config is compliant with data streams. `data_stream => auto` resolved to `true`
[2022-05-10T16:23:47,192][WARN ][logstash.outputs.elasticsearch][test_pipeline] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-05-10T16:23:47,198][WARN ][logstash.filters.grok ][test_pipeline] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-05-10T16:23:47,250][WARN ][logstash.javapipeline ][test_pipeline] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2022-05-10T16:23:47,295][INFO ][logstash.javapipeline ][test_pipeline] Starting pipeline {:pipeline_id=>"test_pipeline", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1, "pipeline.sources"=>["C:/logstash-8.1.2/config/test.config"], :thread=>"#<Thread:0x5e064f51 run>"}
[2022-05-10T16:23:47,901][INFO ][logstash.javapipeline ][test_pipeline] Pipeline Java execution initialization time {"seconds"=>0.6}
[2022-05-10T16:23:47,937][INFO ][logstash.inputs.file ][test_pipeline] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/logstash-8.1.2/data/plugins/inputs/file/.sincedb_ea834d43a2641af0ead9826d153946be", :path=>["C:/elk/spring-boot-elk.log"]}
[2022-05-10T16:23:47,950][INFO ][logstash.javapipeline ][test_pipeline] Pipeline started {"pipeline.id"=>"test_pipeline"}
[2022-05-10T16:23:48,013][INFO ][filewatch.observingtail ][test_pipeline][093a6477bcb6f9e157b916e0c5fc22c1ecfd07395e5eb5cc02cf7fddb3aeba6a] START, creating Discoverer, Watch with file and sincedb collections
[2022-05-10T16:23:48,024][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:test_pipeline], :non_running_pipelines=>[]}
[2022-05-10T16:23:53,461][WARN ][logstash.outputs.elasticsearch][test_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://admin:xxxxxx@135.75.116.157:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '407' contacting Elasticsearch at URL 'http://135.75.116.157:9200/'"}
[2022-05-10T16:23:58,736][WARN ][logstash.outputs.elasticsearch][test_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://admin:xxxxxx@135.75.116.157:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '407' contacting Elasticsearch at URL 'http://135.75.116.157:9200/'"}
[2022-05-10T16:24:04,026][WARN ][logstash.outputs.elasticsearch][test_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://admin:xxxxxx@135.75.116.157:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '407' contacting Elasticsearch at URL 'http://135.75.116.157:9200/'"}
[2022-05-10T16:24:09,307][WARN ][logstash.outputs.elasticsearch][test_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://admin:xxxxxx@135.75.116.157:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '407' contacting Elasticsearch at URL 'http://135.75.116.157:9200/'"}
[2022-05-10T16:24:19,253][WARN ][logstash.outputs.elasticsearch][test_pipeline] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://admin:xxxxxx@135.75.116.157:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '407' contacting Elasticsearch at URL 'http://135.75.116.157:9200/'"}
My conf file is looks like this:
input {
file {
type => "java"
path => "C:/elk/spring-boot-elk.log"
codec => multiline {
pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
negate => "true"
what => "previous"
}
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
}
output {
elasticsearch {
hosts => ["http://135.75.116.157:9200"]
#user => "admin"
#password => "admin"
proxy => "http://anonymous:password@MY_PROXY_URL:8080"
}
stdout { codec => rubydebug }
}
But it doesn't work. The same proxy works with other tools. So the problem might not be that.
Any pointers might be helpful. Thanks!