I am trying to run Logstash in my kubernetes-cluster but I get this error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2020-05-14T11:03:46,981][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2020-05-14T11:03:46,997][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2020-05-14T11:03:47,574][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.1"}
[2020-05-14T11:03:47,592][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"fb6f9398-6832-4f6e-b5ad-db8b39c9e929", :path=>"/usr/share/logstash/data/uuid"}
[2020-05-14T11:03:48,014][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2020-05-14T11:03:48,726][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2020-05-14T11:03:48,944][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
[2020-05-14T11:03:49,002][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-05-14T11:03:49,009][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
[2020-05-14T11:03:49,043][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2020-05-14T11:03:50,207][INFO ][org.reflections.Reflections] Reflections took 38 ms to scan 1 urls, producing 20 keys and 40 values
[2020-05-14T11:03:50,605][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-05-14T11:03:50,610][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3f8ef33e run>"}
[2020-05-14T11:03:51,370][INFO ][logstash.inputs.beats ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-05-14T11:03:51,381][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-14T11:03:51,529][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
[2020-05-14T11:03:51,612][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-05-14T11:03:51,789][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-05-14T11:04:19,028][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2020-05-14T11:04:19,201][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
[2020-05-14T11:04:49,022][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
I guess, Logstash is not using my pipeline output config because is trying to connect to another host. I set xpack.monitoring.enabled to false in the config map included in my yaml file but nothing happens.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
namespace: elk
labels:
task: logging
k8s-app: logstash
data:
logstash.yml: |
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
xpack.monitoring.enabled: false
pipeline.conf: "
input {
beats {
port => 5000
type => \"kube-logs\"
ssl => false
}
}
filter {
grok{
match=>{
\"source\"=>\"%{GREEDYDATA}/%{GREEDYDATA:app}-%{DATA}-%{DATA}_%{DATA:namespace}_%{GREEDYDATA}\"
}
add_tag=>[\"k8s-app-extracted\"]
}
if [app] == \"nginx-ingress-controller\" {\n
grok {
match=>{\n
\"log\"=>\"%{IP:real_ip} %{DATA:http_host} %{DATA:proxy_protocol_ip} %{DATA:remote_addr} - \\[%{DATA:forwarded_for}\\] - %{DATA:remote_user} \\[%{DATA:ingress_time}\\] \\\"%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:http_version}\\\" %{NUMBER:result} %{NUMBER:bytes} \\\"%{DATA:referer}\\\" \\\"%{DATA:agent}\\\" %{DATA:request_length} %{DATA:request_time} \\[%{DATA:upstream}\\] %{DATA:upstream_addr} %{NUMBER:upstream_length} %{NUMBER:upstream_time} %{NUMBER:upstream_result}%{SPACE}%{WORD:request_id}\"
}\n
add_tag=>[\"ingress-access-log\"]\n
}
if \"ingress-access-log\" in [tags] {
mutate {
replace => { \"type\" => \"ingress-access\" }
}
}
}\n
}\n
output {\n
elasticsearch {\n
hosts => [\"192.168.20.50:9200\"]\n
sniffing => false\n
manage_template => false\n
index => \"%{[type]}-%{+YYYY.MM.dd}\"\n
document_type => \"%{[@metadata][type]}\"\n
}\n
}\n"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: logging-logstash
namespace: elk
spec:
replicas: 1
template:
metadata:
labels:
task: logging
k8s-app: logstash
spec:
containers:
- name: logstash
image: dtrvip.sra.local/elk/logstash:7.6.1
imagePullPolicy: Always
volumeMounts:
- mountPath: /config
name: config
- mountPath: /pipeline
name: pipeline
volumes:
- name: config
configMap:
name: logstash-config
items:
- key: logstash.yml
path: logstash.yml
- name: pipeline
configMap:
name: logstash-config
items:
- key: pipeline.conf
path: pipeline.conf
securityContext:
fsGroup: 101
---
apiVersion: v1
kind: Service
metadata:
labels:
task: logging
kubernetes.io/name: logstash
name: logging-logstash
namespace: elk
spec:
ports:
- port: 5000
targetPort: 5000
name: lumberjack
selector:
k8s-app: logstash
Please help!