I've followed these steps: https://github.com/virtualtechmeetup/0003_howto_elasticsearch_kibana_install_ubuntu
I would like to send Syslog traffic from Logstash to Datadog and Elasticsearch. Really just using Elasticsearch & Kibana for testing.
Ubuntu, 20.04. Elastic, 7.14.0. Elastic Plugin, syslog-input and datadog-output.
When starting Logstash, I see the following error.
[2021-08-12T13:30:27,038][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[2021-08-12T13:30:27,858][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::Grok", {"match"=>{"message"=>"%{COMBINEDAPACHELOG}"}, "id"=>"16abf4b71e7bbc19bc65b0a4b6aac50f05b656511e3e49a2604551c09acb9cc0"}]=>[{"thread_id"=>35, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}, {"thread_id"=>36, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}]}}
Since this is a new install, I only have one config under /etc/logstash/conf.d/30-syslog.conf
.
input {
tcp {
port => 10514
type => syslog
}
udp {
port => 10514
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
datadog_logs {
api_key => "API-KEY"
}
}
elasticsearch.conf
cluster.name: lab
node.name: lab1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.type: 'single-node'
indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000
action.destructive_requires_name: 'true'
reindex.remote.whitelist: '*:*'
xpack.monitoring.enabled: 'true'
xpack.monitoring.collection.enabled: 'true'
xpack.monitoring.collection.interval: 30s
xpack.security.enabled: 'true'
xpack.security.audit.enabled: 'false'
node.ml: 'false'
xpack.ml.enabled: 'false'
xpack.watcher.enabled: 'false'
xpack.ilm.enabled: 'true'
xpack.sql.enabled: 'true'
I when I curl http://localhost:9200/
, I have to enter username/password. Where do I set the username/password for Logstash?