The docker.elastic.co/logstash/logstash:5.6.1 image is not sending logs to an ElasticSearch cluster that is running on different servers.
We are running this docket image in docker toolbox on a windows 7.0 PC. The ElasticSearch cluster is running RedHat Linux servers. The Docker Image needs to connect via tcp/ip to the ElasticSearch servers.
The Docker Image is able to receive logs filebeat and write the logs to standard out. But it is unable to send the logs to ElasticCluster.
Using the same configuration files as the docker image, a logstash 5.6.0 distribution on my PC, running outside of docker can send logs to the ElasticSearch Cluster. This implies that:
1) there is an issue with the docker.elastic.co/logstash/logstash:5.6.1 image
2) the docker image needs a change to the configuration files.
I believe the bullet 2 is the most likely case.
DOCKER RUN
docker run -p 5045:5045 -p 9600:9600 --rm
--mount type=bind,source=/c/Users/Public/logstash/config_pipeline,destination=/usr/share/logstash/pipeline
--mount type=bind,source=/c/Users/Public/logstash/config,destination=/usr/share/logstash/config
--mount type=bind,source=/c/Users/Public/logstash/config_kafka,destination=/usr/share/logstash/config_kafka
--mount type=bind,source=/c/Users/Public/logstash/logs,destination=/usr/share/logstash/logs
docker.elastic.co/logstash/logstash:5.6.1 >/c/Users/Public/logstash/logs/stdout_err.txt 2>&1 &
windows command to run logstash
ping monatee-loggy-master-tpc.dev.bnymellon.net
set JAVA_HOME=C:\Program Files\Java\jre1.8.0_152
C:
cd C:\Users\Public\logstash\logstash-5.6.0
bin\logstash --http.host L101CTS06E315 --http.port 9905 --path.data C:/Users/Public/logstash/data -f C:\Users\Public\logstash\config_pipeline --path.logs C:/Users/Public/logstash/logs --path.settings C:/Users/Public/logstash/config >C:/Users/Public/logstash/logs/logstash_stdout.txt 2>&1 &
logstash.yml configurations
path.data: /usr/share/logstash/data
path.config: /usr/share/logstash/pipeline
http.host: "0.0.0.0"
log.level: info
path.logs: /usr/share/logstash/logs
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.elasticsearch.url: http://monatee-loggy-master-tpc.dev.bnymellon.net:80
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.enabled: false
pipeline configuration
The # character at the beginning of a line indicates a comment. Use
comments to describe your configuration.
input {
beats {
port => "5045"
#port => "5055"
type => syslog
}
# tcp {
# port => "5045"
# type => syslog
# }
#tcp {
# port => "514"
# type => syslog
#}
}
The filter part of this file is commented out to indicate that it is
optional.
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
ruby {
code => "event.set('logstash_1_received_time', Time.now.utc.strftime('%FT%T.%L') )"
}
mutate {
add_field => [ "logstash_1_server", "albert_pc" ]
}
}
output {
# stdout { codec => rubydebug }
elasticsearch {
hosts => ["monatee-loggy-master-tpc.dev.bnymellon.net:80"]
# hosts => [ "r00j8dn0c.bnymellon.net", "r00j8en0c.bnymellon.net", "r00j8jn0c.bnymellon.net" ]
# index => "monatee_loggy_tpc_1-%{+YYYY.MM.dd}"
}
#kafka {
# bootstrap_servers => "rsomtapae182.bnymellon.net:9092,rsomtapae183.bnymellon.net:9092,rsomtapae184.bnymellon.net:9092"
# topic_id => "monatee_loggy_tpc"
# jaas_path => "/opt/pki/logstash_config_kafka/kafka_client_jaas_logstash.conf"
# security_protocol => "SASL_PLAINTEXT"
# sasl_kerberos_service_name => "kafka"
# sasl_mechanism => "PLAIN"
# message_key => plain {
# format => "%{beats_message_key}"
# id => "loggy_kafka_output_message_key"
# }
# codec => plain {
# format => "%{logstash_1_received_time} %{logstash_1_server} %{message} %{ip} %{netmask} %{subnet} %{partition} %{beats_message_key}"
# id => "loggy_kafka_output_codec"
# }
#}
}