I am starting with ELK and following the official doc on how to use filebeat with modules:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-quickstart.html
I am using separate containers for Elasticsearch, Kibana and Logstash
I am using ELK as separate containers:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3a3b9a484831 logstash "/docker-entrypoin..." 25 minutes ago Up 25 minutes 0.0.0.0:5044->5044/tcp logstash
7a4829ac080d kibana "/docker-entrypoin..." 50 minutes ago Up 38 minutes 0.0.0.0:5601->5601/tcp kibana
6148a8af18e6 elasticsearch "/docker-entrypoin..." About an hour ago Up About an hour 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch
In the documentation it is required to restart elasticsearch after installing the two beats modules
- ** ingest-geo-ip**
- ** ingestuser-agent**
Both modules are installed properly on elasticsearch container:
bin/elasticsearch-plugin install ingest-geoip
-> Downloading ingest-geoip from elastic
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
Continue with installation? [y/N]y
-> Installed ingest-geoip
# bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100%
-> Installed ingest-user-agent
According to the doc a restart of elasticsearch is required.
After stopping elasticsearch container, I couldn't start it again.
The logs shows that there is an exception with the installed ingest-geo-ip module
[2018-04-09T12:29:52,327][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/ingest-geoip
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.8.jar:5.6.8]
Caused by: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/ingest-geoip
at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:74) ~[?:?]
at org.elasticsearch.ingest.IngestService.(IngestService.java:58) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.node.Node.(Node.java:354) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.8.jar:5.6.8]
... 6 more
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/ingest-geoip
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) ~[?:?]
at java.nio.file.Files.newDirectoryStream(Files.java:457) ~[?:1.8.0_162]
at java.nio.file.Files.list(Files.java:3451) ~[?:1.8.0_162]
at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:85) ~[?:?]
at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:72) ~[?:?]
at org.elasticsearch.ingest.IngestService.(IngestService.java:58) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.node.Node.(Node.java:354) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.node.Node.(Node.java:245) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:233) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.8.jar:5.6.8]
... 6 more
Any hint? Am I doing something wrong?
Thanks for your assistance.