so , i wrote an elasticsearch.yaml , which contains the configuration of elasticsearch, kibana and logstash. i will just attach the text
version: '3.3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
container_name: elasticsearch
restart: always
environment:
- xpack.security.enabled=false
- discovery.type=single-node
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
ports:
- 9200:9200
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.4.0
restart: always
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
ports:
- 5601:5601
depends_on:
- elasticsearch
Logstash:
image: logstash:8.7.0
container_name: logstash
restart: always
# volume:
# - ./logstash/:/logstash_dir
command: logstash -f C:\Users\svats\Downloads\logstash-8.7.0\logstash.conf
depends_on:
- elasticsearch
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
volumes:
elasticsearch-data:
// then the logs in logstash are as follows:
Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
2023-05-17 03:44:05 [2023-05-16T22:14:05,360][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.7.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-linux]"}
2023-05-17 03:44:05 [2023-05-16T22:14:05,362][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx256m, -Xms256m, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
2023-05-17 03:44:05 [2023-05-16T22:14:05,524][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2023-05-17 03:44:05 [2023-05-16T22:14:05,916][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
2023-05-17 03:44:05 Please configure Metricbeat to monitor Logstash. Documentation can be found at:
2023-05-17 03:44:05 https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
2023-05-17 03:44:06 [2023-05-16T22:14:06,198][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2023-05-17 03:44:06 [2023-05-16T22:14:06,265][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
2023-05-17 03:44:06 [2023-05-16T22:14:06,271][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (7.4.0) {:es_version=>7}
2023-05-17 03:44:06 [2023-05-16T22:14:06,271][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2023-05-17 03:44:06 [2023-05-16T22:14:06,311][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
2023-05-17 03:44:06 [2023-05-16T22:14:06,311][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
2023-05-17 03:44:06 [2023-05-16T22:14:06,407][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2023-05-17 03:44:06 [2023-05-16T22:14:06,694][INFO ][org.reflections.Reflections] Reflections took 86 ms to scan 1 urls, producing 132 keys and 462 values
2023-05-17 03:44:06 [2023-05-16T22:14:06,797][INFO ][logstash.javapipeline ] Pipeline `.monitoring-logstash` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
2023-05-17 03:44:06 [2023-05-16T22:14:06,815][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
2023-05-17 03:44:06 [2023-05-16T22:14:06,819][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2023-05-17 03:44:06 [2023-05-16T22:14:06,837][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
2023-05-17 03:44:06 [2023-05-16T22:14:06,842][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.4.0) {:es_version=>7}
2023-05-17 03:44:06 [2023-05-16T22:14:06,842][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2023-05-17 03:44:06 [2023-05-16T22:14:06,870][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
2023-05-17 03:44:06 [2023-05-16T22:14:06,871][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
2023-05-17 03:44:06 [2023-05-16T22:14:06,880][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x1a735e6a@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
2023-05-17 03:44:07 [2023-05-16T22:14:07,268][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.39}
2023-05-17 03:44:07 [2023-05-16T22:14:07,275][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
2023-05-17 03:44:07 [2023-05-16T22:14:07,290][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
2023-05-17 03:44:08 [2023-05-16T22:14:08,875][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
2023-05-17 03:44:09 [2023-05-16T22:14:09,329][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:".monitoring-logstash"}
2023-05-17 03:44:09 [2023-05-16T22:14:09,336][INFO ][logstash.runner ] Logstash shut down.
2023-05-17 03:44:10 Using bundled JDK: /usr/share/logstash/jdk
2023-05-17 03:53:00 Using bundled JDK: /usr/share/logstash/jdk
2023-05-17 03:53:17 Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2023-05-17 03:53:17 [2023-05-16T22:23:17,718][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
2023-05-17 03:53:17 [2023-05-16T22:23:17,722][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.7.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-linux]"}
2023-05-17 03:53:17 [2023-05-16T22:23:17,724][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx256m, -Xms256m, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
2023-05-17 03:53:17 [2023-05-16T22:23:17,918][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2023-05-17 03:53:18 [2023-05-16T22:23:18,363][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
2023-05-17 03:53:18 Please configure Metricbeat to monitor Logstash. Documentation can be found at:
2023-05-17 03:53:18 https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
2023-05-17 03:53:18 [2023-05-16T22:23:18,706][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2023-05-17 03:53:18 [2023-05-16T22:23:18,863][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
2023-05-17 03:53:18 [2023-05-16T22:23:18,870][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (7.4.0) {:es_version=>7}
2023-05-17 03:53:18 [2023-05-16T22:23:18,870][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2023-05-17 03:53:18 [2023-05-16T22:23:18,916][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
2023-05-17 03:53:18 [2023-05-16T22:23:18,916][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
2023-05-17 03:53:18 [2023-05-16T22:23:18,985][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2023-05-17 03:53:19 [2023-05-16T22:23:19,433][INFO ][org.reflections.Reflections] Reflections took 120 ms to scan 1 urls, producing 132 keys and 462 values
2023-05-17 03:53:19 [2023-05-16T22:23:19,554][INFO ][logstash.javapipeline ] Pipeline `.monitoring-logstash` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
2023-05-17 03:53:19 [2023-05-16T22:23:19,576][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
2023-05-17 03:53:19 [2023-05-16T22:23:19,583][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2023-05-17 03:53:19 [2023-05-16T22:23:19,600][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
2023-05-17 03:53:19 [2023-05-16T22:23:19,606][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.4.0) {:es_version=>7}
2023-05-17 03:53:19 [2023-05-16T22:23:19,606][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2023-05-17 03:53:19 [2023-05-16T22:23:19,652][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
2023-05-17 03:53:19 [2023-05-16T22:23:19,652][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
2023-05-17 03:53:19 [2023-05-16T22:23:19,661][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x512565a5@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
2023-05-17 03:53:20 [2023-05-16T22:23:20,168][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.51}
2023-05-17 03:53:20 [2023-05-16T22:23:20,179][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
2023-05-17 03:53:20 [2023-05-16T22:23:20,188][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
2023-05-17 03:53:21 [2023-05-16T22:23:21,654][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
2023-05-17 03:53:22 [2023-05-16T22:23:22,208][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:".monitoring-logstash"}
2023-05-17 03:53:22 [2023-05-16T22:23:22,212][INFO ][logstash.runner ] Logstash shut down.
// in docker desktop , i just added config file in logstash.config in pipeline , and changed nothing else. it doesn't show any massive error , but doesn't start also.