Hello, I would like to ask what is wrong with my logstash configuration. I configured the port to listen to 5044 in the pipeline.yml file, but only 9600 is monitored after logstash is started; the port 5044 is not monitored, and the log does not report any errors. .
pipeline.yml config
logstash.yml config
other.conf config
Please share logstash logs while you start the service.
Also, do not share screenshots, share the plain text using the </> button. Screenshots can be hard to see, some people may not be able to see it, and it is impossible to copy and suggest changes.
ok, thanks
Replenish:
pipeline.yml
- pipeline.id: beats
config.string: |
input { beats { port => 5044 } }
output {
if [source] == 'php-service' {
pipeline { send_to => phplogs }
} else if [source] == 'java-service' {
pipeline { send_to => javalogs }
} else if [source] == 'system' {
pipeline { send_to => syslogs }
}
logstash.yml:
node.name: logstash_node
path.data: /data/logstash/data
#pipeline.id: main
pipeline.workers: 4
pipeline.batch.size: 254
pipeline.batch.delay: 90
pipeline.unsafe_shutdown: false
pipeline.ordered: auto
path.config: /usr/local/logstash/config/conf.d
#config.string
config.test_and_exit: false
config.reload.automatic: true
config.reload.interval: 3s
config.debug: false
http.enabled: true
http.host: 0.0.0.0
http.port: 9600-9700
queue.type: memory
path.queue: /data/logstash/queue
queue.page_capacity: 64mb
queue.max_events: 0
queue.max_bytes: 1024mb
queue.checkpoint.acks: 1024
queue.checkpoint.writes: 1024
queue.checkpoint.interval: 1000
dead_letter_queue.enable: false
log.level: info
path.logs: /data/logstash/logs
test.conf:
input {
pipeline {
address => phplogs
}
}
filter {
mutate {
remove_field => "input"
remove_field => "host"
remove_field => "log"
remove_field => "ecs"
remove_field => "agent"
remove_field => "@version"
remove_field => "_index"
remove_field => "local"
remove_field => '_id'
remove_field => 'fields_ip'
remove_field => "fields_service"
remove_field => "tags"
remove_field => '_score'
}
}
logstash startup log：
[2022-08-22T09:55:41,000][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-08-22T09:55:41,257][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2022-08-22T09:55:41,376][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-08-22T09:55:41,385][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:".monitoring-logstash"}
[2022-08-22T09:55:41,430][INFO ][logstash.runner ] Logstash shut down.
[2022-08-22T09:56:01,013][INFO ][logstash.runner ] Log4j configuration path used is: /usr/local/logstash/config/log4j2.properties
[2022-08-22T09:56:01,026][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-08-22T09:56:01,028][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-08-22T09:56:01,434][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-08-22T09:56:03,890][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2022-08-22T09:56:03,892][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2022-08-22T09:56:04,428][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-08-22T09:56:05,533][INFO ][org.reflections.Reflections] Reflections took 123 ms to scan 1 urls, producing 124 keys and 408 values
[2022-08-22T09:56:05,758][INFO ][logstash.javapipeline ] Pipeline `.monitoring-logstash` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-08-22T09:56:05,856][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://172.16.0.91:9200", "http://172.16.0.92:9200"]}
[2022-08-22T09:56:05,877][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-08-22T09:56:05,897][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@172.16.0.91:9200/, http://logstash_system:xxxxxx@172.16.0.92:9200/]}}
[2022-08-22T09:56:05,904][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.16.0.91:9200", "//172.16.0.92:9200"]}
[2022-08-22T09:56:05,915][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@172.16.0.91:9200/, http://elastic:xxxxxx@172.16.0.92:9200/]}}
[2022-08-22T09:56:05,934][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://logstash_system:xxxxxx@172.16.0.91:9200/"}
[2022-08-22T09:56:05,940][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (8.3.2) {:es_version=>8}
[2022-08-22T09:56:05,941][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-08-22T09:56:05,942][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@172.16.0.91:9200/"}
[2022-08-22T09:56:05,952][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.3.2) {:es_version=>8}
[2022-08-22T09:56:05,953][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-08-22T09:56:05,986][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://logstash_system:xxxxxx@172.16.0.92:9200/"}
[2022-08-22T09:56:05,997][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@172.16.0.92:9200/"}
[2022-08-22T09:56:06,054][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-08-22T09:56:06,070][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-08-22T09:56:06,077][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-08-22T09:56:06,087][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2022-08-22T09:56:06,133][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-08-22T09:56:06,220][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>90, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x16308030@/usr/local/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52 run>"}
[2022-08-22T09:56:06,231][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>254, "pipeline.batch.delay"=>90, "pipeline.max_inflight"=>1016, "pipeline.sources"=>["/usr/local/logstash/config/conf.d/php.conf"], :thread=>"#<Thread:0x2508a4a2 run>"}
[2022-08-22T09:56:07,081][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.86}
[2022-08-22T09:56:07,184][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-08-22T09:56:07,291][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.06}
[2022-08-22T09:56:07,308][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-08-22T09:56:07,387][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
Your pipelines.yml is not being used as you can see in this line:
[2022-08-22T09:56:01,434][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
How are you starting Logstash?
Thank you for your reply, now I have solved the problem, the reason is that there is no comment in path.config in logstash.yml file.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.