DWbank
May 6, 2020, 4:35pm
1
I for the longest time only had one pipeline but now I need to add the second.
The first pipeline starts and works as it always has, the second does not open the ports in a netstat -an also looked at iptables and that is not blocking anything.
Alienvault config
input {
tcp {
port => 5142
type => "ossim-events"
codec => json {
charset => "CP1252"
}
}
}
filter {
##################### This formats the message to so processing works
Sas config
input {
tcp {
port => 5514
type => "sas-logs"
codec => json {
charset => "CP1252"}
}
}
filter {
Here is the only lines in my logstash.yml file not commented out
# * trace
#
log.level: info
path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
# ------------ X-Pack Settings (not applicable for OSS build)--------------
#
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
#xpack.monitoring.enabled: false
xpack.monitoring.elasticsearch.username: user
xpack.monitoring.elasticsearch.password: pass
#xpack.monitoring.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
#xpack.management.pipeline.id: ["main", "apache_logs"]
xpack.management.elasticsearch.username: user
xpack.management.elasticsearch.password: pass
#xpack.management.elasticsearch.hosts: ["https://es1:9200", "https://es2:9200"]
#xpack.management.elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
DWbank
May 6, 2020, 7:54pm
2
Here is the debug
Logs from command line startup
server-1:/etc/logstash# /usr/share/logstash/bin/logstash --log.level debug --config.debug -f /etc/logstash/pipelines.yml
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[DEBUG] 2020-05-06 15:42:52.559 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[DEBUG] 2020-05-06 15:42:52.563 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x743e5472 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2020-05-06 15:42:52.565 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[DEBUG] 2020-05-06 15:42:52.565 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x49da976 @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2020-05-06 15:42:52.827 [LogStash::Runner] runner - -------- Logstash Settings (* means modified) ---------
[DEBUG] 2020-05-06 15:42:52.827 [LogStash::Runner] runner - node.name: "VPWI-ALIENLOG-1"
[DEBUG] 2020-05-06 15:42:52.827 [LogStash::Runner] runner - *path.config: "/etc/logstash/pipelines.yml"
[DEBUG] 2020-05-06 15:42:52.827 [LogStash::Runner] runner - path.data: "/usr/share/logstash/data"
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - modules.cli: []
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - modules: []
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - modules_list: []
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - modules_variable_list: []
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - modules_setup: false
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - config.test_and_exit: false
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - config.reload.automatic: false
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - config.reload.interval: 3000000000
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - config.support_escapes: false
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - config.field_reference.parser: "STRICT"
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - metric.collect: true
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.id: "main"
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.system: false
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.workers: 8
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.batch.size: 125
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.batch.delay: 50
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.unsafe_shutdown: false
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.java_execution: true
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.reloadable: true
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.plugin_classloaders: false
[DEBUG] 2020-05-06 15:42:52.828 [LogStash::Runner] runner - pipeline.separate_logs: false
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - path.plugins: []
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - *config.debug: true (default: false)
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - *log.level: "debug" (default: "info")
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - version: false
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - help: false
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - log.format: "plain"
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - http.host: "127.0.0.1"
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - http.port: 9600..9700
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - http.environment: "production"
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.type: "memory"
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.drain: false
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.page_capacity: 67108864
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.max_bytes: 1073741824
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.max_events: 0
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.checkpoint.acks: 1024
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.checkpoint.writes: 1024
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.checkpoint.interval: 1000
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - queue.checkpoint.retry: false
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - dead_letter_queue.enable: false
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - dead_letter_queue.max_bytes: 1073741824
[DEBUG] 2020-05-06 15:42:52.829 [LogStash::Runner] runner - slowlog.threshold.warn: -1
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - slowlog.threshold.info: -1
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - slowlog.threshold.debug: -1
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - slowlog.threshold.trace: -1
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - keystore.file: "/usr/share/logstash/config/logstash.keystore"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - path.queue: "/usr/share/logstash/data/queue"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - path.logs: "/usr/share/logstash/logs"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.management.enabled: false
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.management.logstash.poll_interval: 5000000000
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.management.pipeline.id: ["main"]
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.management.elasticsearch.username: "logstash_system"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.management.elasticsearch.sniffing: false
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.monitoring.enabled: false
[DEBUG] 2020-05-06 15:42:52.830 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - xpack.monitoring.collection.interval: 10000000000
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - xpack.monitoring.collection.timeout_interval: 600000000000
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.sniffing: false
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - xpack.monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - xpack.monitoring.collection.config.enabled: true
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - node.uuid: ""
[DEBUG] 2020-05-06 15:42:52.831 [LogStash::Runner] runner - --------------- Logstash Settings -------------------
[WARN ] 2020-05-06 15:42:52.859 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-05-06 15:42:52.865 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.5.0"}
[INFO ] 2020-05-06 15:42:52.887 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"7b4cf07a-8596-4e02-bdb7-5d691c89df9c", :path=>"/usr/share/logstash/data/uuid"}
[DEBUG] 2020-05-06 15:42:52.933 [LogStash::Runner] agent - Setting up metric collection
[DEBUG] 2020-05-06 15:42:53.009 [LogStash::Runner] os - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2020-05-06 15:42:53.243 [LogStash::Runner] jvm - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2020-05-06 15:42:53.334 [LogStash::Runner] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2020-05-06 15:42:53.338 [LogStash::Runner] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2020-05-06 15:42:53.360 [LogStash::Runner] persistentqueue - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2020-05-06 15:42:53.372 [LogStash::Runner] deadletterqueue - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2020-05-06 15:42:53.426 [Agent thread] agent - Starting agent
[DEBUG] 2020-05-06 15:42:53.496 [Agent thread] configpathloader - Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/etc/logstash/conf.d", "/etc/logstash/jvm.options", "/etc/logstash/log4j2.properties", "/etc/logstash/logstash-sample.conf", "/etc/logstash/logstash.yml", "/etc/logstash/startup.options"]}
[DEBUG] 2020-05-06 15:42:53.498 [Agent thread] configpathloader - Reading config file {:config_file=>"/etc/logstash/pipelines.yml"}
[DEBUG] 2020-05-06 15:42:53.518 [Agent thread] pipelineconfig - -------- Logstash Config ---------
[DEBUG] 2020-05-06 15:42:53.518 [Agent thread] pipelineconfig - Config from source {:source=>LogStash::Config::Source::Local, :pipeline_id=>:main}
[DEBUG] 2020-05-06 15:42:53.520 [Agent thread] pipelineconfig - Config string {:protocol=>"file", :id=>"/etc/logstash/pipelines.yml"}
DWbank
May 6, 2020, 7:54pm
3
Part 2
[DEBUG] 2020-05-06 15:42:53.520 [Agent thread] pipelineconfig -
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
- pipeline.id: alienvault
path.config: "/etc/logstash/conf.d/logstash.conf"
pipeline.workers: 3
- pipeline.id: saslogs
path.config: "/etc/logstash/conf.d/sas.conf"
queue.type: persisted
[DEBUG] 2020-05-06 15:42:53.521 [Agent thread] pipelineconfig - Merged config
[DEBUG] 2020-05-06 15:42:53.522 [Agent thread] pipelineconfig -
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
- pipeline.id: alienvault
path.config: "/etc/logstash/conf.d/logstash.conf"
pipeline.workers: 3
- pipeline.id: saslogs
path.config: "/etc/logstash/conf.d/sas.conf"
queue.type: persisted
[DEBUG] 2020-05-06 15:42:53.546 [Agent thread] agent - Converging pipelines state {:actions_count=>1}
[DEBUG] 2020-05-06 15:42:53.558 [Converge PipelineAction::Create<main>] agent - Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
[ERROR] 2020-05-06 15:42:53.751 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 5, column 1 (byte 221) after # This file is where you define your pipelines. You can define multiple.\n# For more information on multiple pipelines, see the documentation:\n# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:156:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
[DEBUG] 2020-05-06 15:42:53.805 [Api Webserver] agent - Starting puma
[DEBUG] 2020-05-06 15:42:53.813 [Api Webserver] agent - Trying to start WebServer {:port=>9600}
[DEBUG] 2020-05-06 15:42:53.814 [LogStash::Runner] os - Stopping
[DEBUG] 2020-05-06 15:42:53.826 [LogStash::Runner] jvm - Stopping
[DEBUG] 2020-05-06 15:42:53.827 [LogStash::Runner] persistentqueue - Stopping
[DEBUG] 2020-05-06 15:42:53.828 [LogStash::Runner] deadletterqueue - Stopping
[DEBUG] 2020-05-06 15:42:53.833 [LogStash::Runner] agent - Shutting down all pipelines {:pipelines_count=>0}
[DEBUG] 2020-05-06 15:42:53.835 [LogStash::Runner] agent - Converging pipelines state {:actions_count=>0}
[DEBUG] 2020-05-06 15:42:53.853 [Api Webserver] service - [api-service] start
[INFO ] 2020-05-06 15:42:53.962 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-06 15:42:59.016 [LogStash::Runner] runner - Logstash shut down.
That's the problem. If you are going to use -f it has to point to a configuration file, not a YAML file. If you want to use multiple pipelines then let logstash find pipelines.yml by itself in path.settings, which for you is
runner - path.settings: "/usr/share/logstash/config"
1 Like
DWbank
May 8, 2020, 2:38pm
5
The only way I could figure out how to get debug logs was to run that command, I normally just do a service logstash start. but I am unable to find the logs that start. they are not in /var/log/logstash.
If i just start it as normal using service it starts fine, but does not start the second pipeline, i know it reads the file because I changed the name of first pipeline from main to alienvault. Now it says alienvault in the logs. Logstash does not seem to start the second pipeline.
netstat -an |grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 10.xx.xx.68:5601 0.0.0.0:* LISTEN
tcp6 0 0 :::5142 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 127.0.0.1:9600 :::* LISTEN
tcp6 0 0 10.xx.xx.68:9200 :::* LISTEN
tcp6 0 0 10.xx.xx.68:9300 :::* LISTEN
Iptables
#iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
#iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
config directory
What else can I give to show whats going on.
system
June 5, 2020, 2:38pm
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.