I want send a duplicate or clone of my data throught logstash to another kibana/elastic adminitrador

Elastic Stack Logstash
1

how could i duplicate the data or send de same data to another elastic, the logstash version is 7.17 while elastic version where i want to receive is 8.8.2, i try with the output configuration but i received this error

[2023-07-25T16:21:58,097][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:infraestructura, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 53, column 15 (byte 1460) after output {\nif ([event][module] == \"iis\" and [agent][type] == \"filebeat\" ){\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"%{[event][dataset]}\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n} else if [campos][servicio] == \"Servicio Listener Motor de Pago\"{\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"logs_listener\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"listener\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n\n} else {\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => infraestructura\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120 \npool_max => 10 \npool_max_per_route => 10 \nvalidate_after_inactivity => 750\n}\n\n}else{\nelasticsearch ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}
[2023-07-25T16:22:01,104][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:infraestructura, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 53, column 15 (byte 1460) after output {\nif ([event][module] == \"iis\" and [agent][type] == \"filebeat\" ){\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"%{[event][dataset]}\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n} else if [campos][servicio] == \"Servicio Listener Motor de Pago\"{\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"logs_listener\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"listener\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n\n} else {\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => infraestructura\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120 \npool_max => 10 \npool_max_per_route => 10 \nvalidate_after_inactivity => 750\n}\n\n}else{\nelasticsearch ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}
[2023-07-25T16:22:04,096][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:infraestructura, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 53, column 15 (byte 1460) after output {\nif ([event][module] == \"iis\" and [agent][type] == \"filebeat\" ){\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"%{[event][dataset]}\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n} else if [campos][servicio] == \"Servicio Listener Motor de Pago\"{\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"logs_listener\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"listener\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n\n} else {\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => infraestructura\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120 \npool_max => 10 \npool_max_per_route => 10 \nvalidate_after_inactivity => 750\n}\n\n}else{\nelasticsearch ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}
[2023-07-25T16:22:07,101][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:infraestructura, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 53, column 15 (byte 1460) after output {\nif ([event][module] == \"iis\" and [agent][type] == \"filebeat\" ){\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"%{[event][dataset]}\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n} else if [campos][servicio] == \"Servicio Listener Motor de Pago\"{\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"logs_listener\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"listener\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n\n} else {\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => infraestructura\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120 \npool_max => 10 \npool_max_per_route => 10 \nvalidate_after_inactivity => 750\n}\n\n}else{\nelasticsearch ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}
[2023-07-25T16:22:10,102][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:infraestructura, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 53, column 15 (byte 1460) after output {\nif ([event][module] == \"iis\" and [agent][type] == \"filebeat\" ){\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"%{[event][dataset]}\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n} else if [campos][servicio] == \"Servicio Listener Motor de Pago\"{\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"logs_listener\"\nuser => elastic\npassword => tivolitivoli\npipeline => \"listener\"\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120\npool_max => 10\npool_max_per_route => 10\nvalidate_after_inactivity => 750\n}\n\n} else {\nelasticsearch {\nhosts => [\"https://172.16.107.125:9200\",\"https://172.16.107.126:9200\",\"https://172.16.107.69:9200\",\"https://172.16.107.70:9200\"]\nindex => \"monitoreo_infra\"\nuser => elastic\npassword => tivolitivoli\npipeline => infraestructura\ncacert => \"/etc/logstash/elasticsearch-ca.pem\"\nssl_certificate_verification => false\ntimeout => 120 \npool_max => 10 \npool_max_per_route => 10 \nvalidate_after_inactivity => 750\n}\n\n}else{\nelasticsearch ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}

and this is the yml out put to the 2 elastics

input {
  beats{
  port => 5502
}
}

output {
if ([event][module] == "iis" and [agent][type] == "filebeat" ){
elasticsearch {
hosts => ["https://172.176.107.125:9200","https://172.16.107.126:9200","https://172.16.107.679:9200","https://172.176.107.770:9200"]
index => "monitoreo_infra"
user => elastic
password => tivolitivoli
pipeline => "%{[event][dataset]}"
cacert => "/etc/logstash/elasticsearch-ca.pem"
ssl_certificate_verification => false
timeout => 120
pool_max => 10
pool_max_per_route => 10
validate_after_inactivity => 750
}
} else if [campos][servicio] == "Servicio Listener Motor de Pago"{
elasticsearch {
hosts => ["https://172.16.107.125:9200","https://172.16.107.126:9200","https://172.16.107.69:9200","https://172.16.107.70:9200"]
index => "logs_listener"
user => elastic
password => tivolitivoli
pipeline => "listener"
cacert => "/etc/logstash/elasticsearch-ca.pem"
ssl_certificate_verification => false
timeout => 120
pool_max => 10
pool_max_per_route => 10
validate_after_inactivity => 750
}

} else {
elasticsearch {
hosts => ["https://154.59.63:9200","https://172.16.107.126:9200","https://180.36.100.29:9200","https://172.16.90.50:9200"]
index => "monitoreo_infra"
user => elastic
password => tivolitivoli
pipeline => infraestructura
cacert => "/etc/logstash/elasticsearch-ca.pem"
ssl_certificate_verification => false
timeout => 120
pool_max => 10
pool_max_per_route => 10
validate_after_inactivity => 750
}
}
}
} else {
elasticsearch {
hosts => ["https://172.116.235.537:9200","https://172.126.355.438:9200","https://172.146.355.359:9200","https://182.156.355.50:9200"]
index => "monitoreo_infra"
user => elastic
password => tivolitivoli
pipeline => infraestructura
cacert => "/monit/ca/ca.crt"
ssl_certificate_verification => false
timeout => 120
pool_max => 10
pool_max_per_route => 10
validate_after_inactivity => 750
}
}
}
2

Hi,

Yes, you can send your messages to as many outputs as you want. The logmessages you are listing are typical for an error (like a typo or wrong bracket) in your Logstash configuration. I assume it's the double else. You can only have one else, all others in the list have to be else if.

Remember, that, if one output fails, all other will stop, too. So it might be better to use distinct pipelines and have a broker like Redis between them.

2 Likes
3

thank u i already solved it was permision file the old cluster, nothing to be with an logshtash error, but thanks.

4

yes it also was (like a typo or wrong bracket) thanks

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.