2 index same data

Elastic Stack Elasticsearch
1

Hi everybody ,

I've two index defined in the same logstash and the origin of the data are two different tables of a database.

When i read the data with logstash and paint them into kibana, the data is mixed , why does it happen?

first index

input {
  jdbc {
    jdbc_connection_string => "jdbc:mysql://localhost:3306/eventos?useSSL=false"
    jdbc_user => "labo"
    jdbc_password => "arcsight_L4B0"
    jdbc_driver_library => "/mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    schedule => "*/10 * * * *"
    statement => "SELECT * FROM eventos where ideventos>:sql_last_value"
    use_column_value => true
    clean_run => false
    tracking_column => ideventos
    jdbc_paging_enabled => true
   jdbc_page_size => 25000
 }
}
output {
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "arcsight"
  }
stdout { codec => json_lines }
}

second index

input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/KPIS?useSSL=false"
jdbc_user => "labo"
jdbc_password => "arcsight_L4B0"
jdbc_driver_library => "mysql-connector-java-5.1.47.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
schedule => "46 8 * * 1,2,3,4,5,6,7"
statement => "Select * FROM KPIS.reglasArcsight where idreglasArcsight>:sql_last_value"
use_column_value => true
clean_run => false
tracking_column => "idreglasarcsight"
jdbc_paging_enabled => true
jdbc_page_size => 250000
last_run_metadata_path => "/usr/share/logstash/.logstash_reglas_jdbc_last_run"
}
}
output {
elasticsearch {
"hosts" => "localhost:9200"
"index" => "reglas"
}
stdout { codec => json_lines }
}

2

May be the index pattern in Kibana ?

3

Arcsight pattern have the same fields as reglas pattern but the structure of the database is different.

4

If you put multiple files in the Logstash config directory these are concatenated into a single pipeline where all input data is processed by all filters and go to all outputs unless you control this through conditionals or use the multi pipeline feature. That is why both indices hold exactly the same data. This is a common misunderstanding and you should be able to find many examples in this forum under the Logstash category.

5

Hi christian,

I'll search for multipipeline
many thanks

6

I've use multipipile

path.config: "/etc/logstash/conf.d/arcsight.conf"

  • pipeline.id: reglas
    path.config: "/etc/logstash/conf.d/reglas.conf"

[2020-04-05T10:45:03,537][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"arcsight", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-04-05T10:45:03,916][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"reglas", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-04-05T10:45:04,511][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2020-04-05T10:45:04,513][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2020-04-05T10:45:04,898][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-05T10:45:04,905][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-05T10:45:05,019][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-04-05T10:45:05,021][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-04-05T10:45:05,022][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2020-04-05T10:45:05,034][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2020-04-05T10:45:05,096][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-04-05T10:45:05,097][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-04-05T10:45:05,128][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-04-05T10:45:05,143][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-04-05T10:45:05,182][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2020-04-05T10:45:05,184][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2020-04-05T10:45:05,519][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"arcsight", :thread=>"#<Thread:0x39783ecf@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:48 run>"}
[2020-04-05T10:45:05,521][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"reglas", :thread=>"#<Thread:0x78940b9d run>"}
[2020-04-05T10:45:05,725][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:arcsight, :reglas], :non_running_pipelines=>}

But the index related to reglas doesnt appear. do i miss sth?

7

You need to specify both pipelines in pipeline.yml. To be sure these are being used I would recommend storing these config files outside the conf.d directory. Please share the full pipelines.yml file and make sure you format it properly.

8

Fixed, many thanks!!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.