# Different files are entered into logstash . kibana fields show the same **URL:** \ **Category:** Logstash\ **Created:** [December 18, 2018, 5:54am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243 "2018-12-18T05:54:55Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![CCH0124](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cch0124/32/36436_2.png) [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Post date:** [December 18, 2018, 5:54am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/1 "2018-12-18T05:54:55Z")
Use filebeat to pass the json file to logstash. Read the CSV file using logstash. Both are different indexes. But the two indexes on kibana are reading the same csv file. How to solve it ? ### json #### filebeat ```auto filebeat.config: inputs: enabled: true path: /usr/share/filebeat/*.yml reload.enabled: true reload.period: 10s modules: enabled: true filebeat.inputs: - input_type: log paths: - "/usr/share/filebeat/DataSet/attack-trace.json" document_type: "pcap_file" json.keys_under_root: true json.add_error_key: true output.logstash: enable: true hosts: "logstash:5555" ssl.enabled: false loadbalance: false setup.kibana: enabled: true host: "kibana:5601" ssl.enabled: false ``` #### logstash ```auto input { beats { port => 5555 tags => "pcap" #client_inactivity_timeout => "1200" codec => "json" } } filter { } output { elasticsearch { hosts => "elasticsearch:9200" index => "filebeat-pacp-%{+YYYY.MM.dd}" } } ``` ### CSV #### logstash ```auto input { file { path => ["/usr/share/logstash/DataSet/TBrain_IPS.csv"] start_position => "beginning" sincedb_path => "/dev/null" } } filter { csv { separator => "," autodetect_column_names=> true skip_empty_columns=> false skip_empty_rows=> false skip_header=> false id => "ips" } mutate { split => { "event_rule_reference" => ";" } } #ruby { # code => "event['event_rule_reference'] = event['event_rule_reference'].keys" #} } output { elasticsearch { hosts => "elasticsearch:9200" index => "ips" } stdout { codec => rubydebug } } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 18, 2018, 8:46am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/2 "2018-12-18T08:46:44Z")
When you put more than one config file in the directory, the files are concatenated and not treated as separate pipelines. Data from all inputs will therefore go to all outputs. You can get around this by using conditionals or the multi-pipeline feature. This is a common misunderstanding, so you should be able to find plenty of examples if you search this forum. ---
**Author:** ![CCH0124](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cch0124/32/36436_2.png) [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Post date:** [December 18, 2018, 9:04am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/3 "2018-12-18T09:04:01Z")
> [@Christian\_Dahlqvist](#): > > 條件或多管道功能來解決這個問題。 Can you give an example? Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 18, 2018, 9:13am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/4 "2018-12-18T09:13:33Z")
Did you not find anything when you searched the forum? > [@Logstash 6.4.x : reading multiple files in conf.d](https://discuss.elastic.co/t/logstash-6-4-x-reading-multiple-files-in-conf-d/149515/3): > > Hello Thanks you Christian for your interest. I create 2 separate index, my 2 conf files are: root@kvm:~# cat /etc/logstash/conf.d/auth.conf input { tcp { port =\> "5001" codec =\> json tags =\> ["syslogauth"] } } filter { grok { named\_captures\_only =\> false break\_on\_match =\> true match =\> { "message" =\> [" New session %{NUMBER} of user %{USERNAME:user}."," Accepted password for %{USERNAME:user} from %{IP:ip} port %{NUMBER} ssh2"," Failed passw… > [@Logstash sends multiple copies of data](https://discuss.elastic.co/t/logstash-sends-multiple-copies-of-data/160267/2): > > All config files in the directory are concatenated into a single pipeline. This means that each event generated by an input plugin will go through all filters and be sent to all outputs (all 5 of them). You can get around this by creating a single config file with multiple inputs, use conditionals or use the relatively new multiple pipeline feature. This is a common misunderstanding so you should easily be able to find examples. > [@Message Created Twice in Elasticsearch via Logstash](https://discuss.elastic.co/t/message-created-twice-in-elasticsearch-via-logstash/142020): > > When I ingest document via logstash, I found the documents are created twice. Setup: Filebeat -\> Logstash -\> Elasticsearch Result: Same messages are ingested into Elasticsearch with different \_id So I tried the following testings: Testing 1: Inget Document directly from Filebeat i.e. Filebeat -\> Elasticsearch Result: 1 document is created Finding: the issue should be related to logstash or elasticsearch After reading the article: [https://www.elastic.co/blog/logstash-lessons-handling-du…](https://www.elastic.co/blog/logstash-lessons-handling-duplicates) ---
**Author:** ![CCH0124](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cch0124/32/36436_2.png) [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Post date:** [December 19, 2018, 5:24am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/5 "2018-12-19T05:24:02Z")
我使用 pipelines.yml 定義每個 pipeline 但是果還是一樣。 #### logstash .conf ```auto logstash.conf metricbeat.conf pcap.conf readCSV.conf ``` #### pipelines.yml ```auto - pipeline.id: csv path.config: "/usr/share/logstash/pipeline/readCSV.conf" - pipeline.id: pcap path.config: "/usr/share/logstash/pipeline/pcap.conf" - pipeline.id: metricbeat path.config: "/usr/share/logstash/pipeline/metricbeat.conf" ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2018, 7:33am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/6 "2018-12-19T07:33:15Z")
Is the `pipelines.yml` file in the correct location so it is being picked up? Is there any chance it is still picking up the config from the old location? ---
**Author:** ![CCH0124](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cch0124/32/36436_2.png) [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Post date:** [December 19, 2018, 7:52am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/7 "2018-12-19T07:52:08Z")
It is in the right place. ```auto bash-4.2$ cd config/ bash-4.2$ ls jvm.options log4j2.properties logstash-sample.conf logstash.yml pipelines.yml startup.options bash-4.2$ cat pipelines.yml - pipeline.id: csv path.config: "/usr/share/logstash/pipeline/readCSV.conf" - pipeline.id: pcap path.config: "/usr/share/logstash/pipeline/pcap.conf" - pipeline.id: metricbeat path.config: "/usr/share/logstash/pipeline/metricbeat.conf" ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2018, 8:00am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/8 "2018-12-19T08:00:56Z")
How do you start Logstash? What does the logs say? ---
**Author:** ![CCH0124](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cch0124/32/36436_2.png) [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Post date:** [December 20, 2018, 12:21am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/9 "2018-12-20T00:21:21Z")
> [@Christian\_Dahlqvist](#): > > multi-pipeline Run logstash with Docker #### log ```shell Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties [2018-12-19T10:24:53,713][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [2018-12-19T10:24:53,728][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [2018-12-19T10:24:54,197][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-12-19T10:24:54,213][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.5.0"} [2018-12-19T10:24:54,245][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=>"cb3e0911-1bbc-4931-9c72-8ef208d4edb4", :path=>"/usr/share/logstash/data/uuid"} [2018-12-19T10:24:58,254][WARN][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=>"metricbeat-%{+YYYY.MM.dd}", manage_template=>false, id=>"fe38e84cf8a941fbf650bf7af553dcabeae46374ccb6d0668bec69b2cef3468b", hosts=>[//elasticsearch:9200], document_type=>"metricbeat-system", enable_metric=>true, codec=>"plain_1b04fb65-55fe-4b38-a30e-d8e528e940fc", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>} [2018-12-19T10:25:00,843][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2018-12-19T10:25:01,564][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}} [2018-12-19T10:25:01,577][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"} [2018-12-19T10:25:01,936][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"} [2018-12-19T10:25:02,007][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2018-12-19T10:25:02,012][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2018-12-19T10:25:02,035][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]} [2018-12-19T10:25:02,070][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}} [2018-12-19T10:25:02,072][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"} [2018-12-19T10:25:02,079][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil} [2018-12-19T10:25:02,087][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"} [2018-12-19T10:25:02,100][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2018-12-19T10:25:02,101][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2018-12-19T10:25:02,116][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]} [2018-12-19T10:25:02,126][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}} [2018-12-19T10:25:02,128][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}} ``` ---
**Author:** ![CCH0124](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cch0124/32/36436_2.png) [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Post date:** [December 20, 2018, 10:13am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/10 "2018-12-20T10:13:40Z")
I have already solved it. The reason is that logstash.yml has a path.confing which affects it. After it was annotated, everything went well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 17, 2019, 10:13am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243/11 "2019-01-17T10:13:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.