# ES 6.5.3 : having two separate logstash configuration files (and separate indexes) is causing fields from one index to bleed into the other
**URL:** https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736
**Category:** Logstash
**Created:** [October 29, 2020, 6:11pm UTC](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736 "2020-10-29T18:11:55Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@terry433iid](https://discuss.elastic.co/u/terry433iid)
#### Post date: [October 29, 2020, 6:11pm UTC](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736/1 "2020-10-29T18:11:55Z")
I have two logstash configuration files - when I run logstash it is creating a bizarre situation where one of the indexes from config1 has fields from the other index in config2
See below for steps :-
1. Create an index for CSV files (java class/method signiatures)
* * *
```
elk>cat create_sig_profiler_index.json
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
},
"mappings": {
"doc": {
"properties": {
"testsuite_sp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"java_version_sp": {
"type": "integer"
},
"OS_sp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"platform_sp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"class_method_sp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"checksum_sp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"count_sp": {
"type": "integer"
}
}
}
}
}
```
* * *
1. Create index the index in ES
` elk>curl -s -XPUT http://localhost:9200/sig_profiler -H 'Content-Type: application/json' -d @create_sig_profiler_index.json {"acknowledged":true,"shards_acknowledged":true,"index":"sig_profiler"}`
* * *
1. Verify logstash config files in place (2 files - 2 separate indexes, one for CSV files, the other for jenkins console logs)
` twhelan@qa-elk>ls -l /etc/logstash/conf.d/*.conf -rw-r--r--. 1 root root 4948 Oct 29 11:55 /etc/logstash/conf.d/10-jenkins-qa.conf -rw-r--r--. 1 root root 448 Oct 29 16:50 /etc/logstash/conf.d/sig_profiler.conf`
* * *
1. Verify CSV logstash configuration (file input for CSV files)
* * *
1. Start logstash
`twhelan@qa-elk>sudo systemctl start logstash `
2. Now check sig\_profiler index - is corrupted, it has fields from other logstash configuration (??)
` twhelan@qa-elk>curl -XGET "http://localhost:9200/sig_profiler?pretty" { "sig_profiler" : { "aliases" : { }, "mappings" : { "doc" : { "properties" : { "@timestamp" : { "type" : "date" }, "@version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "OS_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "branch_name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "build" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "checksum_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "class_method_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "console_data" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "count_sp" : { "type" : "integer" }, "finish" : { "type" : "date" }, "guest_vm_version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "host_vm_version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "java_version_sp" : { "type" : "integer" }, "jenkins_job_logs_id" : { "type" : "long" }, "jenkins_job_number" : { "type" : "long" }, "os_name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "owner" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "platform_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "start" : { "type" : "date" }, "state" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "task_id" : { "type" : "long" }, "testsuite" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "testsuite_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } }, "settings" : { "index" : { "creation_date" : "1603989435423", "number_of_shards" : "1", "number_of_replicas" : "1", "uuid" : "7W2bXWWWRoi92TB01zlZvA", "version" : { "created" : "6050399" }, "provided_name" : "sig_profiler" } } } }`
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 29, 2020, 7:00pm UTC](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736/2 "2020-10-29T19:00:09Z")
See [this](https://discuss.elastic.co/t/entries-duplication/247448) post.
---
### Author:  [@terry433iid](https://discuss.elastic.co/u/terry433iid)
#### Post date: [October 30, 2020, 12:52pm UTC](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736/3 "2020-10-30T12:52:53Z")
I have attempted to create 2 separate pipelines (using same directory for the configuration files)
This has not fixed my issue as the index I want to create ('sig\_profiler') is now **not** getting created - see below for details
logstash settings
```
elk>cat /etc/logstash/logstash.yml |grep -v '^#'
path.data: /var/lib/logstash
path.logs: /var/log/logstash
```
Verify separate pipelines setup
(pipeline 1 is for jenkins logs, pipeline2 is for java class/method signatures i.e. sig\_profiler)
```
elk>cat pipelines.yml
- pipeline.id: pipeline_1
path.config: "/etc/logstash/conf.d/10-jenkins-qa.conf"
- pipeline.id: pipeline_2
path.config: "/etc/logstash/conf.d/sig_profiler.conf"
```
Verify two separate logstash configurations exist
```
elk>pwd
/etc/logstash/conf.d
elk>ls -l
total 12
-rw-r--r--. 1 root root 4948 Oct 29 11:55 10-jenkins-qa.conf
-rw-r--r--. 1 root root 448 Oct 30 12:33 sig_profiler.conf
```
Verify setup for importing CSV files (java class/method signatures)
```
elk>cat sig_profiler.conf
input {
file {
path => "/PROFILER/*.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["sp_testsuite", "sp_java_version", "sp_java_vendor", "sp_OS", "sp_platform", "sp_class_method", "sp_checksum", "sp_count"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "sig_profiler"
}
}
```
Verify that sig\_profiler index does not exist before starting logstash
```
elk>curl -XGET "http://localhost:9200/sig_profiler"
{"error":{"root_cause":[{"type":"index_not_found_exception"
```
Start logstash and check logs
```
[2020-10-30T12:40:27,237][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.5.3"}
[2020-10-30T12:40:30,188][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"pipeline_2", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-10-30T12:40:32,185][INFO][logstash.inputs.file] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_1b029dd30b68f0f0485a5a0a201bbbb7", :path=>["/PROFILER/*.csv"]}
[2020-10-30T12:40:32,272][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"pipeline_2", :thread=>"#"}
[2020-10-30T12:40:32,412][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections
[2020-10-30T12:40:33,299][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"pipeline_1", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-10-30T12:40:34,136][INFO][org.logstash.beats.Server] Starting server on port: 5044
[2020-10-30T12:40:34,274][INFO][logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:pipeline_2, :pipeline_1], :non_running_pipelines=>[]}
[2020-10-30T12:40:34,772][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
```
Now check if sig\_profiler index has been created - it has not been created (??)
```
elk>curl -XGET "http://localhost:9200/sig_profiler"
{"error":{"root_cause":[{"type":"index_not_found_exception"
```
---
### Author:  [@terry433iid](https://discuss.elastic.co/u/terry433iid)
#### Post date: [October 30, 2020, 1:23pm UTC](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736/4 "2020-10-30T13:23:41Z")
SOLVED
the problem was that the directory /PROFILER/\*.csv was not readable by logstash user
So I simply moved the CSV files to a directory readable by logstash user and the index got created once I restarted logstash
```
elk>ls -l /opt/logstash/
total 20836
-rw-r--r--. 1 logstash logstash 1098450 Oct 30 13:11 sig_1032668857827157276.csv
-rw-r--r--. 1 logstash logstash 466460 Oct 30 13:11 sig_1052866132025040573.csv
-rw-r--r--. 1 logstash logstash 978012 Oct 30 13:11 sig_1071291317503287906.csv
-rw-r--r--. 1 logstash logstash 1210187 Oct 30 13:11 sig_1106475576936750771.csv
-rw-r--r--. 1 logstash logstash 3410465 Oct 30 13:11 sig_1204491123082986398.csv
-rw-r--r--. 1 logstash logstash 3262800 Oct 30 13:11 sig_124029123589223571.csv
-rw-r--r--. 1 logstash logstash 1057032 Oct 30 13:11 sig_1247352807478409364.csv
-rw-r--r--. 1 logstash logstash 1903764 Oct 30 13:11 sig_1249773956029632223.csv
-rw-r--r--. 1 logstash logstash 1098396 Oct 30 13:11 sig_401140205105203855.csv
-rw-r--r--. 1 logstash logstash 6830750 Oct 30 13:11 sig_6636585400286033712.csv
```
here is index
twhelan@qa-elk\>curl -XGET "[http://localhost:9200/sig\_profiler](http://localhost:9200/sig_profiler)"
{"sig\_profiler":{"aliases":{},"mappings":{"doc":{"properties":{"@timestamp":{"type":"date"},"@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"message":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"path":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_OS":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_checksum":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_class\_method":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_count":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_java\_vendor":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_java\_version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_platform":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"sp\_testsuite":{"type":"text","fields":{"keyword":{"type":"keyword","ignore\_above":256}}}}}},"settings":{"index":{"creation\_date":"1604063696594","number\_of\_shards":"5","number\_of\_replicas":"1","uuid":"s1K0TjwiSs6IQtuxgpN5aQ","version":{"created":"6050399"},"provided\_name":"sig\_profiler"}}}}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 27, 2020, 1:23pm UTC](https://discuss.elastic.co/t/es-6-5-3-having-two-separate-logstash-configuration-files-and-separate-indexes-is-causing-fields-from-one-index-to-bleed-into-the-other/253736/5 "2020-11-27T13:23:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.