ES 6.5.3 : having two separate logstash configuration files (and separate indexes) is causing fields from one index to bleed into the other

I have two logstash configuration files - when I run logstash it is creating a bizarre situation where one of the indexes from config1 has fields from the other index in config2

See below for steps :-

  1. Create an index for CSV files (java class/method signiatures)

elk>cat create_sig_profiler_index.json
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "doc": {
      "properties": {
        "testsuite_sp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "java_version_sp": {
          "type": "integer"
        },
        "OS_sp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "platform_sp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "class_method_sp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "checksum_sp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "count_sp": {
          "type": "integer"
        }
      }
    }
  }
}

  1. Create index the index in ES
    elk>curl -s -XPUT http://localhost:9200/sig_profiler -H 'Content-Type: application/json' -d @create_sig_profiler_index.json {"acknowledged":true,"shards_acknowledged":true,"index":"sig_profiler"}

  1. Verify logstash config files in place (2 files - 2 separate indexes, one for CSV files, the other for jenkins console logs)
    twhelan@qa-elk>ls -l /etc/logstash/conf.d/*.conf -rw-r--r--. 1 root root 4948 Oct 29 11:55 /etc/logstash/conf.d/10-jenkins-qa.conf -rw-r--r--. 1 root root 448 Oct 29 16:50 /etc/logstash/conf.d/sig_profiler.conf

  1. Verify CSV logstash configuration (file input for CSV files)

    elk>cat sig_profiler.conf
    input {
    file {
    path => "/PROFILER/*.csv"
    start_position => "beginning"
    }
    }

    filter {
    csv {
    separator => ","
    columns => [ "sp_testsuite", "sp_java_version", "sp_java_vendor", "sp_OS", "sp_platform", "sp_class_method", "sp_checksum", "sp_count" ]
    }
    }

    output {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "sig_profiler"
    }
    }


  1. Start logstash
    twhelan@qa-elk>sudo systemctl start logstash

  2. Now check sig_profiler index - is corrupted, it has fields from other logstash configuration (??)
    twhelan@qa-elk>curl -XGET "http://localhost:9200/sig_profiler?pretty" { "sig_profiler" : { "aliases" : { }, "mappings" : { "doc" : { "properties" : { "@timestamp" : { "type" : "date" }, "@version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "OS_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "branch_name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "build" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "checksum_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "class_method_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "console_data" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "count_sp" : { "type" : "integer" }, "finish" : { "type" : "date" }, "guest_vm_version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "host_vm_version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "java_version_sp" : { "type" : "integer" }, "jenkins_job_logs_id" : { "type" : "long" }, "jenkins_job_number" : { "type" : "long" }, "os_name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "owner" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "platform_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "start" : { "type" : "date" }, "state" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "task_id" : { "type" : "long" }, "testsuite" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "testsuite_sp" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } }, "settings" : { "index" : { "creation_date" : "1603989435423", "number_of_shards" : "1", "number_of_replicas" : "1", "uuid" : "7W2bXWWWRoi92TB01zlZvA", "version" : { "created" : "6050399" }, "provided_name" : "sig_profiler" } } } }

See this post.

I have attempted to create 2 separate pipelines (using same directory for the configuration files)
This has not fixed my issue as the index I want to create ('sig_profiler') is now not getting created - see below for details

logstash settings

elk>cat /etc/logstash/logstash.yml |grep -v '^#'
path.data: /var/lib/logstash
path.logs: /var/log/logstash

Verify separate pipelines setup
(pipeline 1 is for jenkins logs, pipeline2 is for java class/method signatures i.e. sig_profiler)

elk>cat pipelines.yml 
- pipeline.id: pipeline_1
  path.config: "/etc/logstash/conf.d/10-jenkins-qa.conf"
- pipeline.id: pipeline_2
  path.config: "/etc/logstash/conf.d/sig_profiler.conf"

Verify two separate logstash configurations exist

elk>pwd
/etc/logstash/conf.d

elk>ls -l
total 12
-rw-r--r--. 1 root root 4948 Oct 29 11:55 10-jenkins-qa.conf
-rw-r--r--. 1 root root  448 Oct 30 12:33 sig_profiler.conf

Verify setup for importing CSV files (java class/method signatures)

elk>cat sig_profiler.conf 
input {
  file {
    path => "/PROFILER/*.csv"
    start_position => "beginning"
  }
}

filter {
      csv {
        separator => ","
        columns => [ "sp_testsuite", "sp_java_version", "sp_java_vendor", "sp_OS", "sp_platform", "sp_class_method", "sp_checksum", "sp_count" ]
     }
}

output {
        elasticsearch { 
               hosts => ["localhost:9200"] 
               index => "sig_profiler"
        }
}

Verify that sig_profiler index does not exist before starting logstash

elk>curl -XGET "http://localhost:9200/sig_profiler"
{"error":{"root_cause":[{"type":"index_not_found_exception"

Start logstash and check logs

[2020-10-30T12:40:27,237][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.3"}
[2020-10-30T12:40:30,188][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"pipeline_2", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[2020-10-30T12:40:32,185][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_1b029dd30b68f0f0485a5a0a201bbbb7", :path=>["/PROFILER/*.csv"]}

[2020-10-30T12:40:32,272][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"pipeline_2", :thread=>"#<Thread:0x87e94f3@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:51 run>"}

[2020-10-30T12:40:32,412][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections

[2020-10-30T12:40:33,299][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"pipeline_1", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}

[2020-10-30T12:40:34,136][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2020-10-30T12:40:34,274][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:pipeline_2, :pipeline_1], :non_running_pipelines=>[]}
[2020-10-30T12:40:34,772][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Now check if sig_profiler index has been created - it has not been created (??)

elk>curl -XGET "http://localhost:9200/sig_profiler"
{"error":{"root_cause":[{"type":"index_not_found_exception"

SOLVED
the problem was that the directory /PROFILER/*.csv was not readable by logstash user
So I simply moved the CSV files to a directory readable by logstash user and the index got created once I restarted logstash

elk>ls -l /opt/logstash/
total 20836
-rw-r--r--. 1 logstash logstash 1098450 Oct 30 13:11 sig_1032668857827157276.csv
-rw-r--r--. 1 logstash logstash  466460 Oct 30 13:11 sig_1052866132025040573.csv
-rw-r--r--. 1 logstash logstash  978012 Oct 30 13:11 sig_1071291317503287906.csv
-rw-r--r--. 1 logstash logstash 1210187 Oct 30 13:11 sig_1106475576936750771.csv
-rw-r--r--. 1 logstash logstash 3410465 Oct 30 13:11 sig_1204491123082986398.csv
-rw-r--r--. 1 logstash logstash 3262800 Oct 30 13:11 sig_124029123589223571.csv
-rw-r--r--. 1 logstash logstash 1057032 Oct 30 13:11 sig_1247352807478409364.csv
-rw-r--r--. 1 logstash logstash 1903764 Oct 30 13:11 sig_1249773956029632223.csv
-rw-r--r--. 1 logstash logstash 1098396 Oct 30 13:11 sig_401140205105203855.csv
-rw-r--r--. 1 logstash logstash 6830750 Oct 30 13:11 sig_6636585400286033712.csv

here is index
twhelan@qa-elk>curl -XGET "http://localhost:9200/sig_profiler"
{"sig_profiler":{"aliases":{},"mappings":{"doc":{"properties":{"@timestamp":{"type":"date"},"@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"message":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"path":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_OS":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_checksum":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_class_method":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_count":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_java_vendor":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_java_version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_platform":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sp_testsuite":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}},"settings":{"index":{"creation_date":"1604063696594","number_of_shards":"5","number_of_replicas":"1","uuid":"s1K0TjwiSs6IQtuxgpN5aQ","version":{"created":"6050399"},"provided_name":"sig_profiler"}}}}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.