Import CSV with different column names to same field

Elastic Stack Logstash
4

Ok so, admitting you have files like the following:
image

image

You can even do a single pipeline like the following:

input {
  file {
    path => "/absolute_path_to_file1.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => ["file1"]
  }

  file {
    path => "absolute_path_to_file2.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => ["file2"]
  }
}

filter {
  if "file1" in [tags] {
    csv {
      separator => ","
      columns => ["ID","Date","Name"]
    }

    if [ID] == "ID" {
      drop{}
    }
  } else if "file2" in [tags] {
    csv {
      separator => ","
      columns => ["ID","Ref","Subject","Operator"]
    }

    if [ID] == "Identifier" {
      drop{}
    }
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "test"
    document_id => "%{ID}"
    action => "update"
    doc_as_upsert => true
  }
}

This way, you don't even need to rename anything, since you're directly mapping the Identifier value to the ID field.

In your elasticsearch you'll have something like this:

image
image1839×636 78.4 KB

Does this answer to all your questions?