# Merge data and metadata csv file in logstash

**URL:** <https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815>\
**Category:** Logstash\
**Created:** [January 31, 2022, 10:08am UTC](https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815 "2022-01-31T10:08:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![danova](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@danova](https://discuss.elastic.co/u/danova)\
**Post date:** [January 31, 2022, 10:08am UTC](https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815/1 "2022-01-31T10:08:00Z")

</div>

Hi,  
I´m trying to import/combine two csv files as follows

**logstash\_metadata.csv:**

> **Sample** ,Treatment,Code  
> S4444003,T\_7896\_D3,G10  
> S4444004,T\_4516\_D0t1h,G01

**logstash\_file.csv:**

> **Sample** ,genus,value  
> S4444003,Chloronema,8  
> S4444003,Pseudaminobacter,3

when combining both files i use the Sample field.

```auto
input {
  file {
    path => "logstash_file.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    tags => ["16S"]
  }
}
filter {
  csv {
    skip_header => false
    columns => ["Sample","genus","value"]
    separator => ","

  }
  mutate {
    convert => {
      "value" => "integer"
    }
    }

  translate {
     target => "[@metadata][match]" 
     dictionary_path => "logstash_metadata.csv" 
     source => "Sample" 
           }
   dissect {
    mapping => { 
     "message" => "%{Sample},%{Treatment},%{Code}"
  }
   }

date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    remove_field => ["timestamp"]
  }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "16s_s4444"
    data_stream => false

  }
  stdout {codec => rubydebug {metadata => true}}
}

```

The output is as follows:

```auto

{
             "path" => "logstash_file.csv",
       "@timestamp" => 2022-01-31T09:30:05.998Z,
            "genus" => "Pseudaminobacter",
           "Sample" => "S4444003",
       "Code" => "3",
        "@metadata" => {
         "path" => "logstash_file.csv",
        "match" => "7896_D3",
         "host" => "localhost"
    },
         "@version" => "1",
             "host" => "localhost",
    "Treatment" => "Pseudaminobacter",
          "message" => "S4444003,Pseudaminobacter,3",
            "value" => 3,
             "tags" => [
        [0] "16S"
    ]
}

```

Both the Treatment and the Code are not assigned properly. Treatment should be T\_7896 and Code should be G10.

I probably got something wrong, can you help ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 31, 2022, 6:46pm UTC](https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815/2 "2022-01-31T18:46:19Z")

</div>

When a translate filter [loads a csv](https://github.com/logstash-plugins/logstash-filter-translate/blob/e3670bb3137437ad05a90036a90b4711a67e04ea/lib/logstash/filters/dictionary/csv_file.rb#L21) it only uses the first two columns.

[Treatment] is incorrect because in your dissect filter you are mapping [message], when you probably meant to map [@metadata][match].

If you change the metadata csv to be

> **Sample** ,Treatment,Code  
> S4444003,T\_7896\_D3;G10  
> S4444004,T\_4516\_D0t1h;G01

using semi-colon as the separator for the second and third columns you could use

```
dissect { mapping => { "[@metadata][match]" => "%{Treatment};%{Code}" } }

```

[This](https://discuss.elastic.co/t/translate-filter-plugin/167971) thread discusses other options.

---

<div class="post-metadata">

**Author:** ![danova](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@danova](https://discuss.elastic.co/u/danova)\
**Post date:** [January 31, 2022, 8:13pm UTC](https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815/3 "2022-01-31T20:13:05Z")

</div>

Great !!! Works perfect.

However, and maybe this is expected , I have a strange warning, that i do not understand. See below.

```auto
[2022-01-31T21:06:13,737][WARN][org.logstash.dissect.Dissector][main][05e203d1d6dfb3d7d35f2e0da7b0e487a5e172bd69729fe989b65ade05620b48] Dissector mapping, pattern not found {"field"=>"[@metadata][match]", "pattern"=>"%{Treatment};%{Code}", "event"=>{"@version"=>"1", "Sample"=>"Sample", "value"=>0, "@timestamp"=>2022-01-31T20:06:12.911Z, "host"=>"localhost", "path"=>"logstash_file.csv", "tags"=>["16S", "_dissectfailure"], "message"=>"Sample,genus,value", "genus"=>"genus"}}
{

```

This how my code looks now:

```auto

translate {
     target => "[@metadata][match]" 
     dictionary_path => "logstash_metadata.csv" 
     source => "Sample" 
           }
   dissect {
    mapping => {"[@metadata][match]" => "%{TreatmentTime};%{Donor.Code}"}}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2022, 8:13pm UTC](https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815/4 "2022-02-28T20:13:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.