# Elastic Filter

**URL:** <https://discuss.elastic.co/t/elastic-filter/329104>\
**Category:** Logstash\
**Created:** [April 1, 2023, 2:04pm UTC](https://discuss.elastic.co/t/elastic-filter/329104 "2023-04-01T14:04:26Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![oleksiiorel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiiorel/32/118205_2.png) [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\
**Post date:** [April 1, 2023, 2:04pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/1 "2023-04-01T14:04:26Z")

</div>

I import a csv file via logstash "filter cvs" into Elasticsearch. One of the cells in a table (CVS file) contains several strings example: (categoty, subcategory, sub\_subcategory). I would like to split these strings into separate cells (fields).  
On the forums people say that I need to use "add\_field filter", but how exactly I do not know.

Example:  
ID|Name| Category \column names in cvs  
1, Sofa, "Furniture, Soft furnishings, TestString" \row in CVS

What I need to be stored in ELK  
"\_source": {  
"ID": 1,  
"Name": Sofa,  
"Category": Furniture,  
"SubCategory": "Soft furnishings",  
"SubSubCategory": "TestString",  
}  
What I did :  
input{  
file {  
path =\> "E:/ELK/db/db.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "NULL"  
}   
}

filter{

csv {  
separator =\> ";"  
columns =\> ["Id","Name","Category"]  
}  
}  
output{  
elasticsearch {  
hosts =\> "[http://localhost:9200](http://localhost:9200)"  
index =\> "products"  
user =\> "elastic"  
password =\> "\*\*"  
}  
stdout{}  
}

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2023, 2:19pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/2 "2023-04-01T14:19:42Z")

</div>

Hi @oleksiiorel Welcome to the community.

Perhaps [dissect](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html) is a better choice, it has more control and is also more performant

---

<div class="post-metadata">

**Author:** ![oleksiiorel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiiorel/32/118205_2.png) [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\
**Post date:** [April 1, 2023, 2:30pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/3 "2023-04-01T14:30:17Z")

</div>

Can u please show an example of its use in my case? I could not figure out the dissect I tried something like this  
filter{  
if[message] =~"Category,"{  
dissect {  
mapping =\> {  
"message" =\> "%{Category} %{Subcategory} %{SubSubCategory}"  
}  
}  
csv {  
separator =\> ";"  
columns =\> ["Id","Name","Category"]  
}  
}  
output{  
elasticsearch {  
hosts =\> "[http://localhost:9200](http://localhost:9200)"  
index =\> "product"  
user =\> "elastic"  
password =\> "\*\*"  
}  
stdout{}  
}

But I must be doing something wrong... I'm new in ELK

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2023, 3:24pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/4 "2023-04-01T15:24:22Z")

</div>

`dissect` is literal you need to put in spaces and quotes

Data

```auto
ID|Name|Category
1, Sofa, "Furniture, Soft furnishings, TestString"
2, Desk Chair, "Furniture, Hard furnishings, OtheString"

```

Code

```auto
input {
  file { 
    path => "/Users/sbrown/workspace/sample-data/discuss/dissect/discuss.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null" 
  }
}

filter {
  dissect {
    mapping => {
      "message" => '%{product_id}, %{name}, "%{category}, %{sub_category}, %{alt_category}"'
      # 1, Sofa, "Furniture, Soft furnishings, TestString"
    }
  }
}

output { 
  stdout {} 
}

```

Output note it failed on the first line which is OK

```auto
{
         "event" => {
        "original" => "ID|Name|Category"
    },
          "host" => {
        "name" => "hyperion"
    },
      "@version" => "1",
          "tags" => [
        [0] "_dissectfailure"
    ],
       "message" => "ID|Name|Category",
           "log" => {
        "file" => {
            "path" => "/Users/sbrown/workspace/sample-data/discuss/dissect/discuss.csv"
        }
    },
    "@timestamp" => 2023-04-01T15:23:34.416033Z
}
{
            "host" => {
        "name" => "hyperion"
    },
    "sub_category" => "Soft furnishings",
             "log" => {
        "file" => {
            "path" => "/Users/sbrown/workspace/sample-data/discuss/dissect/discuss.csv"
        }
    },
            "name" => "Sofa",
      "product_id" => "1",
           "event" => {
        "original" => "1, Sofa, \"Furniture, Soft furnishings, TestString\""
    },
        "@version" => "1",
         "message" => "1, Sofa, \"Furniture, Soft furnishings, TestString\"",
        "category" => "Furniture",
      "@timestamp" => 2023-04-01T15:23:34.416294Z,
    "alt_category" => "TestString"
}

```

---

<div class="post-metadata">

**Author:** ![oleksiiorel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiiorel/32/118205_2.png) [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\
**Post date:** [April 1, 2023, 4:22pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/5 "2023-04-01T16:22:34Z")

</div>

Data

```auto
ID,Name,Category
1,Sofa,"Furniture,Soft furnishings,TestString"
2,Desk Chair,"Furniture,Hard furnishings,OtheString"
3,Desk Chair,"Furniture,Hard furnishings,OtheString"

```

```auto
      # 1, Sofa, "Furniture, Soft furnishings, TestString" - is this line for test?
I tried this code:
input{
  file {
	path => "E:/test.csv"  
    start_position => "beginning"
    sincedb_path => "NULL"
  }	  
}

filter {
  dissect {
    mapping => {
      "message" => '%{product_id}, %{name}, "%{category}, %{sub_category}, %{alt_category}"'
    }
  }
}
output{
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "test_products"
	user => "elastic"
    password => "**" 
  }
stdout{}
}

Then copied yours:
input{
  file {
	path => "E:/test.csv"  
    start_position => "beginning"
    sincedb_path => "NULL"
  }	  
}
filter {
  dissect {
    mapping => {
      "message" => '%{product_id}, %{name}, "%{category}, %{sub_category}, %{alt_category}"'
      # 1, Sofa, "Furniture, Soft furnishings, TestString"
    }
  }
}

output{
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "test_products2"
	user => "elastic"
    password => "*" 
  }
stdout{}
}

but the result is the same
 "_index": "test_products",
        "_id": "IcWTPYch3_",
        "_score": 1,
        "_source": {
          "host": {
            "name": "DESKTOP-5QFLCFM"
          },
          "message": """1,Sofa,"Furniture,Soft furnishings,TestString"
""",
          "event": {
            "original": """1,Sofa,"Furniture,Soft furnishings,TestString"
"""
          },
          "@version": "1",
          "@timestamp": "2023-04-01T16:07:58.384915200Z",
          "tags": [
            "_dissectfailure"
          ],
          "log": {
            "file": {
              "path": "C:/test.csv"
            }
          }
        }
      },
      {
        "_index": "test_products",
        "_id": "IsWTPYcBFY9QtlnpVh3_",
        "_score": 1,
        "_source": {
          "host": {
            "name": "DESKTOP-5QFLCFM"
          },
          "message": """2,Desk Chair,"Furniture,Hard furnishings,OtheString"
""",
          "event": {
            "original": """2,Desk Chair,"Furniture,Hard furnishings,OtheString"
"""
          },
          "@version": "1",
          "@timestamp": "2023-04-01T16:07:58.387915800Z",
          "tags": [
            "_dissectfailure"
          ],
          "log": {
            "file": {
              "path": "C:/test.csv"
            }
          }
        }
      }
    ]
 

```

[/quote]  
Some kind of magic, it works for you and not for me. Any idea what the reason?  
the main reason is naturally me=)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2023, 4:31pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/6 "2023-04-01T16:31:15Z")

</div>

`dissect` **is literal** as I said so if your data looks like below **with no spaces between** the commas...

```auto
ID,Name,Category
1,Sofa,"Furniture,Soft furnishings,TestString"
2,Desk Chair,"Furniture,Hard furnishings,OtheString"
3,Desk Chair,"Furniture,Hard furnishings,OtheString"

```

then your dissect should look like **with no spaces between** the commas...

```auto
"message" => '%{product_id},%{name},"%{category},%{sub_category},%{alt_category}"'

```

Which just worked fine for me..

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 1, 2023, 4:31pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/7 "2023-04-01T16:31:17Z")

</div>

The dissect pattern need to exactly match the data. Your sample data does not have spaces after the commas which the data Stephen is using does. Either remove the spaces from you dissect pattern or fix the input data.

---

<div class="post-metadata">

**Author:** ![oleksiiorel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiiorel/32/118205_2.png) [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\
**Post date:** [April 1, 2023, 4:57pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/8 "2023-04-01T16:57:14Z")

</div>

Mr. Stephen, Thank you for your forbearance! You made my day beautiful:)

---

<div class="post-metadata">

**Author:** ![oleksiiorel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiiorel/32/118205_2.png) [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\
**Post date:** [April 1, 2023, 4:57pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/9 "2023-04-01T16:57:52Z")

</div>

Thank u so much!!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 29, 2023, 4:58pm UTC](https://discuss.elastic.co/t/elastic-filter/329104/10 "2023-04-29T16:58:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.