# Csv filter plugin not working
**URL:** https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777
**Category:** Logstash
**Created:** [March 30, 2021, 11:17am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777 "2021-03-30T11:17:34Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [March 30, 2021, 11:17am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/1 "2021-03-30T11:17:34Z")
I would like to use logstash to format my logs.
Specifically, I have a log with the following format
format:
timestamp [thread-name] log-level class-name - log-message
example:
2021-03-30 09:38:20.201 [ConnectionChecker] ERROR c.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]
I am hoping that I can use the filter plugin to get the 5 items separately, but that doesn't work.
If the problem can be solved with a method other than csv filter, that method is also welcome.
Here is the description and output of the filter I set in config.
```auto
... snip ...
filter {
csv {
separator => " "
skip_header => true
}
}
... snip ...
```
```auto
{
"host" => "ITS-ELS-01",
"column1" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
"@timestamp" => 2021-03-30T11:05:40.582Z,
"message" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
"@version" => "1"
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 30, 2021, 4:21pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/2 "2021-03-30T16:21:20Z")
I would suggest using a [dissect](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html) filter.
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [March 31, 2021, 1:12am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/3 "2021-03-31T01:12:08Z")
Thanks for the advice.
I read the dissect page and rewrote the filter plugin as follows.
```auto
filter {
dissect {
mapping => {
"message" => "%{timestamp} [%{thread}] %{loglevel} %{class-name} - %{logmessage}"
}
}
}
```
However, the result is as follows
```auto
{
"tags" => [
[0] "_dissectfailure"
],
"@version" => "1",
"@timestamp" => 2021-03-31T01:08:25.984Z,
"message" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
"host" => "ITS-ELS-01"
}
```
Can you please tell me more about this?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 31, 2021, 1:21am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/4 "2021-03-31T01:21:00Z")
If your message has tab separated fields then you need to use tab as the separator in the dissect.
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [March 31, 2021, 1:25am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/5 "2021-03-31T01:25:09Z")
Thanks for the advice.
I tried both the direct tab and the \t method, but the result was the same.
Is there anything else that could cause this?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 31, 2021, 2:36am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/6 "2021-03-31T02:36:55Z")
> [@its-ogawa](#):
>
> Is there anything else that could cause this?
Many things. I suggest you read [this](https://discuss.elastic.co/t/help-needed-in-grok/213827/2), which talks about grok, but the same applies to dissect. Build the pattern one field at a time.
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [March 31, 2021, 2:56am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/7 "2021-03-31T02:56:35Z")
> Build the pattern one field at a time.
> Hmmm. I'm not sure.
I tried it with spaces and got the following split.
```auto
dissect {
mapping => {
"message" => "%{a} %{b} %{c} %{d}"
}
}
```
```auto
{
"@timestamp" => 2021-03-31T02:49:55.503Z,
"host" => "ITS-ELS-01",
"c" => "-",
"d" => "SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
"@version" => "1",
"a" => "2021-03-30",
"message" => "2021-03-30 09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser - SendPing failed. roomHash:null,roomUserHashnull,error:org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, current state [CLOSING], expecting [OPEN or CONNECTED]",
"b" => "09:38:20.201\t[ConnectionChecker]\tERROR\tc.b.a.websocket.WebSocketOnlineUser"
}
```
To get the intended split, I want to split item "b" further with a tab. Do I use the actual notation " " for the tabs instead of \t?
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [March 31, 2021, 3:22am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/8 "2021-03-31T03:22:50Z")
I'd like to add a few things.
It doesn't seem to work when the delimiter is tab.
I've tried running both, but I can't get it to split properly.
```auto
mapping => {
"message" => "%{a} %{b} %{c} %{d}"
"b" => "%{be}\t%{bf}\t%{bg}\t%{bh}"
}
```
```auto
mapping => {
"message" => "%{a} %{b} %{c} %{d}"
"b" => "%{be} %{bf} %{bg} %{bh}"
}
```
Does dissect support tab as delimiter?
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [March 31, 2021, 3:56am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/9 "2021-03-31T03:56:27Z")
We have sorted out the problems.
- want to
I want to use logstash to format tab-delimited logs.
- issue
Tabbed input cannot be split.
- experiment
input:
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [March 31, 2021, 6:44am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/10 "2021-03-31T06:44:23Z")
Changing the `config.support_escapes` setting in `logstash.yml` from false to true did not change the result.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 31, 2021, 2:32pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/11 "2021-03-31T14:32:08Z")
dissect does support tabs
```
input { generator { count => 1 lines => ['a b'] } }
filter {
dissect { mapping => { "message" => "%{a} %{b}" } }
}
output { stdout { codec => rubydebug { metadata => false } } }
```
Use a literal tab in the dissect filter. That results in
```
"b" => "b",
"message" => "a\tb",
"a" => "a",
```
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [April 1, 2021, 2:07am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/12 "2021-04-01T02:07:45Z")
Does the item mapped by dissect (e.g. %{a}) get added to the field?
Do I need to use mutate's add\_field?
I'm not sure if I need to use the mutate add\_field or not, but the following description shows "%{a}" as it is, not the retrieved value.
I want to use dissect to display the value corresponding to %{a} in the field.
```auto
mutate {
add_field => { "a" => "%{a}" }
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 1, 2021, 2:13am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/13 "2021-04-01T02:13:50Z")
Perhaps I should have used different values
```
input { generator { count => 1 lines => ['x y'] } }
filter {
dissect { mapping => { "message" => "%{a} %{b}" } }
}
output { stdout { codec => rubydebug { metadata => false } } }
```
would produce
```
"b" => "y",
"message" => "x\ty",
"a" => "x",
```
If you use
`mutate { add_field => { "a" => "%{a}" } }`
then that is a no-op. It replaces the value of field [a] with the value of the field [a]
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [April 1, 2021, 2:24am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/14 "2021-04-01T02:24:26Z")
mutate does not work properly.
"%{a}" is still displayed as %{a}.
I want to output the value that applies to %{a}.
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [April 1, 2021, 2:41am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/15 "2021-04-01T02:41:10Z")
> Perhaps I should have used different values
I did the following.
```auto
dissect {
mapping => {
"message" => "%{ia} %{ib}"
}
}
mutate {
add_field => { "a" => "%{ia}" }
add_field => { "b" => "%{ib}" }
}
```
In debug, it displays nicely as follows
```auto
output { stdout { codec => rubydebug { metadata => false } } }
```
```auto
"a" => "a",
"b" => "b"
```
However, when I send it to elasticsearch, I get the same notation (e.g., "%{ia}").
Why is this?
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [April 2, 2021, 1:00am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/16 "2021-04-02T01:00:48Z")
In Kibana, it looks like this
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 2, 2021, 1:53am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/17 "2021-04-02T01:53:54Z")
Please do not post pictures of text. They are unreadable and unsearchable. I have no idea what your Kibana data looks like.
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [April 2, 2021, 1:58am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/18 "2021-04-02T01:58:05Z")
I'm sorry.
It is the same as the explanation above.
It means that the variables are displayed, but the values are not.
This is the way it is in Kibana I have seen.
I just wanted to show that.
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [April 2, 2021, 2:04am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/19 "2021-04-02T02:04:53Z")
Specifically, I want to say that the value of field "a" will not be displayed as "x", but as %{ia}, when set as follows.
```auto
input { generator { count => 1 lines => ['x y'] } }
filter {
dissect { mapping => { "message" => "%{a} %{b}" } }
add_field => { "a" => "%{ia}" }
}
output {
elasticsearch {
hosts => ["localhost"]
index => "other-%{+YYYY.MM.dd}"
}
}
```
---
### Author:  [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)
#### Post date: [April 6, 2021, 1:09am UTC](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777/20 "2021-04-06T01:09:07Z")
Does anyone have any good ideas?
[Next page](https://discuss.elastic.co/t/csv-filter-plugin-not-working/268777.md?page=2)