# Logstash / Filebeat and "|" sparated logs

**URL:** <https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123>\
**Category:** Logstash\
**Created:** [July 21, 2017, 1:57pm UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123 "2017-07-21T13:57:22Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![basti](https://avatars.discourse-cdn.com/v4/letter/b/fbc32d/32.png) [@basti](https://discuss.elastic.co/u/basti)\
**Post date:** [July 21, 2017, 1:57pm UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123/1 "2017-07-21T13:57:22Z")

</div>

Hello,

I just done my first steps with filebeat and logstash.  
I have some servers with log files with lines like:  
value1|value2|value3|.....

Now i want them to be send via (1)FileBeat to (2)Logstash and then to (3)Elasticsearch.

(1) working: Filebeat sends the files to Logstash.

(2) trying to parse the file...  
first I tryed filter "csv" but this doesnt work, because of some escaped " in content....  
which way would be the best to filter this lines?

(3)Some lines are filtered correct, but the were not send to ES.  
There I got no errors/logs or something like that ☹

My config in logstash:

```
input {
    beats {
            port => "5044"
            host => "0.0.0.0"
    }
}
filter {
            csv {
                    columns => ["shop ID", "timestamp", "offer ID",.....
                    separator => "|"
                    quote_char => "~"
            }
}
output {
            elasticsearch {
                    hosts => ["localhost:9200"]
                    index => "actionLog.%{+YYYY.MM.dd}"

            }
}

```

perhaps somebody here has some tips for me?  
thanks in advance!

---

<div class="post-metadata">

**Author:** ![elaair](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@elaair](https://discuss.elastic.co/u/elaair)\
**Post date:** [July 21, 2017, 6:58pm UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123/2 "2017-07-21T18:58:42Z")

</div>

I strip out certain " , ; etc before I send the data to logstash. I have found that hidden non UTF-8 characters can be removed like this(at least on Mac and Linux):  
iconv -f utf-8 -t utf-8 -c dirty.csv \> Clean.csv

I think you could also use gsub to replace characters.  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/gsub-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/gsub-processor.html)

---

<div class="post-metadata">

**Author:** ![basti](https://avatars.discourse-cdn.com/v4/letter/b/fbc32d/32.png) [@basti](https://discuss.elastic.co/u/basti)\
**Post date:** [July 21, 2017, 7:45pm UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123/3 "2017-07-21T19:45:45Z")

</div>

first thanks for your answer!!!

replacing is possible, but in my opinion it is not the solution.  
Isn`t there an other way to explode the separator "|"?  
CSV-Filter is not useful because it struggles with " in the file. Or is there a way to bypass this problem?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![jordansissel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordansissel/32/44957_2.png) [@jordansissel](https://discuss.elastic.co/u/jordansissel)\
**Post date:** [July 21, 2017, 8:21pm UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123/4 "2017-07-21T20:21:15Z")

</div>

CSV format itself supports quotes, so I think the csv filter probably isn't going to work well here.

Maybe you can use the `mutate` filter's `split` feature?

```auto
filter {
  mutate {
    split => { "message" => "|" }
  }
}

```

This will turn `message` field into a list of your original message split by `|`. You'll have to name your fields manually with another mutate:

```auto
filter {
  mutate { split ... }
  mutate {
    add_field => {
      "shop ID" => "[message][0]"
      "timestamp" => "[message][1]"
      # more columns here if you wish ...
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![basti](https://avatars.discourse-cdn.com/v4/letter/b/fbc32d/32.png) [@basti](https://discuss.elastic.co/u/basti)\
**Post date:** [July 24, 2017, 5:37am UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123/5 "2017-07-24T05:37:34Z")

</div>

Hi Jordan!  
Many thanks for your answer!  
This works except one Problem.  
The second mutate "add\_field" does not work as expected. It creates the field "shop ID" but in ES there is only "[meassage[0]]" as content and not the value from this field.

Other Question:  
In our file we have a timestamp in this format: 20170724073612 =\> YYYYMMDDHHIISS is it possible to convert this also in logstash, so we can use this field as timestamp in ES?

THANKS in advance!

---

<div class="post-metadata">

**Author:** ![basti](https://avatars.discourse-cdn.com/v4/letter/b/fbc32d/32.png) [@basti](https://discuss.elastic.co/u/basti)\
**Post date:** [July 24, 2017, 5:45am UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123/6 "2017-07-24T05:45:35Z")

</div>

Hi,

i found my first problem:  
I have to use it this way:  
mutate {  
add\_field =\> {  
"shop ID" =\> "%{[message][0]}"  
}  
}

now this works and I have only to format my timestamp.  
THANKS

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 21, 2017, 5:46am UTC](https://discuss.elastic.co/t/logstash-filebeat-and-sparated-logs/94123/7 "2017-08-21T05:46:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.