Logstash / Filebeat and "|" sparated logs

[basti]

Hello,

I just done my first steps with filebeat and logstash.
I have some servers with log files with lines like:
value1|value2|value3|.....

Now i want them to be send via (1)FileBeat to (2)Logstash and then to (3)Elasticsearch.

(1) working: Filebeat sends the files to Logstash.

(2) trying to parse the file...
first I tryed filter "csv" but this doesnt work, because of some escaped " in content....
which way would be the best to filter this lines?

(3)Some lines are filtered correct, but the were not send to ES.
There I got no errors/logs or something like that

My config in logstash:

input {
    beats {
            port => "5044"
            host => "0.0.0.0"
    }
}
filter {
            csv {
                    columns => ["shop ID", "timestamp", "offer ID",.....
                    separator => "|"
                    quote_char => "~"
            }
}
output {
            elasticsearch {
                    hosts => ["localhost:9200"]
                    index => "actionLog.%{+YYYY.MM.dd}"

            }
}

perhaps somebody here has some tips for me?
thanks in advance!

[elaair (Will)]

I strip out certain " , ; etc before I send the data to logstash. I have found that hidden non UTF-8 characters can be removed like this(at least on Mac and Linux):
iconv -f utf-8 -t utf-8 -c dirty.csv > Clean.csv

I think you could also use gsub to replace characters.
https://www.elastic.co/guide/en/elasticsearch/reference/current/gsub-processor.html

[basti]

first thanks for your answer!!!

replacing is possible, but in my opinion it is not the solution.
Isn`t there an other way to explode the separator "|"?
CSV-Filter is not useful because it struggles with " in the file. Or is there a way to bypass this problem?

Thanks in advance!

[jordansissel]

CSV format itself supports quotes, so I think the csv filter probably isn't going to work well here.

Maybe you can use the mutate filter's split feature?

filter {
  mutate {
    split => { "message" => "|" }
  }
}

This will turn message field into a list of your original message split by |. You'll have to name your fields manually with another mutate:

filter {
  mutate { split ...  }
  mutate {
    add_field => {
      "shop ID" => "[message][0]"
      "timestamp" => "[message][1]"
      # more columns here if you wish ...
    }
  }
}

[basti]

Hi Jordan!
Many thanks for your answer!
This works except one Problem.
The second mutate "add_field" does not work as expected. It creates the field "shop ID" but in ES there is only "[meassage[0]]" as content and not the value from this field.

Other Question:
In our file we have a timestamp in this format: 20170724073612 => YYYYMMDDHHIISS is it possible to convert this also in logstash, so we can use this field as timestamp in ES?

THANKS in advance!

[basti]

Hi,

i found my first problem:
I have to use it this way:
mutate {
add_field => {
"shop ID" => "%{[message][0]}"
}
}

now this works and I have only to format my timestamp.
THANKS