Data Validation in Logstash

Elastic Stack Logstash
1

Hi,
I am trying data validation in logstash.
Example : column 1 in CSV should only allow number (double data type).
No other text should be allowed if any other string found then trigger mail.
Similarly, need date validation. if date is not in required format trigger mail and do not process the CSV file until correct data isn't placed.

How to handle this in Logstash.

Please help me with this.

Thanks in advance

2

You can use a date filter to check whether the date is in the required format. If it fails to parse the date it will add a _dateparsefailure tag to the event.

If you want to check whether a field is a number you could use something like

if [someField] !~ /^[\d\.]+$/ { mutate { add_tag => [ "_numberformatfailure" ] } }

(depending on your number format you may need to adjust the pattern).

Then in the output section route the event based on the tags...

if "_dateparsefailure" in [tags] or "_numberformatfailure" in [tags] {
    mail { ... }
} else {
    some other destination
}

Note that this is per-event, not per-file. Validating all the rows in a file before processing it can probably be done, but it is not a use case that logstash is well adapted to.

3

Thank you so much . I'll check this

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.