# Filtering csv headers with different lengths

**URL:** <https://discuss.elastic.co/t/filtering-csv-headers-with-different-lengths/231526>\
**Category:** Logstash\
**Created:** [May 7, 2020, 11:47am UTC](https://discuss.elastic.co/t/filtering-csv-headers-with-different-lengths/231526 "2020-05-07T11:47:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![eeijlar](https://avatars.discourse-cdn.com/v4/letter/e/96bed5/32.png) [@eeijlar](https://discuss.elastic.co/u/eeijlar)\
**Post date:** [May 7, 2020, 11:47am UTC](https://discuss.elastic.co/t/filtering-csv-headers-with-different-lengths/231526/1 "2020-05-07T11:47:21Z")

</div>

I have two slightly different csv headers i would like to catch under a filter:

```auto
ACTION,DISCRIMINATOR,CHECKPOINT_ID,INSTALLATION_ID,NUMERIC_VALUE,STRING_VALUE,COMPONENT_NAME,TRANSACTION_ID,STATISTIC_DURATION,STATISTIC_NAME,ASSOCIATED_NAME,SINGLE_LATENCY,AVERAGE_LATENCY,MAX_LATENCY,MIN_LATENCY,TPS,MAX_LATENCY_TRANSACTION_ID,CREATED_ON_DATE,TOTAL_TRANSACTIONS,PROCESS_NAME,SLA1,SLA2,EXCEPTION_COUNT,CATEGORY,P95,P99,P999,P9999,P99999

```

and

```auto
ACTION,DISCRIMINATOR,CHECKPOINT_ID,INSTALLATION_ID,NUMERIC_VALUE,STRING_VALUE,COMPONENT_NAME,TRANSACTION_ID,STATISTIC_DURATION,STATISTIC_NAME,ASSOCIATED_NAME,SINGLE_LATENCY,AVERAGE_LATENCY,MAX_LATENCY,MIN_LATENCY,TPS,MAX_LATENCY_TRANSACTION_ID,CREATED_ON_DATE,TOTAL_TRANSACTIONS,PROCESS_NAME,SLA1,SLA2,EXCEPTION_COUNT,CATEGORY

```

To do this I have the following filter:

```auto
filter {
    if [fields][target-index] == "openet-fw-stats" {
     grok {
            match => { "message" =>
"%{WORD:ACTION},%{WORD:DISCRIMINATOR},%{DATA:CHECKPOINT_ID},%{DATA:INSTALLATION_ID},%{INT:NUMERIC_VALUE},%{DATA:STRING_VALUE},%{DATA:COMPONENT_NAME},%{DATA:TRANSACTION_ID},%{INT:STATISTIC_DURATION},%{DATA:STATISTIC_NAME},%{DATA:ASSOCIATED_NAME},%{INT:SINGLE_LATENCY},%{INT:AVERAGE_LATENCY},%{INT:MAX_LATENCY},%{INT:MIN_LATENCY},%{INT:TPS},%{DATA:MAX_LATENCY_TRANSACTION_ID},%{INT:CREATED_ON_DATE},%{INT:TOTAL_TRANSACTIONS},%{DATA:PROCESS_NAME},%{INT:SLA1},%{INT:SLA2},%{INT:EXCEPTION_COUNT},%{WORD:CATEGORY},%{INT:P95},%{INT:P99},%{INT:P999},%{INT:P9999},%{INT:P99999}"
}
     }
     # Support older FW stats format
     grok {
            match => { "message" =>
"%{WORD:ACTION},%{WORD:DISCRIMINATOR},%{DATA:CHECKPOINT_ID},%{DATA:INSTALLATION_ID},%{INT:NUMERIC_VALUE},%{DATA:STRING_VALUE},%{DATA:COMPONENT_NAME},%{DATA:TRANSACTION_ID},%{INT:STATISTIC_DURATION},%{DATA:STATISTIC_NAME},%{DATA:ASSOCIATED_NAME},%{INT:SINGLE_LATENCY},%{INT:AVERAGE_LATENCY},%{INT:MAX_LATENCY},%{INT:MIN_LATENCY},%{INT:TPS},%{DATA:MAX_LATENCY_TRANSACTION_ID},%{INT:CREATED_ON_DATE},%{INT:TOTAL_TRANSACTIONS},%{DATA:PROCESS_NAME},%{INT:SLA1},%{INT:SLA2},%{INT:EXCEPTION_COUNT},%{WORD:CATEGORY}"
}
     }
        if "_grokparsefailure" in [tags] {
            drop {}
        } else {
                ruby {
                code => "event.set('FW_DATE',Time.at(event.get('CREATED_ON_DATE')[0..9].to_i).strftime('%Y.%m.%d'))"
                }
                if "_rubyexception" in [tags] {
                drop {}
            } else {
                mutate {
                    remove_field => ["ACTION","DISCRIMINATOR","CHECKPOINT_ID","STRING_VALUE","P95","P99","P999","P9999","P99999"]
                    gsub => [
                        "ASSOCIATED_NAME", "\"", "",
                        "COMPONENT_NAME", "\"", "",
                        "STATISTIC_NAME", "\"", "",
                        "INSTALLATION_ID", "\"", "",
                        "PROCESS_NAME", "[\",]", "",
                        "PROCESS_NAME", "^[0-9]*", ""
                    ]
                    convert => {
                        "CREATED_ON_DATE" => "integer"
                        "NUMERIC_VALUE" => "integer"
                        "STATISTIC_DURATION" => "integer"
                        "SINGLE_LATENCY" => "integer"
                        "AVERAGE_LATENCY" => "integer"
                        "MAX_LATENCY" => "integer"
                        "MIN_LATENCY" => "integer"
                        "TPS" => "integer"
                        "TOTAL_TRANSACTIONS" => "integer"
                        "SLA1" => "integer"
                        "SLA2" => "integer"
                        "EXCEPTION_COUNT" => "integer"
                    }
                    replace => {
                        "INSTALLATION_ID" => "%{[host][name]}"
                    }
                }
                date {
                    match => ["CREATED_ON_DATE", "UNIX_MS"]
                    target => "@timestamp"
                }
            }
        }
    }
}

```

I am wondering if this is the right way to do it? In the remove\_field section it is removing fields which won't exist in the second grok filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 4, 2020, 11:47am UTC](https://discuss.elastic.co/t/filtering-csv-headers-with-different-lengths/231526/2 "2020-06-04T11:47:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.