Rename multiple fields in Logstash?

Elastic Stack Logstash
1

Hi All,

I'm using Logstash 2.3.0 to receive and process Netflow data. Average indexing is about 50K events per minute, but as I put the mutate block to rename fields, throughput drops to 15K per minute.

## Rename all netflow fields
    mutate { 
        rename => { "IN_BYTES" => "tx-bytes" } 
        rename => { "IN_PKTS" => "tx-pkts" }
        rename => { "IN_SRC_MAC" => "src-mac" }
        rename => { "IPV4_DST_ADDR" => "dst-ip" }
        rename => { "IPV4_SRC_ADDR" => "src-ip" }
        rename => { "L4_DST_PORT" => "dst-port" }
        rename => { "L4_SRC_PORT" => "src-port" }
        rename => { "L7_PROTO_NAME" => "protocol"}
        rename => { "OUT_BYTES" => "rx-bytes" }
        rename => { "OUT_DST_MAC" => "dst-mac" }
        rename => { "OUT_PKTS" => "rx-pkts" }
        rename => { "PROTOCOL_MAP" => "protocol-map" }
        rename => { "TOTAL_FLOWS_EXP" => "total-flows-exp" }
    }

Above is the mutate block. Logstash is running on a box with 2x 12 core Xeon X5650 2.67GHz and 192GB of ram with lots of free CPU and memory. Logstash starts with 24 threads.

Is there a better way the rename multiple fields in Logstash and does not hurt throughput because I cannot change field names from the Netflow source?

Thanks,

1 Like
2

If there's CPU headroom you should be able to increase the number of pipeline workers.

3

For 2x 12 core CPU with HT I should have 48 cores, is it possible to increase to 48 threads without hurting server performance? I have ES nodes run on the same server.

4

I don't think there's a more efficient way of renaming the fields, so it all boils down to CPU horsepower. If Logstash uses more CPU then there will obviously be less of it for ES, but again, if you have lots of spare CPU cycles you should be able to let Logstash use more of it without sacrificing the ES performance.

5

Thanks, I currently have to run two separate Logstash instances to handle two sources of Netflow for a total of 100K event per minute, which is not a large amount of events. I guess it's just the mutate filter that hurts Logstash throughput. I used to have some if statements in the filter section but it was even worse that I had to remove them.

Just increased workers to 32 and I'll see how it goes.