Hi,
In filter part I have a parser like here:
kv {
field_split => ","
trim_value => "\""
value_split => "="
include_keys => ["Hostname","SlotId","EOCTimestamp","RequestStartTimestamp","ResponseStartTimestamp","AVRProfileName","VSName","POOLIP","POOLIPRouteDomain","POOLPort","URLString","ClientIP","ClientPort","MethodString","ResponseCode","GeoCode","ServerLatency","RequestSize","ResponseSize","RequestHeader","ResponseHeader","RequestPayload","RequestHeaderTruncated","ResponseHeaderTruncated","RequestPayloadTruncated","ResponsePayloadTruncated","MitigatedByDoSL7","RequestStartTimestampMicro","ResponseStartTimestampMicro","QualifiedForJSInjection","SessionId","BrowserName","OsName","ApplicationResponseTime","ClientTtfb","ClientSideNetworkLatency","ServerSideNetworkLatency","RequestDuration","ResponseDuration","ContentType","DeviceId","Referer","XffList","errdefs_msgno","Entity","AggrInterval","HitCount","VipName","ServerLatency","ServerLatencyHitCount","ClientConcurrentConns","ServerConcurrentConns","MaxClientConcurrentConns","MaxServerConcurrentConns","ClientNewConns","ServerNewConns","ServerNewConns","FailedConns","ExpiredConns","AbandonedConns","ClientBytesIn","ServerBytesOut","ServerBytesIn","ClientOutBytes","ClientPktsIn","ServerPktsOut","ServerPktsIn","ClientPktsOut","ConcurrentIps","ConcurrentBlockedIps","ConcurrentIpsParticipatingInAttacks","ConcurrentAttacks","ServerLatencyHealth","ConcurrentConnectionsHealth","ThroughputHealth","specialConcurrentIpsForAllVips"] }
I have about 300k-850k events in 15 minutes, when i use that parser cpu usage increases from 2 vcpu to 20-25vcpu and logstash loses some logs. Do you have any idea how I can improve the performance of this parser? change it to grok or use ruby?
Best,
Patryk