I am very new to this ELK ,
I have few questions ,
1) I am trying to visualize below log line kibana for that I used 4 components filebeat which is reading the log file from server and send data to logstash logstash filter the data and send the data to the elasticsearch then in happy case scenario I can see the real time data to the kibana visualization but sometime it happens that I couldnot see the real time data in kibana , I could not identify the reason but there is not error in log of filebeat,logstash and elastic search .
2) I am trying to extract only expected field by using logstash filter and send these field to the elasticsearch for visualization .
logstash config file
--------------------------------
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
beats {
port => 5044
type => "log"
host => "njsqlpaecom06"
}
}
filter {
if "QC update received:" in [message] {
grok {
match => { "message" => "InternalId=%{WORD:InternalId},.*E2ETime=%{INT:E2ETime},\s*CurrentTS=%{TIMESTAMP_ISO8601:currentTime}" }
}
mutate {
convert => {
"E2ETime" => "integer"
}
}
prune {
whitelist_names => ["^InternalId$","^E2ETime$","^CurrentTS$"]
}
}
if "_grokparsefailure" in [tags] { drop {} }
}
output {
stdout { codec => json }
elasticsearch {
hosts => "njsqlpaecom06:19660"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}
log line from log file
2021-09-30 07:19:48.205 DEBUG [ pool-hashed-thread-3] .e.s.d.p.QCDeltaPriceCachedUpdateHandler - BATS_17370113: QC update received: {Tier1_BidQty=0.0, Tier2_BidQty=0.0, ContainerSpread=46.725, PECorpBidSpread=0.48173475, OfferingContainerEnabled=Y, Tier1_StatusCode=0, Tier2_Info=Bid: Ask:RfqHistoryQuoteModifier: 0.0, PricingDist: MED_LIQ_SHORT_DUR_MED_RATING, PricingTier: Streaming, BaseWinProb: 25.0, baseDtm: 3.6776478268176565, mid:46.725, skewedDtm: 4.413177392181187, effectivePos: 1000000.0, targetPos: 0.0, Implied Output Probability: 17.655892079790437, Implied Output DTM: 4.413177392181187, added skew of 1.031, added multiplier of 2.0, Tier1_AskSpread=0.45966886303909404, Tier2_AskSpreadBps=43.760297607818806, Tier2_StatusCode=0, Tier1_Info=Bid: Ask:RfqHistoryQuoteModifier: 0.0, PricingDist: MED_LIQ_SHORT_DUR_MED_RATING, PricingTier: Streaming, BaseWinProb: 25.0, baseDtm: 3.6776478268176565, mid:46.725, skewedDtm: 2.2065886960905936, effectivePos: 1000000.0, targetPos: 0.0, Implied Output Probability: 43.394001973831564, Implied Output DTM: 2.2065886960905936, added skew of 1.031, Tier2_Active=Y, Tier1_Active=Y, Tiers=Tier1,Tier2,, InternalId=BATS_17370113, Tier1_AskSpreadBps=45.966886303909405, Tier1_AskQty=575000.0, PECorpAskSpread=0.48173475, Tier2_AskQty=575000.0, Tier1_ErrorMessage=, Tier2_AskSpread=0.43760297607818804, Tier2_ErrorMessage=, AlgoLive=Y, StrategyLastUpdatedTimestamp=2021-09-30T11:19:48.104Z, RbcBenchSecurityId=BATS_17584794, BookId=NCREALGO, ErrorMessage=, UseContainer=Y}, E2ETime=101, CurrentTS=2021-09-30T11:19:48.205Z
any help is appreciate , if any further information is required then please update via comment
