I could not see the real Time data on Kibana

Elastic Stack Elasticsearch
1 
I am very new to this ELK ,
I have few questions ,
1) I am trying to visualize below log line kibana for that I used 4 components filebeat which is reading the log file from server and send data to logstash logstash filter the data and send the data to the elasticsearch then in happy case scenario I can see the real time data to the kibana visualization but sometime it happens that I couldnot see the real time data in kibana , I could not identify the reason but there is not error in log of filebeat,logstash and elastic search . 
2) I am trying to extract only expected field by using logstash filter and send these field to the elasticsearch for visualization .
logstash config file 
--------------------------------
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {

 beats {
    port => 5044
    type => "log"
    host => "njsqlpaecom06"
  }
 }

 filter {
  
if "QC update received:" in [message] {
grok {
 match => { "message" => "InternalId=%{WORD:InternalId},.*E2ETime=%{INT:E2ETime},\s*CurrentTS=%{TIMESTAMP_ISO8601:currentTime}" }
}


mutate {
	    convert => {
	        "E2ETime" => "integer"
}
}
prune {
        whitelist_names => ["^InternalId$","^E2ETime$","^CurrentTS$"]
      }

  }

if "_grokparsefailure" in [tags] { drop {} }
}

output {

 stdout { codec => json }
 elasticsearch {
    hosts => "njsqlpaecom06:19660"    
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  } 
} 


log line from log file 
2021-09-30 07:19:48.205 DEBUG [     pool-hashed-thread-3] .e.s.d.p.QCDeltaPriceCachedUpdateHandler - BATS_17370113: QC update received: {Tier1_BidQty=0.0, Tier2_BidQty=0.0, ContainerSpread=46.725, PECorpBidSpread=0.48173475, OfferingContainerEnabled=Y, Tier1_StatusCode=0, Tier2_Info=Bid:  Ask:RfqHistoryQuoteModifier: 0.0, PricingDist: MED_LIQ_SHORT_DUR_MED_RATING, PricingTier: Streaming, BaseWinProb: 25.0, baseDtm: 3.6776478268176565, mid:46.725, skewedDtm: 4.413177392181187, effectivePos: 1000000.0, targetPos: 0.0, Implied Output Probability: 17.655892079790437, Implied Output DTM: 4.413177392181187, added skew of 1.031, added multiplier of 2.0, Tier1_AskSpread=0.45966886303909404, Tier2_AskSpreadBps=43.760297607818806, Tier2_StatusCode=0, Tier1_Info=Bid:  Ask:RfqHistoryQuoteModifier: 0.0, PricingDist: MED_LIQ_SHORT_DUR_MED_RATING, PricingTier: Streaming, BaseWinProb: 25.0, baseDtm: 3.6776478268176565, mid:46.725, skewedDtm: 2.2065886960905936, effectivePos: 1000000.0, targetPos: 0.0, Implied Output Probability: 43.394001973831564, Implied Output DTM: 2.2065886960905936, added skew of 1.031, Tier2_Active=Y, Tier1_Active=Y, Tiers=Tier1,Tier2,, InternalId=BATS_17370113, Tier1_AskSpreadBps=45.966886303909405, Tier1_AskQty=575000.0, PECorpAskSpread=0.48173475, Tier2_AskQty=575000.0, Tier1_ErrorMessage=, Tier2_AskSpread=0.43760297607818804, Tier2_ErrorMessage=, AlgoLive=Y, StrategyLastUpdatedTimestamp=2021-09-30T11:19:48.104Z, RbcBenchSecurityId=BATS_17584794, BookId=NCREALGO, ErrorMessage=, UseContainer=Y}, E2ETime=101, CurrentTS=2021-09-30T11:19:48.205Z

any help is appreciate , if any further information is required then please update via comment

2

Welcome to our community! :smiley:

Please edit your post and remove the code formatting from the text portion of your post, it's very hard to follow what you are asking with the way it's currently formatted :slight_smile:

3

You are dropping anything that results in a failure which means you could be losing logs that way. To troubleshoot I would recommend writing this data somewhere, e.g. a separate index or a file, so you can analyse what is being dropped and correct any issues.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.