Kibana is not showing the result immediately

[shubhrant (Shubhrant Chauhan)]

Its a general question ..

Actually everything is working fine, log sending and receiving between filebeat and logstash.

But when filebeat is sending apache logs then it is reflecting on dashboard after 10 or 15 min
or sometime it takes 1 hr to reflect , I don't know why it is taking so much of time for apache logs only.

Beacause from the same machine syslog is reflecting normally ...

I am also using packetbeat in the same machine so HTTP log mysql logs all are reflecting normally, actually it is a apache client server that's why I've configured the packetbeat on it

1 more thing I want to add that I did not configure the elastic cluster I simply run a server of 15GB ram with xeon processor . I've configured the ELK stack in the same machine.

anybody have any idea about it ..

[Joe_Fleming]

@shubhrant

But when filebeat is sending apache logs then it is reflecting on dashboard after 10 or 15 minor sometime it takes 1 hr to reflect , I don't know why it is taking so much of time for apache logs only.

That's strange. Elasticsearch is an eventually-consistent system, so writes take a little bit before they show up, but that should be measures in seconds, not several minutes. I wonder if you are seeing some side-effects of time bucketing... can you tell me more about how you're trying to view the data you're ingesting in Kibana?

[shubhrant (Shubhrant Chauhan)]

Sorry for the late reply ,
actually i was not in office so can not reply...

this is my logstash config

input {
beats {
port => 5044
congestion_threshold => "40"
}
}

filter {
if [type] == "apache" {

grok {
  match => { "message" => "%{COMBINEDAPACHELOG}" }
}

date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}

geoip {
source => "clientip"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}

mutate {
  convert => [ "[geoip][coordinates]", "float"]
}

useragent {
source => "agent"
target => "useragent"
}

}
}

output {

if [type] == "syslog" {
elasticsearch {
hosts => ["localhost:9200"]
index => "syslog-%{+YYYY.MM.dd}"
}
}
else if [type] == "apache" {
elasticsearch {
hosts => ["localhost:9200"]
index => "apache12-%{+YYYY.MM.dd}"
}
}

else {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

}