While sending IIS logs from filebeat --> Logstash --> ElasticSearch --> Kibana , we lose couple of fields like (user_agent and geo location ) even after applying GROK pattern inside logstash FILTER.
Hence couple of visualization (access map , browser breakdown) in IIS default dashboard do not show any data.
We have to send IIS logs to logstash ( and not directly to ElasticSearch) because we also need to send application logs from filebeat to logstash from same server ( and we can send log to EITHER logstash or elastic search only)
So , do you know any way to pull useragent and geoip fields ?
The only other way we know is to configure 2 filebeat on client (application servers ).
Filebeat_IIS
It will send IIS logs to Elastic search ( and NOT through logstash )
Filebeat_applogs
It will send application logs to logstash (where we will apply grok pattern ) and then to Elastic search
We tried installing below plugins in logstash
bin\logstash-plugin install logstash-filter-useragent
bin\logstash-plugin install logstash-filter-geoip
But still it does not show DATA in following fields in kibana index pattern
source.geoip
source.geo.location
user_agent.name
user_agent.version
user_agent.os.name
user_agent.os.version
Please let me know if you have any solution
FIlebeat , Kibana , Logstash and Elastic Search ( all on v 7.5.1)