Hello elastic team!
Good day to you.
May I ask about logstash index from CWL and inject to ES
I have application logs stored in Amazon CloudWatch Logs.
Eg: Under Test-Log-Group, App1-Log-Stream, App2-Log-Stream, App3-Log-Stream, App4-Log-Stream, App5-Log-Stream
These log streams will get the logs continously.
I'm trying to use Logstash to index from AWS CloudWatch Logs and format some logs in logstash pipeline and only ship the formatted logs to AWS ElasticSearch Domain.
These are used plugins >> logstash-input-cloudwatch-logs input plugin, do grokking the logs and logstash-output-amazon_es output plugin.
The problem is that logs are missing in ES Domain when I do monitor in kibana ui.
Eg: index 500-logs from cloudwatch log, grokking 50 logs which is necessary to store in ES domain.
And ship that 50 logs to ES.
At that time, sometime I see my grokked logs, sometimes I don’t see my some grokked log.
Please advice to me which part makes cause this problem input plugin? output plugin? (I’ve checked my grokked pattern in grok debugger. They’re ok.)
Does cloudwatch_logs/.sincedb* support mulitple log streams?
And may I know recommended logstash input plugin of AWS Cloudwatch Logs and logstash output plugin of AWS ElasticSearch.
I can't seem to find another plugins to accomplish this so far.
Thank you.
Can someone please help on this?????