How to get a function beat (configured to listen to kinesis stream) to push data into Elasticsearch

[ryantomaselli]

Hi everyone...

I'm wet behind the ears on the Elasticsearch product so bear with me.

My goal is to stream data from AWS DynamoDB (via a AWS Kinesis stream) to my hosted Elasticsearch instance. I'm using the functionbeat to do this.

It would seem that everything is working fine for the part where the functionbeat lambda is listening on changes to the stream (so when I make changes to the database I see log churning out in Cloudwatch log for the lambda).

Looking at the log messages it would seem that there are no error so I'm expecting to be able to go the Discover part of the Elasticsearch web console and see some data...but see nada.

I'm not clear on what more I need to do to get data into the Elasticsearch instance.

I've played with some of the processors (namely include_fields and decode_json_fields) and they work.

Any pointers appreciated.

Thanks!

[ryantomaselli]

Found my answer (turns out the data is coming into Elasticsearch I just needed to do the following to get a look at it)

Go into Stack Management >Index patterns and add a pattern (for the function beat data coming through. I added a pattern of "functionbeat*".

I then went to Kibana > Discovery and selected the "functionbeat*" index pattern created in the previous step...the screen reloads and I see all the data that was ingested.