Not seeing cloudwatch streamed logs via functionbeat in elasticsearch

I am fairly new to Elastic stack and I am trying use functionbeat to stream logs to logstash and then to elasticsearch. I tested my streaming with the following logstash config:

    input {
      beats {
        port => 5044
      }
    }
    output {
      file {
         path => "/tmp/cloudwatch_out.text"
         codec => rubydebug
         create_if_deleted => true
      }

When I tail that file I do see logs streaming into it.

But when I changed the logstash config to go to elasticsearch

   input {
      beats {
        port => 5044
      }
    }
    output {
      elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      }
    }

I created an index and an index pattern
I then try to run this in the console

    GET functionbeat-7.8.0/_search
    {
      "query": {
        "match_all": {}
      }
    }

A few documents are returned but it doesn't seem to be all the documents and I never see any new documents.
I also did

curl -X GET http://localhost:9200/functionbeat-7.8.0/_search?pretty=true

With same results. Most of the instructions to set this up seem pretty simple but I suspect I am missing something basic


Basically I'm asking how I can see or visualize these logs

I figured it out, I need this in the curl command

-H 'Content-Type: application/json'

Full curl command looks like this:

curl -XGET -H 'Content-Type: application/json' "http://localhost:9200/functionbeat-7.8.0/_search?pretty=true&scroll=10m&size=200" -d'
  {
     "query": {
       "match_all": {}
     }
  }'

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.