Logstash not indexed to Elasticsearch

[johntadd]

Searching the forum, but I don't find any threads that actually fixes my problem. So posting the question here.

The main problem is I don't find data processed by logstash gets indexed to elasticsearch.

curl 'localhost:9200/_cat/indices?v'
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana hIm_g8DDQPu9HC3aOAiZKQ   1   1          1            0      3.2kb          3.2kb

My setup flow is logstash read local gz files, indexing those data to elasticsearch. Config contents include:

input {
  file {
    type => "gzip"
    path => "/path/to/log/*.log.gz"
    start_position => beginning
    sincedb_path => "/dev/null"
    codec => "gzip_lines"
    max_open_files => 1000000
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I am sure elasticsearch is up because it responses message when posting with curl command curl localhost:9200

{
  "name" : "-6IWhCo",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "J6OXwQmcSUS6R9q0nRmEvg",
  "version" : {
    "number" : "5.5.0",
    "build_hash" : "260387d",
    "build_date" : "2017-06-30T23:16:05.735Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  }
  "tagline" : "You Know, for Search"
}

Also my ELK stack runs on the same EC2 server.

In addition, logstash does process local gz files but sometimes it complains observe_read_file: general error reading /path/to/....gz - error: java.lang.IllegalArgumentException: Object ¿´óxY^T<U+0A8D>.Áæ^^Àf{ú@^GESCi·Mé<U+0094># n, though more often it just prints Received line {:path =>"/path/to/....gz", :text=>"\xBF\xB4\xF3LY\x14\x8D...."} when debug mode is enabled (e.g. by adding --debug flag in /etc/systemd/system/logstash.service file)

There is no firewall rules that might block routing message between logstash and easticsearch, and I am able to telnet localhost 9200 as well

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

The only warning I found it might be weird is [logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x3aa8a5e>} but that seems to be normal as I notice other people's logstash log file (when googling or searching in the forum) also contains that line. And elasticsearch log also prints ][o.e.c.r.a.AllocationService] [-6IWhCo] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]). that looks up and running.

What other places I can check that might prevent elasticsearch from indexing data?

Thanks