Index not present in new ELK instance

[lynnptilby (Lynn)]

As root
curl -s 'http://localhost:9200/_cat/indices?v'
Gives:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_1 vmG8X7ucTPWOTDn-YQxaow 1 0 3 0 11.9kb 11.9kb

My /etc/logstash/logstash.conf file is:

input {
file {
"path" => "/s3/my_bucket/misc_info/2018-12-06/HWM/*"
start_position => "beginning"
}
}

filter {
grok{
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}" ]
}

date{
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSSZ" ]
target => "timestamp"
}
}

output {
elasticsearch {
"hosts" => "127.0.0.1"
"index" => "time2_hwm"
"id" => "time2_hwm"
}
}

According to what I have read, time2_hwm should show up in the above curl output. Are there owner/role restrictions on indexes? Note, that I am restarting logstash with 'systemctl restart logstash' with each change of my /etc/logstash/logstash.conf file.

Does what user logstash is run as, make a difference in the access to the index that is created?

What user should I be when doing the above curl command? root elasticsearch kibana or any user?

Where do I go to change the user/role/ownership/permissions in Kibana?

I have not seen any errors in /var/log/syslog indicating problems, but no index.

THANKS for your help!

[elasticforme (Sachin)]

please try to run this on command line and you will see what kind of result you are getting.
Also post sample of what is in yhour file.

/usr/share/logstash/bin/logstash -f <config_file_name>

Also put this in your output section to dump output on standard output

stdout { codec => rubydebug }

you are also missing port on your hosts
hosts => ["localhost:9200"]

you don't have to have " on hosts, index, id

I can also see that your index name is same as id. but actually it is document_id

here is sample file that I use for testing.

list_sample_test file will have only two column
A1 B1

input {
file {
path => "/tmp/list_sample_test"
start_position => "beginning"
close_older => 2
discover_interval => 3
sincedb_path => "/dev/null"
}
}

filter {
csv {
separator => " "
columns => [ "col1", "col2" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "test"
document_id => "%{col1}"
}
stdout { codec => rubydebug }
}