ES 5.0 new installation question


#1

After getting less-than-desired results trying to upgrade my ELK stack to version 5.0, I thought I'd start fresh, and create a brand new ELK 5.0 node. I have reused some config files (e.g. elasticsearch.yml, kibana.yml, logstash config files) from my older setup. So far, so good. ES, Logstash, and Kibana start OK, and I can access the Kibana Web UI. However, I can't configure an index pattern. For example, the pattern logstash-* returns the dreaded "Unable to fetch mapping. ... ". When I look in /var/lib/elasticsearch/nodes/0/indices, I see idices with names like Lk8wl5UtRASVyldRHspZQg, instead of e.g. logstash-2016.11.08. When I go to http://localhost:9200/_cat/indices, I see the following:

yellow open .monitoring-data-2 s147bGxoQ5WgDgsP4XlECA 1 1 4 0 13.5kb 13.5kb
yellow open .monitoring-es-2-2016.11.08 PcZAvTv8TjmTJgZ6e-rm0Q 1 1 2900 40 2.4mb 2.4mb
yellow open .monitoring-kibana-2-2016.11.08 m1MyHbtXRCiNPdeDr97aZA 1 1 550 0 335kb 335kb
yellow open .kibana Lk8wl5UtRASVyldRHspZQg 1 1 1 0 3.2kb 3.2kb

What happened to my logstash, filebeat, winlogbeat, and topbeat indices?

Thanks


(Mark Walkom) #2

Did you push data from beats etc into ES? That might sound obvious, but it doesn't look like you have.


#3

warkolm, I take it you mean from all of the hosts that are forwarding logs? I would think so. I'm using logstash as output. Here's my output configuration (30-elasticsearch-output.conf):

output {
if [@metadata][beat] {
elasticsearch {
hosts => ["http://10.0.101.101:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else {
elasticsearch { hosts => ["10.0.101.101:9200"] }
stdout { codec => rubydebug }
}
}


(Mark Walkom) #4

And do you know that data is reaching the output? Do things show if you use a stdout?


#5

Actually, I don't know. If I've followed installation instructions to-the-letter, but used various config files from the old installation, maybe that's where the problem lies. I've posted the various configs on Pastebin (http://pastebin.com/dcEbc32k). I would most appreciate it if someone could take a look. That way, either a misconfig will be uncovered, or it'll rule out that that's where the problem lies.

With thanks.


(Mark Walkom) #6

You should add a stdout to check :slight_smile:


#7

OK. But, er (shuffling feet, looking sheepish), how do I do that?


(Mark Walkom) #8

It's totally ok to ask that :smiley:

output {
  stdout {
    codec => rubydebug
  }
}

#9

Thanks so much. I'm taking it I add this to my logstash output config? And I should access, or see, output where?


(Mark Walkom) #10

Yeah add it to your config, it'll show the output of your pipeline to the console, aka stdout.


#11

OK. Once added, and services restarted, am I supposed to see output flowing to stdout (I don't, as of now), or must I do something else?


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.