Error while testing pipeline

Elastic Stack Logstash
1

Hi,

When we run the following command to test the pipeline -

curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=response=200'

output is -

{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "logstash-$DATE",
"index_uuid" : "na",
"index" : "logstash-$DATE"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "logstash-$DATE",
"index_uuid" : "na",
"index" : "logstash-$DATE"
},
"status" : 404
}

Our goal is send filebeat output to logstash and logstash output to elasticsearch.

Please Note - Output of - "curl 'localhost:9200/_cat/indices?v'"

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open filebeat-6.2.2-2018.02.23 _kjNzUMkRia5nOmVKOW-Tw 3 1 3210 0 558.4kb 558.4kb

We have followed the guideline - https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html

2

You've single-quoted the URL in your command so the $DATE environment variable reference won't be expanded. If you instead double-quote the string the variable will be expanded.

I'm of course assuming that you're setting the DATE varaible to a reasonable value.

3

Below is the conf file "first-pipeline.conf" we have used here. The file is at path -- /etc/logstash/conf.d/first-pipeline.conf

The # character at the beginning of a line indicates a comment. Use

comments to describe your configuration.

input {
beats {
port => "5044"
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

In case we are doing anything wrong regarding setting the DATE varaible to a reasonable value, please guide.

As you suggested, we have executed the same command with double-quotes, but result was same. Below is the output.

[root@elkstack ~]# curl -XGET "localhost:9200/logstash-$DATE/_search?pretty&q=response=200"
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "logstash-",
"index_uuid" : "na",
"index" : "logstash-"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "logstash-",
"index_uuid" : "na",
"index" : "logstash-"
},
"status" : 404
}

4

So you're not setting the DATE variable in your shell? Then why do you expect a shell command that includes $DATE to work? I don't understand what you're trying to do.

5

DATE variable is defined, please find the below output -

[root@elkstack ~]# echo $DATE
2018.02.26

6

The most recent command output you posted indicates that the variable isn't defined. If you prepend the command with echo you'll see exactly what you would execute without actually executing it.

7

Happy to let you know that the issue is resolved. As we were using Filebeat and output is set to Logstash, the index is of the pattern "Filebeat-$DATE".
The index is now also accessible at Kibana :slight_smile:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.