Beginner question: Verifying that a file has successfully been entered in ES

Elastic Stack Logstash
1

Greetings Logstash Community,

I am very new to the Logstash/ElasticSearch application and I have a very basic question:

I have the following configuration file where my goal is to take a log file and insert it in ElasticSearch (for now, without indexing any data).

input {
file }
path => "C:\Users\Admin\Desktop\Logs_elasticsearch\logfile.txt"
type => "GVP"
}
}

#filter {

grok {

}

#}

output {
elasticsearch {
host => localhost
}
stdout {codec => rubydebug}
}

After I load the configuration file using the /bin/logstash -f logstash.conf command, how can I validate that the file has successsfully been entered in ElasticSearch?

Thanks for the explanation,

Regards,

2

If you intend to use Kibana later on (probably) you might as well install and configure it now and use it for verification. Otherwise Elasticsearch's search API is easy to use. You might also find the cat count API useful if you want to check how many documents are indexed and make sure that show correlate to the number of lines in the file.

3

Thanks for your reply Magnus.

I ran the command http://localhost:9200/_search/search_type=count. However, I conclude that only the individual entries were entered in ES, and not the log file that I tried to enter while running the conf file. Any feedback?

The info I have in the conf file:

input {
file {
path => "C:\Users\Admin\Desktop\Logs_elasticsearch\logfile.txt"
type => "GVP"
}
}

output {
elasticsearch {
host => localhost
}
stdout {codec => rubydebug}
}

I appreciate your assistance.

4

"Individual entries"? I don't understand.

One thing to keep in mind is that all outputs emit messages in lockstep, so if the stdout output is writing data to stdout then that means that the elasticsearch output is also successfully sending data. (well, okay, I think at least with Logstash <1.5 ES could actually reject messages without Logstash taking much notice).

5

When I refer to individual entries, I mean entering data in ES using the PUT command. However, I am not able to input complete log files within ES.

I do get the following error whenever I start LS on windows 7 64 bit: "IO/console not supported; tty will not be manipulated". This error happens both when I don't define a Config file and when I specify a Config file. Could this be related?

6

When I refer to individual entries, I mean entering data in ES using the PUT command. However, I am not able to input complete log files within ES.

Okay. Turning up Logstash's logging with --verbose or even --debug will provide clues. Again, if it has problems posting to ES it will tell you in the log.

I do get the following error whenever I start LS on windows 7 64 bit: "IO/console not supported; tty will not be manipulated". This error happens both when I don't define a Config file and when I specify a Config file. Could this be related?

Unrelated, ignore. See Warning when running any shell commands · Issue #3087 · elastic/logstash · GitHub.