Document not created

Elastic Stack Logstash
1

I'm newbie in elasticsearch.
I'm running logstash with command 'bin\logstash --verbose -f logstash,conf'.

Logstash.conf is:
#input { stdin { } }

input {
file {
path => "C:\Users\pawnow\logstash\logstash_input.log"
start_position => "beginning"
}
}

#May 24 10:20:15 User1 CREATE "Create a new folder"
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:user} %{WORD:action_performed} %{QUOTEDSTRING:action_description}"}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "cs_users"
}
stdout { codec => rubydebug }
}

New index is being created in elasticsearch, but there is no document inside the index. My log file is like:
May 24 10:20:15 User1 CREATE "Create a new folder"
May 24 10:20:15 User2 CREATE "Create a new folder"

So i assume there should be 2 documents inside index having properties timestamp user action action_description like is defined in grok first with 'User1' and second with 'User2'.
Question 1) Why index is empty?

In logstash console logstash logs:
{
"timestamp" => "May 24 10:20:15",
"host" => "DVIG1673",
"user" => "User2",
"@version" => "1",
"message" => "May 24 10:20:15 User1 CREATE "Create a new folder"\r",
"path" => "C:\Users\pawnow\logstash\logstash_input.log",
"action_description" => ""Create a new folder"",
"tags" => [
[0] "_dateparsefailure"
],
"action_performed" => "CREATE",
"@timestamp" => 2018-06-25T13:09:02.288Z
}
Question 2) Where is the log for 'User 2'
Question 3) Every time I start logstash I have to delete .sincedb to get it working

2

Question 1) Why index is empty?

How did you verify that it's empty? Have you tried bumping up Logstash's loglevel to get more information about what it's doing?

Question 2) Where is the log for 'User 2'

What if you comment out the elasticsearch output, do you get both log entries to your terminal?

Question 3) Every time I start logstash I have to delete .sincedb to get it working

This is expected. Logstash's default behavior is to continuously monitor log files and avoid reprocessing old data.

3

Thanks for reply Magnus

Question 1
You are right. After additional search I can see that my index is not empty, but...
After indexing I tried to get document with 'GET /index_name/type_name/1?pretty' command which was incorrect, but why?
Using URL 'http://localhost:9200/cu_users/_search?pretty=true&q=:' I can see all indexed documents, so what is command to get single document.

Question 2
I still have a problem. Terminal log is:
{
"action_performed" => "CREATE",
"user" => "User1",
"path" => "C:\Users\pawnow\logstash\mysample-input.log",
"@timestamp" => 2018-06-26T10:56:06.470Z,
"@version" => "1",
"action_description" => ""Create a new folder"",
"host" => "DVIG1673",
"tags" => [
[0] "_dateparsefailure"
],
"timestamp" => "May 01 01:01:01",
"message" => "May 01 01:01:01 User1 CREATE "Create a new folder"\r"
}
{
"action_performed" => "OPEN",
"user" => "User2",
"path" => "C:\Users\pawnow\logstash\mysample-input.log",
"@timestamp" => 2018-06-26T10:56:06.497Z,
"@version" => "1",
"action_description" => ""Open a new folder"",
"host" => "DVIG1673",
"tags" => [
[0] "_dateparsefailure"
],
"timestamp" => "January 02 02:02:02",
"message" => "January 02 02:02:02 User2 OPEN "Open a new folder"\r"
}
But file is:
May 01 01:01:01 User1 CREATE "Create a new folder"
January 02 02:02:02 User2 OPEN "Open a new folder"
December 03 03:03:03 User3 SHUTDOWN "Shutdown a comupter"

Question 3
That's make sense. Thanks for explanation

4

After indexing I tried to get document with 'GET /index_name/type_name/1?pretty' command which was incorrect, but why?

You apparently don't have a document with id "1". They're not numbered sequentially. The search output tells you the id of each document.

But file is:
May 01 01:01:01 User1 CREATE "Create a new folder"
January 02 02:02:02 User2 OPEN "Open a new folder"
December 03 03:03:03 User3 SHUTDOWN "Shutdown a comupter"

Does the file end with a newline character?

5

That solved my problem. Thank You

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.