Config file not getting read by logstash

Elastic Stack Logstash
1

Hi, I have set up Elk stack on my windows machine with the following :

  • Elasticserach
  • Logstash
  • Kibana

My logstash.conf'
input {
file {
path => "\bin\MylogFile.log"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => localhost:9200
}
}

MylogFile.log(Apache Log)

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"

When i run logstash.conf it creates the following index in elasticsearch :

   health  status   index                            
   yellow  open     logstash-2016.10.06

The following index is empty and does not get any data from my log file. Please help? I am very new to Elk stack.

2

Things to look into:

  • Logstash could be tailing the file. Check the sincedb file (see logs for its path) to be sure. Delete it or set sincedb_path to /dev/null.
  • If the input file is older than 24 hours you need to adjust the ignore_older option.
  • Does the user Logstash runs as have permission to the file?
  • I wonder if you need to include the drive letter in the absolute path.
3

Thank you for your fast response Magnus, where is the sincedb file located ?

Do you mean like this :
input {
file {
path => "\bin\MylogFile.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
elasticsearch {
hosts => localhost:9200
}
}

4

where is the sincedb file located ?

See the file input plugin documentation. The exact path is listed in Logstash's log, at least if you enable verbose logging.

sincedb_path => "/dev/null"

/dev/null is for non-Windows, sorry. Use nul on Windows.

5

I have found my issue. It appears i was looking at the wrong URL.

The URL that i was looking at was URL : localhost:9200/logstash-2016.10.10?=pretty=true which returns the setting of the index.

I then used this URL : http://localhost:9200/logstash-2016.10.10/_search?=pretty=true

My log file :

1234 hello
1234 hello
1234 hello
1234 hello
1234 hello

When i do a search query URL : http://localhost:9200/logstash-2016.10.10/_search?=pretty=true

Output :

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 387,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "logstash-2016.10.10",
      "_type" : "logs",
      "_id" : "AVey5QtVcMR1QqUFiopI",
      "_score" : 1.0,
      "_source" : {
        **"message" : "1234 hello\r",**
        "@version" : "1",
        "@timestamp" : "2016-10-10T08:38:57.043Z",
        "path" : "C:\\Elk\\logstash\\bin\\MylogFile.log",
        "host" : "admin1"
      }
    }, {
      "_index" : "logstash-2016.10.10",
      "_type" : "logs",
      "_id" : "AVey5QtVcMR1QqUFiopM",
      "_score" : 1.0,
      "_source" : {
        **"message" : "1234 hello\r",**
        "@version" : "1",
        "@timestamp" : "2016-10-10T08:38:57.046Z",
        "path" : "C:\\utils\\Elk\\logstash\\bin\\MylogFile.log",
        "host" : "admin1"

I now can see 1234 hello pass in the above query.

I have a few questions as to what i want to achieve with my Elk stack, please guide.

  • Logstash is creating its own index file when I run logstash.conf, how do i change the name of this index to my own name ?

  • Is grok built into logstash or do i need to install it as a plugin?

  • Does Kibana require filtered data to generated a visual representation of data?

Thank you for your response Magnus , i really appreciate it as i am very new to the Elk Stack.

6

Logstash is creating its own index file when I run logstash.conf, how do i change the name of this index to my own name ?

Use the elasticsearch output's index option, but beware that the default index template that Logstash installs only applies to indexes named logstash-*. Until you understand index templates and field mappings I suggest you keep the default index name.

Is grok built into logstash or do i need to install it as a plugin?

It's a plugin that's installed by default.

Does Kibana require filtered data to generated a visual representation of data?

If you want to visualize anything other than the number of events per unit of time, yes.

1 Like
7

I would like to try and see if i can get my data indexed into kibana by using a grok filter in my config file. I will follow up soon with feedback. Thank you for your reply,

8

Hi Magnus,

I managed to filter out example data with grok and its passing through into elasticsearch and Kibana. :slight_smile:

I have a few more question, I apologize for the inconvenience as I am new to Elk Stack.

  • I'm using windows, I have installed CURL then i ran the following in PowerShell :

curl http://localhost:9200/logstash-2016.10.12

OUTPUT
StatusCode : 200
StatusDescription : OK
Content : {"logstash-2016.10.12":{"aliases":{},"mappings":{"_default_":{"_all":``{"enabled":true,"omit_norms":true},"dynamic_templates":[{"message_field":{"mapping":``{"index":"analyzed","omit_norms":true,"fielddat
RawContent : HTTP/1.1 200 OK
Content-Length: 2384
Content-Type: application/json; charset=UTF-8

Then I tried the following command in order to delete an index:

curl -XDELETE http://localhost:9200/logstash-2016.10.12

OUTPUT - Error
Invoke-WebRequest : A parameter cannot be found that matches parameter name 'XDELETE'.
At line:1 char:6
+ curl -XDELETE http://localhost:9200/logstash-2016.10.12
+ ~~~~~~~~
+CategoryInfo : InvalidArgument: (:) [Invoke-WebRequest], ParameterBindingException
+ FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

I'm not sure why i cant use these commands in PowerShell ?

  • I would like to find out what tools can i use to create my own regular expression

Example
search?word=cars*

How can I go about creating a regular expression for the above to extract key word "cars" that is Grok friendly?

Thank You Magnus

9

I'm not sure why i cant use these commands in PowerShell ?

I don't know PowerShell and its HTTP client, but what you're looking for is changing the HTTP method from GET to DELETE. You might need to do that differently.

How can I go about creating a regular expression for the above to extract key word "cars" that is Grok friendly?

You want to extract the string "cars" from "search?word=cars*"?

1 Like
10

Thanks for your fast response.

I don't know PowerShell and its HTTP client, but what you're looking for is changing the HTTP method from GET to DELETE. You might need to do that differently.

I will do more research on PowerShell using CURL.

You want to extract the string "cars" from "search?word=cars*"?

yes, i want to extract the string "cars", how do i go about doing this please advise and i will give it a try, are there any tools for this ?

11

I don't know PowerShell and its HTTP client, but what you're looking for is changing the HTTP method from GET to DELETE. You might need to do that differently.

I will do more research on PowerShell using CURL.

I managed to get this working in PowerShell by using

**Invoke-RestMethod -Uri** http://localhost:9200/logstash-2016.10.12 -Method Delete:grin:

12

Hi Magnus,

Thank you for your advice i managed to find something called a grok debugger which can help me extract the string from "search?word=cars*"?.

Thank you for help.