Logstash Help Config file

Elastic Stack Logstash
1

Hi,

I am trying to ingest a bunch of logs into my Elastic cluster through Logstash, however I am having a hard time with the config file.

Here is what my log files look like:

    #Software: Microsoft Internet Information Services 10.0
    #Version: 1.0
    #Date: 2019-10-16 22:52:00
    #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
    2019-10-16 22:52:00 ::1 GET / - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/77.0.3865.120+Safari/537.36 - 200 0 0 578
    2019-10-16 22:52:00 ::1 GET /iisstart.png - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/77.0.3865.120+Safari/537.36 http://localhost/ 200 0 0 2
    2019-10-16 22:52:00 ::1 GET /favicon.ico - 80 - ::1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/77.0.3865.120+Safari/537.36 http://localhost/ 404 0 2 10

And here is my config file (I am using dissect for filter, is that an appropriate use?):

    input {
    	file {
    		path => "C:\inetpub\logs\LogFiles\W3SVC1\*"
    		start_position => "beginning"
    		sincedb_path => "NUL"
    	}
    }
    filter {
    	dissect {
    		mapping => {
    			"message" => 
    				'%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}'
    		}
    	}
    }
    output {
    	elasticsearch {
    		hosts => ["172.16.1.81:9200", "172.16.1.125:9200", "172.16.1.126:9200"]
    		index => "vs_php_logs_v0"
    		document_type => "_doc"
    	}
    	stdout{}
    }

When I run logstash with this config file, here's the output:

    Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option UseConcMarkSweepGC; support was removed in 14.0
    Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option CMSInitiatingOccupancyFraction; support was removed in 14.0
    Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option UseCMSInitiatingOccupancyOnly; support was removed in 14.0
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by org.jruby.runtime.encoding.EncodingService (file:/C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.Console.cs
    WARNING: Please consider reporting this to the maintainers of org.jruby.runtime.encoding.EncodingService
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    Sending Logstash logs to C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logs which is now configured via log4j2.properties
    [2020-04-29T16:42:59,519][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2020-04-29T16:42:59,532][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.8"}
    [2020-04-29T16:43:00,159][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 4, column 22 (byte 86) after input {\r\n\tfile {\r\n\t\tpath => \"C:\\inetpub\\logs\\LogFiles\\W3SVC1\\\"\r\n\t\tstart_position => \"", :backtrace=>["C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/pipeline_action/create.rb:43:in `block in execute'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/agent.rb:96:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in `synchronize'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/agent.rb:96:in `exclusive'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/pipeline_action/create.rb:39:in `execute'", "C:/Users/vincent.stevenson/Documents/logstash-6.8.8/logstash-core/lib/logstash/agent.rb:334:in `block in converge_state'"]}
    [2020-04-29T16:43:00,380][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

However, it just sits here and doesn't actually put any documents into the index I created: vs_php_logs_v0 despite there being several files in this directory.

How should I modify the config file to iterate through the log files in this directory and put them into the elastic cluster?

Thanks!

Vincent

2

I do not understand why you are getting that error for your configuration file. The config parser is not looking for the value of an option after 'start_position => "'. The most common reason I see that is if you actually have a curly quote ending the previous line, so that it does not end the path option. But you appear to have a straight double quote there. :no_idea:

This is a good example of where dissect is appropriate.

Do not use backslash in the path option of a file input, use forward slash (or, if you really insist, double backslash, so that the first escapes the second).

3 
    input {
    	file {
    		type => "IISLog"
    		path => "C:/inetpub/logs/LogFiles/W3SVC*/*.log"
    		start_position => "beginning"
    	}
    }

    filter {

    	# ignore log comments
    	if [message] =~ "^#" {
    		drop {}
    	}
     
     	# check that fields match your IIS log settings
    	grok {
        match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} (%{URI:referer})? %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}"]
    	}
      
    	# set the event timestamp from the log
    	# https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
    	date {
    		match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
    		timezone => "Etc/UCT"
    	}
    	
    	# matches the big, long nasty useragent string to the actual browser name, version, etc
    	# https://www.elastic.co/guide/en/logstash/current/plugins-filters-useragent.html
    	useragent {
    		source=> "useragent"
    		prefix=> "browser_"
    	}
    	
    	mutate {
    		remove_field => [ "log_timestamp"]
    	}
    }

    # output logs to console and to elasticsearch
    output {
    stdout { codec => rubydebug }
    	elasticsearch { hosts => ["localhost:9200"] }
    }

This config file helped me (source: https://gist.github.com/rebornix/f2dd3cd8bfaca14f40a9#file-logstash-conf-L4)

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.