Input File Path (not working) vs Input Beats (working) all conf setting are the same but Input

Elastic Stack Logstash
1

Hi @ all,

I mess arround with follwing and hope you have some Ideas:

input {
    file {
        path => ["C:\EsKiLo_6.5.4\Test.csv"]
		start_position => "beginning"
	}
}

filter {	
		csv {
			columns =>["Date","Time","Year"]
			separator => "|"
			}
	}	
output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "testone"
			}
}

------ vs: ------

input {
 beats {
    port => 5044
	}
}

filter {	
		csv {
			columns =>["Date","Time","Year"]
			separator => "|"
			}
	}	
output {

		elasticsearch {
			hosts => ["localhost:9200"]
			index => "testone"
			}
}

Filebeat:

filebeat.prospectors:
- input_type: log
  paths:
    - C:\EsKiLo_6.5.4\Test.csv
  fields:
    logfiletype: test

Path = no errors in log AND no Index is created?
Beats = boom ready

What's up with Logstash using a path on a Windows 64bit PC
For Quick Testes I need to cleanup Filebeat instead of just using the logstash.conf, thats a bit hm hm hm...

Thanks in advance for suggestions

2

Does adding this to the file input help?

sincedb_path => "NUL"
3

Hi,
no nfortunately not :frowning:

[2019-01-18T15:04:33,844][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x15d7664 run>"}
[2019-01-18T15:04:33,941][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-01-18T15:04:33,992][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-01-18T15:04:34,454][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

grafik

It is ok If I have to use Beat (however why), but what I'm really interested in is why it doesn't work with Logstsh.

4

Enable trace level logging and look for filewatch. You can turn on trace globally on the command line or through logstash.yml, or you can enable trace just for the filewatch modules through the api as described here.

5

Hi,
OK tried but (due to lack of experiences not right!) :slight_smile:

curl -XGET localhost:9600/_node/logging?pretty = it works and give me an json overview

curl -XPUT localhost:9600/_node/logging?pretty -H 'Content-Type: application/json' -d'
{
"logger.filewatch.discoverer" : "TRACE"
}
'
Result:
.... block in spawn_thread'"]}curl: (6) Could not resolve host: application

Logstash log:

[2019-01-18T17:13:14,515][ERROR][logstash.agent           ] Internal API server error {:status=>500, :request_method=>"PUT", :path_info=>"/_node/logging", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*", :error=>"Unexpected Internal Error", :class=>"LogStash::Json::ParserError", :message=>"Unexpected character (''' (code 39)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: (byte[])\"'\"; line: 1, column: 2]", :backtrace=>["C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/json.rb:16:in `jruby_load'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/modules/logging.rb:28:in `block in PUT /'", "org/jruby/RubyMethod.java:127:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block in route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'", "org/jruby/RubyKernel.java:1114:in `catch'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/xss_header.rb:18:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/path_traversal.rb:16:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/json_csrf.rb:18:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/base.rb:49:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-protection-1.5.5/lib/rack/protection/frame_options.rb:31:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/nulllogger.rb:9:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/head.rb:13:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/urlmap.rb:66:in `block in call'", "org/jruby/RubyArray.java:1734:in `each'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/urlmap.rb:50:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/rack_app.rb:57:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/logstash-core/lib/logstash/api/rack_app.rb:31:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.11/lib/rack/builder.rb:153:in `call'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:557:in `handle_request'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:404:in `process_client'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:270:in `block in run'", "C:/Analytics/EsKiLo_6.5.4/Logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:106:in `block in spawn_thread'"]}
[2019-01-18T17:13:14,539][ERROR][logstash.agent           ] API HTTP Request {:status=>500, :request_method=>"PUT", :path_info=>"/_node/logging", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}

I'm off here !

Thanks for reply

6

Finaly made it!

I add thus to Input:

codec => plain {
charset => "ISO-8859-1"
}

And used this commandline toi start for debug:

logstash --log.level debug -f aTestfile_path_3cols.conf

and found that entry in logstash logs:

[2019-01-18T22:18:09,404][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

Search the Forum and find this:

Yeah made my day when changed from \ to / in path!

Lesson learned ELK Stack is quite a bit hypersensitive based on whatever matter because in Filebeat I can still use \ instead of / ?
...time for beer... :slight_smile:

Regards

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.