Logstash executable accepts filebeat data, but when installed as service not working with same conf files


(shah) #1

Hi All,

I am in process of setting up ELK for monitoring my data. Elastic and Kibana is installed as service.

For testing purpose i had filebeat and logstash running from executable. I checked everything is working fine and so i tried to install both filebeat and logstash as service. Now all of sudden everything is stopped.

So to debug i uninstalled logstash as service. Ran it from executable, but no change. Then when i restart the filebeat service, logstash pickups the new change.

I need to install them as service. Can someone please help me here. For installing as service i used NSSM. Is there any setting required on service level?

Filebeat config:

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
#- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    #- C:\temp\sunny\*
    - C:\temp\DK\*
    - C:\temp\WW\*
    - C:\temp\NW\*
    - C:\temp\HS\*


#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
  
output.console:
  enabled: true

Logstash config:--

input {
  beats {
   port => 5044
   type => "log"
  }
}



filter {

  if [source] =~ "C:\\temp\\DK\\.*" {
  
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : Test run configuration: RunId: (?<runid>[^\s]*), Team: (?<Team>[^\s]*), FrameWork:CTFXUnit, Branch: (?<Branch>[^\s]*), Environment:(?<Environment>[^\s]*), Portal:(?<Portal>[^\s]*), Browser:(?<Browser_Type>[^\s]*), EnableDatabaseLog:True, EnableTestRailLog:True, TestRailConfiguration:Milestone: , TestRunType: TestPlan, TestPlan:(?<Test_Plan>[^\s]*), UpdateExistingTestRailRun: True., DebugLogLevel:2, TraitFilters:Name:Category, Operator:Equal, Value:Dothraki,"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} ERROR: \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	add_field => ["Trait_Value", "Dothraki"]
	}

  date {
    match => [ "timestamp", "ISO8601"]
	remove_field => ["timestamp"]
	}

  if "_grokparsefailure" in [tags] 
  
  { drop {} }
	
  }

  if [source] =~ "C:\\temp\\WW\\.*" {
  
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : Test run configuration: RunId: (?<runid>[^\s]*), Team: (?<Team>[^\s]*), FrameWork:CTFXUnit, Branch: (?<Branch>[^\s]*), Environment:(?<Environment>[^\s]*), Portal:(?<Portal>[^\s]*), Browser:(?<Browser_Type>[^\s]*), EnableDatabaseLog:True, EnableTestRailLog:True, TestRailConfiguration:Milestone: , TestRunType: TestPlan, TestPlan:(?<Test_Plan>[^\s]*), UpdateExistingTestRailRun: True., DebugLogLevel:2, TraitFilters:Name:Category, Operator:Equal, Value:WhiteWalkers,"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} ERROR: \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	add_field => ["Trait_Value", "WhiteWalkers"]
	}

  date {
    match => [ "timestamp", "ISO8601"]
	remove_field => ["timestamp"]
	}

  if "_grokparsefailure" in [tags] 
  
  { drop {} }
	
  }

  
  if [source] =~ "C:\\temp\\NW\\.*" {
  
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : Test run configuration: RunId: (?<runid>[^\s]*), Team: (?<Team>[^\s]*), FrameWork:CTFXUnit, Branch: (?<Branch>[^\s]*), Environment:(?<Environment>[^\s]*), Portal:(?<Portal>[^\s]*), Browser:(?<Browser_Type>[^\s]*), EnableDatabaseLog:True, EnableTestRailLog:True, TestRailConfiguration:Milestone: , TestRunType: TestPlan, TestPlan:(?<Test_Plan>[^\s]*), UpdateExistingTestRailRun: True., DebugLogLevel:2, TraitFilters:Name:Category, Operator:Equal, Value:NightsWatch,"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} ERROR: \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	add_field => ["Trait_Value", "NightsWatch"]
	}

  date {
    match => [ "timestamp", "ISO8601"]
	remove_field => ["timestamp"]
	}

  if "_grokparsefailure" in [tags] 
  
  { drop {} }
	
  }
  
  
  
   if [source] =~ "C:\\temp\\HS\\.*" {
  
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : Test run configuration: RunId: (?<runid>[^\s]*), Team: (?<Team>[^\s]*), FrameWork:CTFXUnit, Branch: (?<Branch>[^\s]*), Environment:(?<Environment>[^\s]*), Portal:(?<Portal>[^\s]*), Browser:(?<Browser_Type>[^\s]*), EnableDatabaseLog:True, EnableTestRailLog:True, TestRailConfiguration:Milestone: , TestRunType: TestPlan, TestPlan:(?<Test_Plan>[^\s]*), UpdateExistingTestRailRun: True., DebugLogLevel:2, TraitFilters:Name:Category, Operator:Equal, Value:HouseStark,"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} ERROR: \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} INFO : \[(?<Test_Run_Status>[^\s]*)\] LMS.XUnit.Tests.(?<Test_Name>[^\s]*)"}
	add_field => ["Trait_Value", "HouseStark"]
	}

  date {
    match => [ "timestamp", "ISO8601"]
	remove_field => ["timestamp"]
	}

  if "_grokparsefailure" in [tags] 
  
  { drop {} }
	
  }
}

output {
  stdout {}
  elasticsearch {
	hosts => "localhost:9200"
	index => "logstash-json"
  }
}

(Rodolphe Redouté) #2

try this in your logstash config (i think it's the correct syntax : hosts => ["localhost:9200"]
the "[]" are important.

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts

if it doesn't work, can you give the error log pls ?


(shah) #3

Hello Rodolphe, thks for your reply. I am new to ELK. When i am running from console it always gives me error which i have attached here. Also please if you can guide me what configuration i can do to generate error log. I do see there is logstash.yml file and it says something about logging but i have no clue on how to pass the values.


(shah) #4

FYI, i use to ignore the error msg and it still works fine from executable. But when i install it as service it does not work. I did the change u asked for but same result.


(shah) #5

Also logstash-plain.log and slow log both files are blank. what i did is i deleted those files from log folder and then restarted the logstash service. These files got created in log folder but they are empty. When i start from service it does write to these log files.


(Rodolphe Redouté) #6

i'm running Logstash on centos so i don't really know how it is on windows, but, the error happen when you try to start logstash service ? ou when you try to start filebeat service ?


(shah) #7

I was just able to get log output. Hope this helps to see what issue is


(shah) #8

I think when i start logstash as service its not able to find my logstash.conf file.


(Rodolphe Redouté) #9

did you informed the path.config in your logstash.yml file then ?


(shah) #10

No you are right i have not specified it. my logstash.conf is at below location

C:\Users\sgorasia\Downloads\logstash-5.5.1.2\logstash-5.5.1\bin

So what would be the format i need to give in logstash.yml file? Please advise

pathconfig


(Rodolphe Redouté) #11

it should be :
path.config: "C:\Users\sgorasia\Downloads\logstash-5.5.1.2\logstash-5.5.1\bin"
(without the "#" on the line)

look here it can be helpful :
https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html


(shah) #12

i tried entering the value as u mentioned but now something went wrong. log file is empty again.


(Rodolphe Redouté) #13

delete the "logstash.conf" you need a path to the directory, not a path to the file itself


(Christian Dahlqvist) #14

Yaml is also sensitive to indentation, and it looks like you have a space ahead of your line that you should remove.


(shah) #15

Guys did as mentioned and got new error msg now.

Error msg:

[2017-08-30T15:30:54,738][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "" hasn't been registered>, :backtrace=>["C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/settings.rb:32:in get_setting'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/settings.rb:64:inset_value'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/settings.rb:83:in merge'", "org/jruby/RubyHash.java:1342:ineach'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/settings.rb:83:in merge'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/settings.rb:135:invalidate_all'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:244:in execute'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:209:in run'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "C:\Users\sgorasia\Downloads\logstash-5.5.1.2\logstash-5.5.1\lib\bootstrap\environment.rb:71:in `(root)'"]}


(shah) #16

I changed the way i was specifying path.config value and it gave different error this time

Error:
[2017-08-30T15:55:13,640][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 3, column 1 (byte 26) after #!/usr/bin/env bin/ruby\n\n", :backtrace=>["C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/pipeline.rb:59:in initialize'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/pipeline.rb:156:ininitialize'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/agent.rb:286:in create_pipeline'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/agent.rb:95:inregister_pipeline'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:314:in execute'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/logstash-core/lib/logstash/runner.rb:209:in run'", "C:/Users/sgorasia/Downloads/logstash-5.5.1.2/logstash-5.5.1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "C:\Users\sgorasia\Downloads\logstash-5.5.1.2\logstash-5.5.1\lib\bootstrap\environment.rb:71:in `(root)'"]}
[2017-08-30T15:55:13,643][DEBUG][logstash.agent ] starting agent
[2017-08-30T15:55:13,646][DEBUG][logstash.agent ] Starting puma
[2017-08-30T15:55:13,646][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-08-30T15:55:13,647][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-08-30T15:55:13,648][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping


(Aaron Mildenstein) #17

Part of the issue is that this is a YAML file, and you have an indent (even a single space is an indent). path.config should be all the way to the left, with no preceding space.

Next, there should be a space after the colon, and you should not be looking in the bin directory. Your line should look more like

path.config: "C:\Users\sgorasia\Downloads\logstash-5.5.1.2\logstash-5.5.1\my_configuration"

...where my_configuration is where the pipeline files are found, e.g. logstash.conf. As Logstash looks at all files in this directory, it is not appropriate to point it to the bin directory. This directory does not need to exist within the Logstash installation path, and could instead be something like C:\Users\sgorasia\Documents\Logstash


(shah) #18

Alright guys it works now. Thanks Aaron, Christian and Rodolphe.

Syntax is

 path.config: C:\Users\sgorasia\Downloads\logstash-5.5.1.2\logstash-5.5.1\logstash_grok

(system) #19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.