Logstash unable to read file in 6.6.0

Elastic Stack Logstash
1

Hello Team,

I have a .conf file which works perfectly fine with Logstash 6.3.2 but the same conf file is not working in 6.6.0

Please see below the conf file;

input
{
	#stdin { }
	
	file
	{
		path => ["C:\ELK\Test.txt"]
		start_position => "beginning"
		sincedb_path => "NUL"
	}
}

filter
{	
	
	if "'Client:'" in [message]
	{
		grok
		{
			match => { "message" => "\[%{TIMESTAMP_ISO8601:logtime}\] %{LOGLEVEL:loglevel} *%{DATA:systemdata} \[\[%{DATA:app_name}\].%{DATA:rubbish}: Recieved Request: 'Query Params:' ParameterMap{\[%{DATA:mapdata}\]\} 'Client:' %{GREEDYDATA:client}" }
		}
		
		kv
		{
			source => "mapdata"
			field_split_pattern => ", "
		}
		mutate { remove_field => ["rubbish", "mapdata"] }
	}
	
	date 
	{
		match => [ "logtime" , "yyyy-MM-dd HH:mm:ss.SSS Z" , "ISO8601", "yyyy-MM-dd HH:mm:ss.SSS" , "yyyy-MM-dd HH:mm:ss.S" ]
		target => "@timestamp"
		remove_field => [ "logtime" ]
	}
}

output
{	
	
	if "'Client:'" in [message]
	{
		stdout { codec => dots }
		
		#stdout { codec => rubydebug }
		
		elasticsearch
		{
			id => "Test"
			index => "Test"
			hosts => ["localhost:9200"]
		}
	}
}

Any help highly appreciated!

Regards

Shilpa

2

Can you show an example message that this is processing?

3

Hey @Badger here is the sample lines of the logfile that i am feeding as input;

[2019-02-11 13:44:51.831] INFO    com.mulesoft.ch.monitoring.MonitoringCoreExtension [qtp801696708-35]: Registering ping flow injector...
[2019-02-11 13:44:52.119] INFO    com.mulesoft.ch.queue.boot.PersistentQueueCoreExtension [qtp801696708-35]: The PersistentQueueManager is NOT configured. The normal VM queue manager will be used.
[2019-02-11 13:50:54.149] INFO    org.mule.api.processor.LoggerMessageProcessor [[pim-tab-experience-api-prod].httpListenerConfig.worker.01]: Recieved Request: 'Query Params:' ParameterMap{[sortOrder=[modifiedFirst], pageSize=[10], pageNumber=[1], countryCode=[US], languageCode=[EN], gtins=[00079400392954], brands=[AXE]]} 'Client:' Aman-TestAPP

The expectation is only to process the third line in the above set of lines, which is perfectly working with Logstash 6.3.2 but not working with Logstash 6.6.0

Please kindly let us know what exactly has changed and what could be the issue?

4

With both 6.5.4 and 6.6.0 I get

    "app_name" => "pim-tab-experience-api-prod",
    "pageSize" => "10",
       "gtins" => "00079400392954",
  "pageNumber" => "1",
 "countryCode" => "US",
      "brands" => "AXE",
"languageCode" => "EN",
      "client" => "Aman-TestAPP ",
  "@timestamp" => 2019-02-11T18:50:54.149Z,
   "sortOrder" => "modifiedFirst",
    "loglevel" => "INFO",
  "systemdata" => "org.mule.api.processor.LoggerMessageProcessor"
5

Hi @Badger please see below as i cannot see any output after running the logstash.

C:\Users\Downloads\logstash-6.6.0\logstash-6.6.0\bin>logstash -f "C:\Users\Downloads\ELK\pimtab-exp-api.conf"
Sending Logstash logs to C:/Users/Downloads/logstash-6.6.0/logstash-6.6.0/logs which is now configured via log4j2.properties
[2019-02-17T01:35:39,123][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-17T01:35:39,169][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.0"}
[2019-02-17T01:35:55,107][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-17T01:35:55,833][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-02-17T01:35:56,209][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-02-17T01:35:56,300][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-02-17T01:35:56,315][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-02-17T01:35:56,395][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-02-17T01:35:56,417][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-02-17T01:35:56,470][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-02-17T01:35:57,533][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4dbfdcbe run>"}
[2019-02-17T01:35:57,643][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-17T01:35:57,664][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-17T01:35:58,179][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I dont see any dots as per my conf file. Did you try to use the same conf file data i have posted here earlier?

Please let me know if you need anything more to resolve or help?

6

Change that to C:/ELK/Test.txt.

7

@Badger It is working now! :slight_smile: :+1:

Why did this had to change in Logstash 6.6.0 :smile:

Anyways, thanks a lot for identifying the issue.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.