Index not appearing in kibana via logstash

Elastic Stack Logstash
1

Hello,
I am new to ELK, trying to read my application logs and wishing to show it on kibana dashboard for analysis purposes. Elasticsearch, kibana and logstash instances are up and running. While trying to see the newly created index via _cat/indices, it's not appearing.
image

Logstash logs are also not showing any error, as below:-

./logstash -f ../config/Server_Logs.conf
Using bundled JDK: /data01/logstash-7.11.2/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /data01/logstash-7.11.2/logs which is now configured via log4j2.properties
[2022-06-26T13:51:59,237][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.11.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2022-06-26T13:52:00,089][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-26T13:52:02,785][INFO ][org.reflections.Reflections] Reflections took 42 ms to scan 1 urls, producing 23 keys and 47 values
[2022-06-26T13:52:03,933][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9202/]}}
[2022-06-26T13:52:04,122][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9202/"}
[2022-06-26T13:52:04,171][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2022-06-26T13:52:04,175][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2022-06-26T13:52:04,232][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9202"]}
[2022-06-26T13:52:04,347][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-06-26T13:52:04,459][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2022-06-26T13:52:04,484][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/data01/logstash-7.11.2/config/Sprint_FinnAxia_Server_Logs.conf"], :thread=>"#<Thread:0x378c0ed1 run>"}
[2022-06-26T13:52:05,427][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.94}
[2022-06-26T13:52:05,705][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-26T13:52:05,766][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-06-26T13:52:06,049][INFO ][filewatch.observingtail  ][main][3c69c4644d304af52cd42ba88cbcbdc73f17c1e8b684ec79f51ecb8b315765a4] START, creating Discoverer, Watch with file and sincedb collections
[2022-06-26T13:52:06,130][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

My logstash conf file as below:-

input {
  file 
  {
	path => "/data01/ApplicationLogs/services.log"
	start_position => "beginning"
	sincedb_path => "/dev/null"
	ignore_older => 1
	type => "application_dev"
	codec => multiline {	
		pattern => "^%{NUMBER}"
		negate => "true"
		what => "previous"
		auto_flush_interval => 5
	 }
   }
}

filter {
  grok {
     	match => { "message" => "(?m)^%{TIMESTAMP_ISO8601:Log_Timestamp},%{NUMBER:No} \[(?<LogSeverity>(INFO|DEBUG|WARNING|SEVERE|ERROR|WARN|FAIL|APEK))\] \[%{GREEDYDATA:WorkerID}\] \[%{GREEDYDATA:ClassName}:%{NUMBER:LineNo}\] %{GREEDYDATA:LogMessage}"}
	} 
}

output {
  stdout {
  codec => json 
  }
  elasticsearch 
  {
	hosts => ["localhost:9202"]
	index => "dev_logging"
	retry_on_conflict => 3
  } 
}

Not sure if i have missed anything.

2

You can see right here it is running the wrong / not specified conf

-f try an absolute path

3

I initially placed my .conf file in config folder(hence the relative path) but later moved to bin folder. Fresh debug logs

./logstash -f Sprint_Server_Logs.conf
Using bundled JDK: /data01/logstash-7.11.2/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /data01/logstash-7.11.2/logs which is now configured via log4j2.properties
2022-06-27 10:18:16,229 main DEBUG Apache Log4j Core 2.13.3 initializing configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@65a03e98
2022-06-27 10:18:16,231 main DEBUG Installed 2 script engines
2022-06-27 10:18:16,252 main DEBUG JSR 223 JRuby Engine version: 9.2.13.0, language: ruby, threading: THREAD-ISOLATED, compile: true, names: [ruby, jruby], factory class: org.jruby.embed.jsr223.JRubyEngineFactory
2022-06-27 10:18:16,521 main DEBUG Oracle Nashorn version: 11.0.8, language: ECMAScript, threading: Not Thread Safe, compile: true, names: [nashorn, Nashorn, js, JS, JavaScript, javascript, ECMAScript, ecmascript], factory class: jdk.nashorn.api.scripting.NashornScriptEngineFactory
2022-06-27 10:18:16,521 main DEBUG PluginManager 'Core' found 122 plugins
2022-06-27 10:18:16,522 main DEBUG PluginManager 'Level' found 0 plugins
2022-06-27 10:18:16,522 main DEBUG PluginManager 'Lookup' found 16 plugins
2022-06-27 10:18:16,526 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2022-06-27 10:18:16,527 main DEBUG createAppenderRef(ref="deprecation_plain_rolling", level="null", Filter=null)
2022-06-27 10:18:16,527 main DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2022-06-27 10:18:16,528 main DEBUG createLogger(additivity="false", level="WARN", name="deprecation", includeLocation="null", ={deprecation_plain_rolling}, ={}, Configuration(LogstashPropertiesConfig), Filter=null)
2022-06-27 10:18:16,528 main DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2022-06-27 10:18:16,529 main DEBUG createLogger(additivity="true", level="ERROR", name="logstash.licensechecker.licensereader", includeLocation="null", ={}, ={}, Configuration(LogstashPropertiesConfig), Filter=null)
2022-06-27 10:18:16,530 main DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2022-06-27 10:18:16,531 main DEBUG createLogger(additivity="true", level="FATAL", name="org.apache.http", includeLocation="null", ={}, ={}, Configuration(LogstashPropertiesConfig), Filter=null)
2022-06-27 10:18:16,531 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2022-06-27 10:18:16,532 main DEBUG createAppenderRef(ref="plain_rolling_slowlog", level="null", Filter=null)
2022-06-27 10:18:16,532 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2022-06-27 10:18:16,532 main DEBUG createAppenderRef(ref="plain_console_slowlog", level="null", Filter=null)
2022-06-27 10:18:16,532 main DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2022-06-27 10:18:16,533 main DEBUG createLogger(additivity="false", level="TRACE", name="slowlog", includeLocation="null", ={plain_rolling_slowlog, plain_console_slowlog}, ={}, Configuration(LogstashPropertiesConfig), Filter=null)
2022-06-27 10:18:16,533 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2022-06-27 10:18:16,533 main DEBUG createAppenderRef(ref="deprecation_plain_rolling", level="null", Filter=null)
2022-06-27 10:18:16,534 main DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2022-06-27 10:18:16,534 main DEBUG createLogger(additivity="false", level="WARN", name="org.logstash.deprecation, deprecation", includeLocation="null", ={deprecation_plain_rolling}, ={}, Configuration(LogstashPropertiesConfig), Filter=null)
2022-06-27 10:18:16,534 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2022-06-27 10:18:16,535 main DEBUG createAppenderRef(ref="plain_console", level="null", Filter=null)
2022-06-27 10:18:16,535 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2022-06-27 10:18:16,535 main DEBUG createAppenderRef(ref="plain_rolling", level="null", Filter=null)
2022-06-27 10:18:16,537 main DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2022-06-27 10:18:16,537 main DEBUG createAppenderRef(ref="pipeline_routing_appender", level="null", Filter=null)
2022-06-27 10:18:16,537 main DEBUG Building Plugin[name=root, class=org.apache.logging.log4j.core.config.LoggerConfig$RootLogger].
2022-06-27 10:18:16,538 main DEBUG createLogger(additivity="null", level="INFO", includeLocation="null", ={plain_console, plain_rolling, pipeline_routing_appender}, ={}, Configuration(LogstashPropertiesConfig), Filter=null)
2022-06-27 10:18:16,538 main DEBUG Building Plugin[name=loggers, class=org.apache.logging.log4j.core.config.LoggersPlugin].
2022-06-27 10:18:16,538 main DEBUG createLoggers(={deprecation, logstash.licensechecker.licensereader, org.apache.http, slowlog, org.logstash.deprecation, deprecation, root})
2022-06-27 10:18:16,539 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
2022-06-27 10:18:16,539 main DEBUG PatternLayout$Builder(pattern="[%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n", PatternSelector=null, Configuration(LogstashPropertiesConfig), Replace=null, charset="null", alwaysWriteExceptions="null", disableAnsi="null", noConsoleNoAnsi="null", header="null", footer="null")
2022-06-27 10:18:16,540 main DEBUG PluginManager 'Converter' found 44 plugins
2022-06-27 10:18:16,541 main DEBUG Building Plugin[name=TimeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.TimeBasedTriggeringPolicy].
2022-06-27 10:18:16,543 main DEBUG TimeBasedTriggeringPolicy$Builder(interval="1", modulate="true", maxRandomDelay="null")
2022-06-27 10:18:16,544 main DEBUG Building Plugin[name=SizeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.SizeBasedTriggeringPolicy].
2022-06-27 10:18:16,544 main DEBUG createPolicy(size="100MB")
2022-06-27 10:18:16,546 main DEBUG Building Plugin[name=Policies, class=org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy].
2022-06-27 10:18:16,546 main DEBUG createPolicy(={TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)})
2022-06-27 10:18:16,546 main DEBUG Building Plugin[name=Script, class=org.apache.logging.log4j.core.script.Script].
2022-06-27 10:18:16,550 main DEBUG createScript(name="filter_no_pipelined", language="JavaScript", scriptText="false == false || !(logEvent.getContextData().containsKey("pipeline.id"))")
2022-06-27 10:18:16,550 main DEBUG Building Plugin[name=filter, class=org.apache.logging.log4j.core.filter.ScriptFilter].
2022-06-27 10:18:16,551 main DEBUG createFilter(Script(filter_no_pipelined), onMatch="null", onMismatch="null", Configuration(LogstashPropertiesConfig))
2022-06-27 10:18:16,557 main DEBUG Building Plugin[name=DefaultRolloverStrategy, class=org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy].
2022-06-27 10:18:16,559 main DEBUG DefaultRolloverStrategy$Builder(max="30", min="null", fileIndex="null", compressionLevel="null", ={}, stopCustomActionsOnError="null", tempCompressedFilePattern="null", Configuration(LogstashPropertiesConfig))
2022-06-27 10:18:16,559 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.RollingFileAppender].
2022-06-27 10:18:16,570 main DEBUG RollingFileAppender$Builder(fileName="/data01/logstash-7.11.2/logs/logstash-plain.log", filePattern="/data01/logstash-7.11.2/logs/logstash-plain-%d{yyyy-MM-dd}-%i.log.gz", append="null", locking="null", Policies(CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])), DefaultRolloverStrategy(DefaultRolloverStrategy(min=1, max=30, useMax=true)), advertise="null", advertiseUri="null", createOnDemand="null", filePermissions="null", fileOwner="null", fileGroup="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", PatternLayout([%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n), name="plain_rolling", Configuration(LogstashPropertiesConfig), ScriptFilter(filter_no_pipelined), ={})
2022-06-27 10:18:16,573 main DEBUG Returning file creation time for /data01/logstash-7.11.2/logs/logstash-plain.log
2022-06-27 10:18:16,573 main DEBUG Starting RollingFileManager /data01/logstash-7.11.2/logs/logstash-plain.log
2022-06-27 10:18:16,574 main DEBUG PluginManager 'FileConverter' found 2 plugins
2022-06-27 10:18:16,577 main DEBUG Setting prev file time to 2022-06-26T16:30:37.869+0530
2022-06-27 10:18:16,577 main DEBUG Initializing triggering policy CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])
2022-06-27 10:18:16,577 main DEBUG Initializing triggering policy TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)
2022-06-27 10:18:16,578 main DEBUG Initializing triggering policy SizeBasedTriggeringPolicy(size=104857600)
2022-06-27 10:18:16,578 main DEBUG Building Plugin[name=Route, class=org.apache.logging.log4j.core.appender.routing.Route].
2022-06-27 10:18:16,583 main DEBUG createRoute(ref="null", key="null", Node=Route)
2022-06-27 10:18:16,583 main DEBUG Building Plugin[name=Script, class=org.apache.logging.log4j.core.script.Script].
2022-06-27 10:18:16,584 main DEBUG createScript(name="routing_script", language="JavaScript", scriptText="logEvent.getContextData().containsKey("pipeline.id") ? logEvent.getContextData().getValue("pipeline.id") : "sink";")
2022-06-27 10:18:16,584 main DEBUG Building Plugin[name=Route, class=org.apache.logging.log4j.core.appender.routing.Route].
2022-06-27 10:18:16,584 main DEBUG createRoute(ref="null", key="sink", Node=Route)
2022-06-27 10:18:16,584 main DEBUG Building Plugin[name=Routes, class=org.apache.logging.log4j.core.appender.routing.Routes].
2022-06-27 10:18:16,585 main DEBUG Routes$Builder(Configuration(LogstashPropertiesConfig), pattern="null", Script(routing_script), ={Route(type=dynamic - type=Route default), Route(type=dynamic - type=Route key='sink')})
2022-06-27 10:18:16,590 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.routing.RoutingAppender].
2022-06-27 10:18:16,591 main DEBUG RoutingAppender$Builder(Script=null, Routes({Route(type=dynamic - type=Route default),Route(type=dynamic - type=Route key='sink')}), RewritePolicy=null, PurgePolicy=null, ignoreExceptions="null", Layout=null, name="pipeline_routing_appender", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,592 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.JsonLayout].
2022-06-27 10:18:16,599 main DEBUG JsonLayout$Builder(propertiesAsList="null", objectMessageAsJsonObject="null", ={}, eventEol="true", endOfLine="null", compact="true", complete="null", locationInfo="null", properties="null", includeStacktrace="null", stacktraceAsString="null", includeNullDelimiter="null", includeTimeMillis="null", ={}, charset="null", footerSerializer=null, headerSerializer=null, Configuration(LogstashPropertiesConfig), footer="null", header="null")
2022-06-27 10:18:16,750 main DEBUG Building Plugin[name=TimeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.TimeBasedTriggeringPolicy].
2022-06-27 10:18:16,750 main DEBUG TimeBasedTriggeringPolicy$Builder(interval="1", modulate="true", maxRandomDelay="null")
2022-06-27 10:18:16,751 main DEBUG Building Plugin[name=SizeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.SizeBasedTriggeringPolicy].
2022-06-27 10:18:16,751 main DEBUG createPolicy(size="100MB")
2022-06-27 10:18:16,751 main DEBUG Building Plugin[name=Policies, class=org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy].
2022-06-27 10:18:16,752 main DEBUG createPolicy(={TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)})
2022-06-27 10:18:16,752 main DEBUG Building Plugin[name=Script, class=org.apache.logging.log4j.core.script.Script].
2022-06-27 10:18:16,752 main DEBUG createScript(name="filter_no_pipelined", language="JavaScript", scriptText="false == false || !(logEvent.getContextData().containsKey("pipeline.id"))")
2022-06-27 10:18:16,752 main DEBUG Building Plugin[name=filter, class=org.apache.logging.log4j.core.filter.ScriptFilter].
2022-06-27 10:18:16,753 main DEBUG createFilter(Script(filter_no_pipelined), onMatch="null", onMismatch="null", Configuration(LogstashPropertiesConfig))
2022-06-27 10:18:16,757 main DEBUG Building Plugin[name=DefaultRolloverStrategy, class=org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy].
2022-06-27 10:18:16,763 main DEBUG DefaultRolloverStrategy$Builder(max="30", min="null", fileIndex="null", compressionLevel="null", ={}, stopCustomActionsOnError="null", tempCompressedFilePattern="null", Configuration(LogstashPropertiesConfig))
2022-06-27 10:18:16,763 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.RollingFileAppender].
2022-06-27 10:18:16,764 main DEBUG RollingFileAppender$Builder(fileName="/data01/logstash-7.11.2/logs/logstash-plain.log", filePattern="/data01/logstash-7.11.2/logs/logstash-plain-%d{yyyy-MM-dd}-%i.log.gz", append="null", locking="null", Policies(CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])), DefaultRolloverStrategy(DefaultRolloverStrategy(min=1, max=30, useMax=true)), advertise="null", advertiseUri="null", createOnDemand="null", filePermissions="null", fileOwner="null", fileGroup="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", JSONLayout(org.apache.logging.log4j.core.layout.JsonLayout@43b8d302), name="json_rolling", Configuration(LogstashPropertiesConfig), ScriptFilter(filter_no_pipelined), ={})
2022-06-27 10:18:16,764 main DEBUG PluginManager 'FileConverter' found 2 plugins
2022-06-27 10:18:16,765 main DEBUG Initializing triggering policy TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)
2022-06-27 10:18:16,765 main DEBUG Initializing triggering policy SizeBasedTriggeringPolicy(size=104857600)
2022-06-27 10:18:16,765 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
2022-06-27 10:18:16,766 main DEBUG PatternLayout$Builder(pattern="[%d{ISO8601}][%-5p][%-25c] %m%n", PatternSelector=null, Configuration(LogstashPropertiesConfig), Replace=null, charset="null", alwaysWriteExceptions="null", disableAnsi="null", noConsoleNoAnsi="null", header="null", footer="null")
2022-06-27 10:18:16,766 main DEBUG Building Plugin[name=TimeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.TimeBasedTriggeringPolicy].
2022-06-27 10:18:16,766 main DEBUG TimeBasedTriggeringPolicy$Builder(interval="1", modulate="true", maxRandomDelay="null")
2022-06-27 10:18:16,767 main DEBUG Building Plugin[name=SizeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.SizeBasedTriggeringPolicy].
2022-06-27 10:18:16,767 main DEBUG createPolicy(size="100MB")
2022-06-27 10:18:16,767 main DEBUG Building Plugin[name=Policies, class=org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy].
2022-06-27 10:18:16,767 main DEBUG createPolicy(={TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)})
2022-06-27 10:18:16,768 main DEBUG Building Plugin[name=DefaultRolloverStrategy, class=org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy].
2022-06-27 10:18:16,768 main DEBUG DefaultRolloverStrategy$Builder(max="30", min="null", fileIndex="null", compressionLevel="null", ={}, stopCustomActionsOnError="null", tempCompressedFilePattern="null", Configuration(LogstashPropertiesConfig))
2022-06-27 10:18:16,768 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.RollingFileAppender].
2022-06-27 10:18:16,769 main DEBUG RollingFileAppender$Builder(fileName="/data01/logstash-7.11.2/logs/logstash-slowlog-plain.log", filePattern="/data01/logstash-7.11.2/logs/logstash-slowlog-plain-%d{yyyy-MM-dd}-%i.log.gz", append="null", locking="null", Policies(CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])), DefaultRolloverStrategy(DefaultRolloverStrategy(min=1, max=30, useMax=true)), advertise="null", advertiseUri="null", createOnDemand="null", filePermissions="null", fileOwner="null", fileGroup="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", PatternLayout([%d{ISO8601}][%-5p][%-25c] %m%n), name="plain_rolling_slowlog", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,769 main DEBUG Returning file creation time for /data01/logstash-7.11.2/logs/logstash-slowlog-plain.log
2022-06-27 10:18:16,770 main DEBUG Starting RollingFileManager /data01/logstash-7.11.2/logs/logstash-slowlog-plain.log
2022-06-27 10:18:16,770 main DEBUG PluginManager 'FileConverter' found 2 plugins
2022-06-27 10:18:16,770 main DEBUG Setting prev file time to 2022-06-26T13:25:21.372+0530
2022-06-27 10:18:16,770 main DEBUG Initializing triggering policy CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])
2022-06-27 10:18:16,771 main DEBUG Initializing triggering policy TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)
2022-06-27 10:18:16,771 main DEBUG Initializing triggering policy SizeBasedTriggeringPolicy(size=104857600)
2022-06-27 10:18:16,771 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.JsonLayout].
2022-06-27 10:18:16,772 main DEBUG JsonLayout$Builder(propertiesAsList="null", objectMessageAsJsonObject="null", ={}, eventEol="true", endOfLine="null", compact="true", complete="null", locationInfo="null", properties="null", includeStacktrace="null", stacktraceAsString="null", includeNullDelimiter="null", includeTimeMillis="null", ={}, charset="null", footerSerializer=null, headerSerializer=null, Configuration(LogstashPropertiesConfig), footer="null", header="null")
2022-06-27 10:18:16,772 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.ConsoleAppender].
2022-06-27 10:18:16,772 main DEBUG ConsoleAppender$Builder(target="null", follow="null", direct="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", JSONLayout(org.apache.logging.log4j.core.layout.JsonLayout@4c489191), name="json_console", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,775 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
2022-06-27 10:18:16,775 main DEBUG PatternLayout$Builder(pattern="[%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n", PatternSelector=null, Configuration(LogstashPropertiesConfig), Replace=null, charset="null", alwaysWriteExceptions="null", disableAnsi="null", noConsoleNoAnsi="null", header="null", footer="null")
2022-06-27 10:18:16,775 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.ConsoleAppender].
2022-06-27 10:18:16,776 main DEBUG ConsoleAppender$Builder(target="null", follow="null", direct="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", PatternLayout([%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n), name="plain_console", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,777 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
2022-06-27 10:18:16,778 main DEBUG PatternLayout$Builder(pattern="[%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n", PatternSelector=null, Configuration(LogstashPropertiesConfig), Replace=null, charset="null", alwaysWriteExceptions="null", disableAnsi="null", noConsoleNoAnsi="null", header="null", footer="null")
2022-06-27 10:18:16,778 main DEBUG Building Plugin[name=TimeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.TimeBasedTriggeringPolicy].
2022-06-27 10:18:16,779 main DEBUG TimeBasedTriggeringPolicy$Builder(interval="1", modulate="true", maxRandomDelay="null")
2022-06-27 10:18:16,779 main DEBUG Building Plugin[name=SizeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.SizeBasedTriggeringPolicy].
2022-06-27 10:18:16,779 main DEBUG createPolicy(size="100MB")
2022-06-27 10:18:16,779 main DEBUG Building Plugin[name=Policies, class=org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy].
2022-06-27 10:18:16,780 main DEBUG createPolicy(={TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)})
2022-06-27 10:18:16,780 main DEBUG Building Plugin[name=DefaultRolloverStrategy, class=org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy].
2022-06-27 10:18:16,780 main DEBUG DefaultRolloverStrategy$Builder(max="30", min="null", fileIndex="null", compressionLevel="null", ={}, stopCustomActionsOnError="null", tempCompressedFilePattern="null", Configuration(LogstashPropertiesConfig))
2022-06-27 10:18:16,780 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.RollingFileAppender].
2022-06-27 10:18:16,792 main DEBUG RollingFileAppender$Builder(fileName="/data01/logstash-7.11.2/logs/logstash-deprecation.log", filePattern="/data01/logstash-7.11.2/logs/logstash-deprecation-%d{yyyy-MM-dd}-%i.log.gz", append="null", locking="null", Policies(CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])), DefaultRolloverStrategy(DefaultRolloverStrategy(min=1, max=30, useMax=true)), advertise="null", advertiseUri="null", createOnDemand="null", filePermissions="null", fileOwner="null", fileGroup="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", PatternLayout([%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n), name="deprecation_plain_rolling", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,792 main DEBUG Returning file creation time for /data01/logstash-7.11.2/logs/logstash-deprecation.log
2022-06-27 10:18:16,803 main DEBUG Starting RollingFileManager /data01/logstash-7.11.2/logs/logstash-deprecation.log
2022-06-27 10:18:16,803 main DEBUG PluginManager 'FileConverter' found 2 plugins
2022-06-27 10:18:16,803 main DEBUG Setting prev file time to 2022-06-26T15:13:03.956+0530
2022-06-27 10:18:16,804 main DEBUG Initializing triggering policy CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])
2022-06-27 10:18:16,804 main DEBUG Initializing triggering policy TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)
2022-06-27 10:18:16,804 main DEBUG Initializing triggering policy SizeBasedTriggeringPolicy(size=104857600)
2022-06-27 10:18:16,804 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
2022-06-27 10:18:16,805 main DEBUG PatternLayout$Builder(pattern="[%d{ISO8601}][%-5p][%-25c] %m%n", PatternSelector=null, Configuration(LogstashPropertiesConfig), Replace=null, charset="null", alwaysWriteExceptions="null", disableAnsi="null", noConsoleNoAnsi="null", header="null", footer="null")
2022-06-27 10:18:16,805 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.ConsoleAppender].
2022-06-27 10:18:16,805 main DEBUG ConsoleAppender$Builder(target="null", follow="null", direct="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", PatternLayout([%d{ISO8601}][%-5p][%-25c] %m%n), name="plain_console_slowlog", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,807 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.JsonLayout].
2022-06-27 10:18:16,807 main DEBUG JsonLayout$Builder(propertiesAsList="null", objectMessageAsJsonObject="null", ={}, eventEol="true", endOfLine="null", compact="true", complete="null", locationInfo="null", properties="null", includeStacktrace="null", stacktraceAsString="null", includeNullDelimiter="null", includeTimeMillis="null", ={}, charset="null", footerSerializer=null, headerSerializer=null, Configuration(LogstashPropertiesConfig), footer="null", header="null")
2022-06-27 10:18:16,808 main DEBUG Building Plugin[name=TimeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.TimeBasedTriggeringPolicy].
2022-06-27 10:18:16,808 main DEBUG TimeBasedTriggeringPolicy$Builder(interval="1", modulate="true", maxRandomDelay="null")
2022-06-27 10:18:16,808 main DEBUG Building Plugin[name=SizeBasedTriggeringPolicy, class=org.apache.logging.log4j.core.appender.rolling.SizeBasedTriggeringPolicy].
2022-06-27 10:18:16,808 main DEBUG createPolicy(size="100MB")
2022-06-27 10:18:16,809 main DEBUG Building Plugin[name=Policies, class=org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy].
2022-06-27 10:18:16,809 main DEBUG createPolicy(={TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)})
2022-06-27 10:18:16,809 main DEBUG Building Plugin[name=DefaultRolloverStrategy, class=org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy].
2022-06-27 10:18:16,809 main DEBUG DefaultRolloverStrategy$Builder(max="30", min="null", fileIndex="null", compressionLevel="null", ={}, stopCustomActionsOnError="null", tempCompressedFilePattern="null", Configuration(LogstashPropertiesConfig))
2022-06-27 10:18:16,809 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.RollingFileAppender].
2022-06-27 10:18:16,810 main DEBUG RollingFileAppender$Builder(fileName="/data01/logstash-7.11.2/logs/logstash-slowlog-plain.log", filePattern="/data01/logstash-7.11.2/logs/logstash-slowlog-plain-%d{yyyy-MM-dd}-%i.log.gz", append="null", locking="null", Policies(CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)])), DefaultRolloverStrategy(DefaultRolloverStrategy(min=1, max=30, useMax=true)), advertise="null", advertiseUri="null", createOnDemand="null", filePermissions="null", fileOwner="null", fileGroup="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", JSONLayout(org.apache.logging.log4j.core.layout.JsonLayout@66e56837), name="json_rolling_slowlog", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,810 main DEBUG PluginManager 'FileConverter' found 2 plugins
2022-06-27 10:18:16,810 main DEBUG Initializing triggering policy TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)
2022-06-27 10:18:16,811 main DEBUG Initializing triggering policy SizeBasedTriggeringPolicy(size=104857600)
2022-06-27 10:18:16,811 main DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.JsonLayout].
2022-06-27 10:18:16,811 main DEBUG JsonLayout$Builder(propertiesAsList="null", objectMessageAsJsonObject="null", ={}, eventEol="true", endOfLine="null", compact="true", complete="null", locationInfo="null", properties="null", includeStacktrace="null", stacktraceAsString="null", includeNullDelimiter="null", includeTimeMillis="null", ={}, charset="null", footerSerializer=null, headerSerializer=null, Configuration(LogstashPropertiesConfig), footer="null", header="null")
2022-06-27 10:18:16,812 main DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.ConsoleAppender].
2022-06-27 10:18:16,812 main DEBUG ConsoleAppender$Builder(target="null", follow="null", direct="null", bufferedIo="null", bufferSize="null", immediateFlush="null", ignoreExceptions="null", JSONLayout(org.apache.logging.log4j.core.layout.JsonLayout@38c9a48), name="json_console_slowlog", Configuration(LogstashPropertiesConfig), Filter=null, ={})
2022-06-27 10:18:16,813 main DEBUG Building Plugin[name=appenders, class=org.apache.logging.log4j.core.config.AppendersPlugin].
2022-06-27 10:18:16,825 main DEBUG createAppenders(={plain_rolling, pipeline_routing_appender, json_rolling, plain_rolling_slowlog, json_console, plain_console, deprecation_plain_rolling, plain_console_slowlog, json_rolling_slowlog, json_console_slowlog})
2022-06-27 10:18:16,825 main DEBUG Configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@65a03e98 initialized
2022-06-27 10:18:16,825 main DEBUG Starting configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@65a03e98
2022-06-27 10:18:16,826 main DEBUG Started configuration org.apache.logging.log4j.core.config.properties.PropertiesConfiguration@65a03e98 OK.
2022-06-27 10:18:16,826 main DEBUG Appender STDOUT stopped with status true
2022-06-27 10:18:16,832 main DEBUG Reconfiguration complete for context[name=277050dc] at URI /data01/logstash-7.11.2/config/log4j2.properties (org.apache.logging.log4j.core.LoggerContext@4cec414d) with optional ClassLoader: null
[2022-06-27T10:18:16,847][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.11.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
2022-06-27 10:18:16,854 main DEBUG Script filter_no_pipelined is compilable
2022-06-27 10:18:17,216 main DEBUG Formatting file name. useCurrentTime=false. currentFileTime=1656305297215, prevFileTime=1656181800000
2022-06-27 10:18:17,217 main DEBUG Current file: /data01/logstash-7.11.2/logs/logstash-plain.log
2022-06-27 10:18:17,218 main DEBUG Formatting file name. useCurrentTime=false. currentFileTime=1656305297215, prevFileTime=1656181800000
2022-06-27 10:18:17,220 main DEBUG OutputStream closed
2022-06-27 10:18:17,220 main DEBUG RollingFileManager executing synchronous FileRenameAction[/data01/logstash-7.11.2/logs/logstash-plain.log to /data01/logstash-7.11.2/logs/logstash-plain-2022-06-26-1.log, renameEmptyFiles=false]
2022-06-27 10:18:17,221 main DEBUG RollingFileManager executing async GzCompressAction[/data01/logstash-7.11.2/logs/logstash-plain-2022-06-26-1.log to /data01/logstash-7.11.2/logs/logstash-plain-2022-06-26-1.log.gz, deleteSource=true]
2022-06-27 10:18:17,244 main DEBUG Now writing to /data01/logstash-7.11.2/logs/logstash-plain.log at 2022-06-27T10:18:17.243+0530
2022-06-27 10:18:17,246 main DEBUG AsyncLogger.ThreadNameStrategy=UNCACHED (user specified null, default is UNCACHED)
2022-06-27 10:18:17,246 main DEBUG org.apache.logging.log4j.core.util.SystemClock supports precise timestamps.
[2022-06-27T10:18:17,969][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-06-27T10:18:20,933][INFO ][org.reflections.Reflections] Reflections took 44 ms to scan 1 urls, producing 23 keys and 47 values
2022-06-27 10:18:20,937 Converge PipelineAction::Create<main> DEBUG Script filter_no_pipelined is compilable
[2022-06-27T10:18:21,590][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/data01/logstash-7.11.2/bin/Sprint_Server_Logs.conf"], :thread=>"#<Thread:0xc5749c2 run>"}
2022-06-27 10:18:21,600 [main]-pipeline-manager DEBUG Script filter_no_pipelined is compilable
[2022-06-27T10:18:22,575][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.95}
[2022-06-27T10:18:22,795][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-06-27T10:18:22,833][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
2022-06-27 10:18:22,835 Agent thread DEBUG Script filter_no_pipelined is compilable
[2022-06-27T10:18:22,901][INFO ][filewatch.observingtail  ][main][430c23d53546d1ffb13ff11a12a75c033490b9a409a49f06d1d3abe5e34667cb] START, creating Discoverer, Watch with file and sincedb collections
2022-06-27 10:18:22,906 [main]<file DEBUG Script filter_no_pipelined is compilable
[2022-06-27T10:18:23,205][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
2022-06-27 10:18:23,208 Api Webserver DEBUG Script filter_no_pipelined is compilable
4

You have multiline perhaps that is not working, they are tricky..

I would take it out and see if logstash sends messages... Then fix the multi line if that is the case.

You also have ignore older. I would take all of that out. Start simple then add back.

Perhaps you should provide a couple samples of your data otherwise it's hard to help.

5

Here's the sample log O/P to be picked up by logstash.

2021-04-08 02:00:21,781 [ERROR] [pool-41-thread-2] [ServiceImpl:64] Error while executing service org.springframework.transaction.UnexpectedRollbackException: Transaction silently rolled back because it has been marked as rollback-only
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:753)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:712)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:631)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:385)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:99)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)

Actions taken:-
Tried running the logstash after removing the codec but still the same behavior.
Tried taking only console o/p but still nothing worked.

input {
  file 
  {
	path => "/data01/ApplicationLogs/services.log"
	start_position => "beginning"
	sincedb_path => "/dev/null"
	ignore_older => 1
	type => "application_dev"
   }
}
filter {
  grok {
     	match => { "message" => "(?m)^%{TIMESTAMP_ISO8601:Log_Timestamp},%{NUMBER:No} \[(?<LogSeverity>(INFO|DEBUG|WARNING|SEVERE|ERROR|WARN|FAIL|APEK))\] \[%{GREEDYDATA:WorkerID}\] \[%{GREEDYDATA:ClassName}:%{NUMBER:LineNo}\] %{GREEDYDATA:LogMessage}"}
	} 
}

output {
  stdout {
  codec => json 
  }
}

Do i need to change the ignore_folder and type configs under file?

6

That will cause any file that was modified more than one second ago to be ignored. If you are trying to read log messages from over a year ago (which is the example you showed) then that is probably not what you want.

With the default value for stat_interval (also one second) this will cause some files that are actively being updated to be ignored as well, because stat_interval => 1 does not mean that stat is called once per second, it means that after doing all the file processing currently required, there will be a sleep for one second before stat is called again. Any file that is updated while the file input is reading data rather than sleeping could get ignored.

1 Like
7

@Badger
I tried updating the config with what you said but still can't see the index.

Any information on other areas to check will be helpful.

8

Any leads on this would be helpful, i am still not able to get the index on dashboard.
Just to update, my logstash.yml only has data for below keys:-
node.name:
path.data:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.