Some logs are missing in elk

[Afrin]

Hi,

The problem we are facing is some logs are missing in elk while few are available. We cannot find the root cause for this issue because the logs which are missing and those which are available look similar. We cannot see any errors(only warnings and those warning logs are also available) in logstash logs too. We are pretty sure that logs are not entering into elk machine itself as we queried in elasticsearch. We also tried changing the output of logstash to file and checked there, but we cannot find the missing logs there too. Logstash seems to be in running state but we cannot get all logs.

Below are the configurations we use in both production elk as well as non-production elk (prod elk and non-prod elk has set up in different machine). In prod it works fine but in non-prod we are facing this issue. In production there are only 4-5 applns running whereas in non-prod there are about 5 different environment and each environment contains 10-13 applns.

logstash version"=>"6.8.7"
elasticsearch "6.8.7",
Kibana 6.8.7

logback.xml

    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
	
	<destination>${LOGSTASH_DESTINATION}</destination> <!-- ip:port -->
	<encoder
		class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
		<providers>
			<mdc /> 		<!-- MDC variables on the Thread will be written as JSON fields -->
			<context /> 	<!--Outputs entries from logback's context -->
			<version /> 	<!-- Logstash json format version, the @version field in the output -->
			<logLevel />
			<loggerName />
			<threadName />
			<pattern>
				<pattern>
					{
					"indexName": "xxx"   <!-- dev/qa/uat/preuat-->
					}
				</pattern>
			</pattern>
			<message />
			<logstashMarkers />
			<arguments />
			<stackTrace />
		</providers>
	</encoder>
</appender>
<root level="INFO">
    <appender-ref ref="LOGSTASH" />
</root>

logstash conf
input{
tcp {
port => 5000 codec => json
}
}
output{
elasticsearch {
hosts => {"localhost:5000"} index => "yy-%{indexName}"
}
}