Fields not coming in discover

Shubham_Singh · March 31, 2022, 10:27am

I have a stack_trace field in log file but in kibana it turn out to be in hidden field with no value. Rest other fields are coming as expected. I can see the field in vizualization but not in discover . The field type is keyword for strack_trace field

{"@timestamp":"2022-03-31T09:39:59.185+00:00","@version":1,"message":"Exception:","logger_name":"com.nethum.errorhandling.exception.handler.RestResponseEntityExceptionHandler","thread_name":"http-nio-10020-exec-1","level":"ERROR","level_value":40000,"stack_trace":"java.lang.NullPointerException: null\n\tat com.engati.livechat.v2.controller.DevOpsController.test(DevOpsController.java:21)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:635)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:155)\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:123)\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.lang.Thread.run(Thread.java:748)\n","app_name":"livechat-service","log_type":"app"}

majagrubic · March 31, 2022, 5:25pm

Hi,
Could you please paste a request and response from the Inspect tab in Discover, as well as the name of the field you're unable to see in Discover?

Shubham_Singh · April 1, 2022, 4:50am

I can see the filed when i uncheck hidden fields in discover. The logs are coming in json but the fields seems to be empty. Let me know if any changes required to fix this

majagrubic · April 1, 2022, 6:47am

Sorry, I meant - can you paste the actual request from "Request" tab and the actual response from the "Response" tab?

Hidden field means just what that second popup say - the field IS present in your Elasticsearch mapping, but not in the documents you requested, so Discover will hide it by default.

Shubham_Singh · April 5, 2022, 6:30am

Request
{
"track_total_hits": true,
"size": 1000,
"sort": [
{
"@timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"version": true,
"fields": [
{
"field": "",
"include_unmapped": "true"
},
{
"field": "@timestamp",
"format": "strict_date_optional_time"
},
{
"field": "aws.cloudtrail.digest.end_time",
"format": "strict_date_optional_time"
},
{
"field": "aws.cloudtrail.digest.newest_event_time",
"format": "strict_date_optional_time"
},
{
"field": "aws.cloudtrail.digest.oldest_event_time",
"format": "strict_date_optional_time"
},
{
"field": "aws.cloudtrail.digest.start_time",
"format": "strict_date_optional_time"
},
{
"field": "aws.cloudtrail.user_identity.session_context.creation_date",
"format": "strict_date_optional_time"
},
{
"field": "azure.auditlogs.properties.activity_datetime",
"format": "strict_date_optional_time"
},
{
"field": "azure.enqueued_time",
"format": "strict_date_optional_time"
},
{
"field": "azure.signinlogs.properties.created_at",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.agentReceiptTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.deviceCustomDate1",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.deviceCustomDate2",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.deviceReceiptTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.endTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.fileCreateTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.fileModificationTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.flexDate1",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.managerReceiptTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.oldFileCreateTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.oldFileModificationTime",
"format": "strict_date_optional_time"
},
{
"field": "cef.extensions.startTime",
"format": "strict_date_optional_time"
},
{
"field": "checkpoint.subs_exp",
"format": "strict_date_optional_time"
},
{
"field": "cisco.amp.threat_hunting.incident_end_time",
"format": "strict_date_optional_time"
},
{
"field": "cisco.amp.threat_hunting.incident_start_time",
"format": "strict_date_optional_time"
},
{
"field": "cisco.amp.timestamp_nanoseconds",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.EndTimestamp",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.IncidentEndTime",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.IncidentStartTime",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.ProcessEndTime",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.ProcessStartTime",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.StartTimestamp",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.Timestamp",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.event.UTCTimestamp",
"format": "strict_date_optional_time"
},
{
"field": "crowdstrike.metadata.eventCreationTime",
"format": "strict_date_optional_time"
},
{
"field": "cyberarkpas.audit.iso_timestamp",
"format": "strict_date_optional_time"
},
{
"field": "event.created",
"format": "strict_date_optional_time"
},
{
"field": "event.end",
"format": "strict_date_optional_time"
},
{
"field": "event.ingested",
"format": "strict_date_optional_time"
},
{
"field": "event.start",
"format": "strict_date_optional_time"
},
{
"field": "file.accessed",
"format": "strict_date_optional_time"
},
{
"field": "file.created",
"format": "strict_date_optional_time"
},
{
"field": "file.ctime",
"format": "strict_date_optional_time"
},
{
"field": "file.mtime",
"format": "strict_date_optional_time"
},
{
"field": "file.x509.not_after",
"format": "strict_date_optional_time"
},
{
"field": "file.x509.not_before",
"format": "strict_date_optional_time"
},
{
"field": "google_workspace.admin.email.log_search_filter.end_date",
"format": "strict_date_optional_time"
},
{
"field": "google_workspace.admin.email.log_search_filter.start_date",
"format": "strict_date_optional_time"
},
{
"field": "google_workspace.admin.user.birthdate",
"format": "strict_date_optional_time"
},
{
"field": "gsuite.admin.email.log_search_filter.end_date",
"format": "strict_date_optional_time"
},
{
"field": "gsuite.admin.email.log_search_filter.start_date",
"format": "strict_date_optional_time"
},
{
"field": "gsuite.admin.user.birthdate",
"format": "strict_date_optional_time"
},
{
"field": "juniper.srx.elapsed_time",
"format": "strict_date_optional_time"
},
{
"field": "juniper.srx.epoch_time",
"format": "strict_date_optional_time"
},
{
"field": "juniper.srx.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "kafka.block_timestamp",
"format": "strict_date_optional_time"
},
{
"field": "microsoft.defender_atp.lastUpdateTime",
"format": "strict_date_optional_time"
},
{
"field": "microsoft.defender_atp.resolvedTime",
"format": "strict_date_optional_time"
},
{
"field": "microsoft.m365_defender.alerts.creationTime",
"format": "strict_date_optional_time"
},
{
"field": "microsoft.m365_defender.alerts.lastUpdatedTime",
"format": "strict_date_optional_time"
},
{
"field": "microsoft.m365_defender.alerts.resolvedTime",
"format": "strict_date_optional_time"
},
{
"field": "misp.campaign.first_seen",
"format": "strict_date_optional_time"
},
{
"field": "misp.campaign.last_seen",
"format": "strict_date_optional_time"
},
{
"field": "misp.intrusion_set.first_seen",
"format": "strict_date_optional_time"
},
{
"field": "misp.intrusion_set.last_seen",
"format": "strict_date_optional_time"
},
{
"field": "misp.observed_data.first_observed",
"format": "strict_date_optional_time"
},
{
"field": "misp.observed_data.last_observed",
"format": "strict_date_optional_time"
},
{
"field": "misp.report.published",
"format": "strict_date_optional_time"
},
{
"field": "misp.threat_indicator.valid_from",
"format": "strict_date_optional_time"
},
{
"field": "misp.threat_indicator.valid_until",
"format": "strict_date_optional_time"
},
{
"field": "netflow.collection_time_milliseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.exporter.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_end_microseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_end_milliseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_end_nanoseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_end_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_start_microseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_start_milliseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_start_nanoseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.flow_start_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.max_export_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.max_flow_end_microseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.max_flow_end_milliseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.max_flow_end_nanoseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.max_flow_end_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.min_export_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.min_flow_start_microseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.min_flow_start_milliseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.min_flow_start_nanoseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.min_flow_start_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.monitoring_interval_end_milli_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.monitoring_interval_start_milli_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.observation_time_microseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.observation_time_milliseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.observation_time_nanoseconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.observation_time_seconds",
"format": "strict_date_optional_time"
},
{
"field": "netflow.system_init_time_milliseconds",
"format": "strict_date_optional_time"
},
{
"field": "okta.debug_context.debug_data.suspicious_activity.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "package.installed",
"format": "strict_date_optional_time"
},
{
"field": "panw.panos.factorcompletiontime",
"format": "strict_date_optional_time"
},
{
"field": "pensando.dfw.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "postgresql.log.session_start_time",
"format": "strict_date_optional_time"
},
{
"field": "process.parent.start",
"format": "strict_date_optional_time"
},
{
"field": "process.start",
"format": "strict_date_optional_time"
},
{
"field": "rsa.internal.lc_ctime",
"format": "strict_date_optional_time"
},
{
"field": "rsa.internal.time",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.effective_time",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.endtime",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.event_queue_time",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.event_time",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.expire_time",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.recorded_time",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.stamp",
"format": "strict_date_optional_time"
},
{
"field": "rsa.time.starttime",
"format": "strict_date_optional_time"
},
{
"field": "snyk.vulnerabilities.disclosure_time",
"format": "strict_date_optional_time"
},
{
"field": "snyk.vulnerabilities.introduced_date",
"format": "strict_date_optional_time"
},
{
"field": "snyk.vulnerabilities.publication_time",
"format": "strict_date_optional_time"
},
{
"field": "sophos.xg.date",
"format": "strict_date_optional_time"
},
{
"field": "sophos.xg.eventtime",
"format": "strict_date_optional_time"
},
{
"field": "sophos.xg.start_time",
"format": "strict_date_optional_time"
},
{
"field": "sophos.xg.starttime",
"format": "strict_date_optional_time"
},
{
"field": "sophos.xg.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "suricata.eve.alert.created_at",
"format": "strict_date_optional_time"
},
{
"field": "suricata.eve.alert.updated_at",
"format": "strict_date_optional_time"
},
{
"field": "suricata.eve.flow.start",
"format": "strict_date_optional_time"
},
{
"field": "suricata.eve.tls.notafter",
"format": "strict_date_optional_time"
},
{
"field": "suricata.eve.tls.notbefore",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.anomali.modified",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.anomali.valid_from",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.indicator.first_seen",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.indicator.last_seen",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.misp.attribute.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.misp.date",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.misp.publish_timestamp",
"format": "strict_date_optional_time"
},
{
"field": "threatintel.misp.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "tls.client.not_after",
"format": "strict_date_optional_time"
},
{
"field": "tls.client.not_before",
"format": "strict_date_optional_time"
},
{
"field": "tls.client.x509.not_after",
"format": "strict_date_optional_time"
},
{
"field": "tls.client.x509.not_before",
"format": "strict_date_optional_time"
},
{
"field": "tls.server.not_after",
"format": "strict_date_optional_time"
},
{
"field": "tls.server.not_before",
"format": "strict_date_optional_time"
},
{
"field": "tls.server.x509.not_after",
"format": "strict_date_optional_time"
},
{
"field": "tls.server.x509.not_before",
"format": "strict_date_optional_time"
},
{
"field": "x509.not_after",
"format": "strict_date_optional_time"
},
{
"field": "x509.not_before",
"format": "strict_date_optional_time"
},
{
"field": "zeek.kerberos.valid.from",
"format": "strict_date_optional_time"
},
{
"field": "zeek.kerberos.valid.until",
"format": "strict_date_optional_time"
},
{
"field": "zeek.ntp.org_time",
"format": "strict_date_optional_time"
},
{
"field": "zeek.ntp.rec_time",
"format": "strict_date_optional_time"
},
{
"field": "zeek.ntp.ref_time",
"format": "strict_date_optional_time"
},
{
"field": "zeek.ntp.xmt_time",
"format": "strict_date_optional_time"
},
{
"field": "zeek.ocsp.revoke.time",
"format": "strict_date_optional_time"
},
{
"field": "zeek.ocsp.update.next",
"format": "strict_date_optional_time"
},
{
"field": "zeek.ocsp.update.this",
"format": "strict_date_optional_time"
},
{
"field": "zeek.pe.compile_time",
"format": "strict_date_optional_time"
},
{
"field": "zeek.smb_files.times.accessed",
"format": "strict_date_optional_time"
},
{
"field": "zeek.smb_files.times.changed",
"format": "strict_date_optional_time"
},
{
"field": "zeek.smb_files.times.created",
"format": "strict_date_optional_time"
},
{
"field": "zeek.smb_files.times.modified",
"format": "strict_date_optional_time"
},
{
"field": "zeek.smtp.date",
"format": "strict_date_optional_time"
},
{
"field": "zeek.snmp.up_since",
"format": "strict_date_optional_time"
},
{
"field": "zeek.x509.certificate.valid.from",
"format": "strict_date_optional_time"
},
{
"field": "zeek.x509.certificate.valid.until",
"format": "strict_date_optional_time"
},
{
"field": "zoom.meeting.start_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.participant.join_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.participant.leave_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.phone.answer_start_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.phone.call_end_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.phone.connected_start_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.phone.date_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.phone.ringing_start_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.recording.recording_file.recording_end",
"format": "strict_date_optional_time"
},
{
"field": "zoom.recording.recording_file.recording_start",
"format": "strict_date_optional_time"
},
{
"field": "zoom.recording.start_time",
"format": "strict_date_optional_time"
},
{
"field": "zoom.timestamp",
"format": "strict_date_optional_time"
},
{
"field": "zoom.webinar.start_time",
"format": "strict_date_optional_time"
}
],
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"fixed_interval": "30s",
"time_zone": "UTC",
"min_doc_count": 1
}
}
},
"script_fields": {},
"stored_fields": [
""
],
"runtime_mappings": {},
"_source": false,
"query": {
"bool": {
"must": ,
"filter": [
{
"range": {
"@timestamp": {
"gte": "2022-04-05T06:11:59.903Z",
"lte": "2022-04-05T06:26:59.903Z",
"format": "strict_date_optional_time"
}
}
}
],
"should": ,
"must_not":
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"fragment_size": 2147483647
}
}

majagrubic · April 5, 2022, 7:13am

You haven't pasted the response, so I can't be exactly sure, but it looks like stack_trace field is not being requested. Most likely this is due to the fact that it is an unmapped field in your ES index AND you have field filters on your index pattern in Kibana enabled. To overcome this, you have a few options. Either:

remove field filters from your index pattern
update the mapping of your ES index and make stack_trace a mapped field (this is the preferred way of overcoming this)
in Kibana Advanced Settings, under Discover, enable readFieldsFromSource

Shubham_Singh · April 7, 2022, 4:33am

I Have updated my fields.yml from default to only required fields

key: mybeat
title: mybeat
description: These are the fields used by mybeat.
fields:
- name: message
  type: keyword
  required: true
  description: The actual message name.
- name: @timestamp
  type: date
  required: true
  description: The timestamp field.
- name: @version
  type: number
  required: false
  description: Comment made by the user.
- name: logger_name
  type: text
  required: true
  description: Comment made by the user.
- name: thread_name
  type: text
  required: true
  description: Comment made by the user.
- name: level
  type: text
  required: false
  description: Comment made by the user.
- name: level_value
  type: number
  required: true
  description: Comment made by the user.
- name: app_name
  type: keyword
  required: true
  description: Comment made by the user.
- name: log_type
  type: text
  required: true
  description: Comment made by the user.
- name: stack_trace
  type: text
  required: true
  description: Comment made by the user.

My filebeat config -
filebeat.inputs:

type: log
enabled: true
paths:
- /var/log/engati//.log
- /home/centos/standalone_apps//logs/.log
input_type: log
json.keys_under_root: true
json.add_error_key: true
json.message_key: message
reload.enabled: false
json.overwrite_keys: true
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
setup.ilm.enabled: false
setup.template.name: "engati-appserver111"
setup.template.pattern: "engati-appserver111*"
setup.template.fields: "fields.yml"
setup.template.enabled: true
setup.template.overwrite: true
http.enabled: true
http.host: appserver11.dev.engati.local
output.Elasticsearch:
hosts: ["elk.dev.engati.local:9200"]
index: "test-%{+yyyy.MM.dd}"

I deleted the older index template and tried to start filebeat but new index template and index is not created. Could you tell me what is wrong in my config

majagrubic · April 7, 2022, 6:11am

I am not super familar with filebeats, for that it's best to ask in the Beats forum.

system · May 5, 2022, 6:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Fields found in discover not found when creating a visualize Kibana	3	826	July 26, 2017
Field Exist in Index Pattern but didnt show in Discover Kibana	2	1370	January 9, 2019
Field is present but missing Logstash	4	367	November 14, 2018
Fields are not shown in "Available fields" in KIbana Logstash	8	21969	May 5, 2020
Field issue Kibana	8	287	May 21, 2018

Fields not coming in discover

Related Topics