Kibana (7.11.1) isn't showing logs

AnneHermann · March 25, 2021, 5:57pm

Hi,

I have an index, that is containing log data, but the data can't be seen under Observability -> Logs. I have also created an index pattern and choose a time field.
I adjusted the settings of Observability -> Logs to "Log indices: {my-index-name}*."
But I just get " There are no log messages to display. Try adjusting your filter."
Apparently there is an Internal Server Error with the following message:

[query_shard_exception] No mapping found for [@timestamp] in order to sort on, with { index_uuid="BhrWkvASSDakWeMZJByc6w" & index="scs-client-log" }

The name of the time field is in the index pattern is "timeStamp". Is there a problem because of the "@"?

stephenb · March 25, 2021, 6:16pm

Hi @AnneHermann Welcome to the community.

Also I am not sure where that screenshot is from, a little more context helps us help you. Which App / Visualization is that?

When you created the index pattern what did you select at the Time field?

If you you are not using the default @timestamp then you should select the field that your timestamp is in

AnneHermann · March 26, 2021, 9:02am

Thanks for the quick reply

In the screenshot above you can see my time field in my index pattern.

I have chosen "timeStamp" as my time field when creating the index pattern:

stephenb · March 26, 2021, 2:23pm

First just to be clear so I don't make assumptions when you showed setting the indices in log viewer you showed

{my-index-name}*.

You should put in the index pattern name like

scs-client-logs-*

2nd thing I notice is that you have a field

properties.@timestamp

Which is highly unusual so let's back up a bit and take a look perhaps at the indices and mapping.

In Kibana / Dev Tools can you run

GET _cat/indices?v

And post the results?

If there are indices with documents can you post

GET /<indexname>/

That should show the mapping.

Finally you could also post

GET /<indexname>/_search

And post a couple docs

This should help us diagnos

AnneHermann · March 26, 2021, 5:02pm

GET _cat/indices?v

GET /scs-client-log/

{
  "scs-client-log" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "className" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "exception" : {
          "type" : "object"
        },
        "fileName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "fix" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "fullInfo" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "hostName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "level" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "lineNumber" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "messageObject" : {
          "type" : "object"
        },
        "methodName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "properties" : {
          "properties" : {
            "@timestamp" : {
              "type" : "date"
            },
            "log4net:HostName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "tab" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "timeStamp" : {
          "type" : "date"
        }
      }
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "scs-client-log",
        "creation_date" : "1614264757339",
        "number_of_replicas" : "1",
        "uuid" : "BhrWkvASSDakWeMZJByc6w",
        "version" : {
          "created" : "7110199"
        }
      }
    }
  }
}

GET /scs-client-log/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1317,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "scs-client-log",
        "_type" : "logEvent",
        "_id" : "ySWv2XcBYgpFxvFAgAvO",
        "_score" : 1.0,
        "_source" : {
          "timeStamp" : "2021-02-25T11:55:33.3922395Z",
          "message" : "SCS.Client (3.4.0.297) - LogManager initialisiert",
          "messageObject" : { },
          "exception" : { },
          "loggerName" : null,
          "domain" : null,
          "identity" : null,
          "level" : "INFO",
          "className" : "LogManager",
          "fileName" : "",
          "lineNumber" : "0",
          "fullInfo" : "LogManager.(:0)",
          "methodName" : "",
          "fix" : "All",
          "properties" : {
            "log4net:UserName" : null,
            "log4net:Identity" : null,
            "tab" : """	""",
            "log4net:HostName" : "DE106YDX",
            "@timestamp" : "2021-02-25T11:55:33.3922395Z"
          },
          "userName" : null,
          "threadName" : null,
          "hostName" : "DE106YDX"
        }
      },

I hope that this helps. I've tried using "scs-client-logs-*" in the index pattern, but there is the same error.

stephenb · March 26, 2021, 8:31pm

In general looks pretty good, there are a few things I might fix, but those should not keep you from seeing the logs. We can come back to those.

Lets try this.

Go delete the existing index pattern.

Recreate the index pattern for now just use scs-client-log no * or anything.

Use the timeStamp field as the time field.

Go to Discover

Select index pattern scs-client-log and in the time picker select last 90 days and lets see what you get...

AnneHermann · March 29, 2021, 5:27am

The log data appears under Discover, but not under Logs.

stephenb · March 29, 2021, 2:30pm

Hi @AnneHermann

I should have recognized this earlier, but it has been recently deprecated. The logviewier uses ECS (Elastic Common Schema) fields so it is based on the @timestamp field which is the correct ECS field.

In the Logs / Settings you will see this

What we really need to do is fix your timestamp in the ingest process.

This is what I was referring

I just did not realize we have deprecated setting in the log viewer, so it needs to be fixed first.

You want the either rename this field to @timestamp (yes the @ is correct)

or move this field to the root level

Get that fixed and delete and recreate the index patterns and you should be good to go.

and of course you need to add your index pattern here in the logs settings

AnneHermann · March 29, 2021, 4:40pm

okay and we have no other option because the function you mentioned is deprecated? I have to change the name of the timeStamp field (or move the other to the root level)?
Because I am using Log4Net.ElasticSearch to log messages to the ElasticSearch document database and I think the name of the field is set in this library.

stephenb · March 29, 2021, 4:53pm

Yes that is correct, and what you really should do anyways.

Best practice is the timestamp in the @timestamp field in the root level, that complies with ECS

How are you ingesting the data? Logstash? Beats?

Let me know if you need help with that.

stephenb · March 29, 2021, 11:49pm

@AnneHermann

Sorry I just saw, that is the format from Log4Net.ElasticSearch someone should really write an issue to get that fixed @timestamp should be the default timestamp field It also looks that this logger is becoming legacy it is still setting _type which legacy / deprecated and will not longer be supported going forward.

This is deprecated see here

        "_type" : "logEvent",

I believe there is an ECS Compliant Logger Here

But that may not be an option.

So what you can do is use an ingest pipeline to rename the field and we can get ride of that _type too

PUT _ingest/pipeline/dotnet-logger-pipeline
{
  "processors": [
    {
      "rename": {
        "field": "timeStamp",
        "target_field": "@timestamp"
      }
    },
    {
      "remove": {
        "field": "_type"
      }
    }
  ]
}

You can create a _template and name that pipeline as the index default pipeline

PUT _template/scs-client-log
{
  "index_patterns": [
    "scs-client-log*"
  ],
  "settings": {
    "default_pipeline": "dotnet-logger-pipeline"
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
     ..... <- The rest of your mapping 
    }
  }
}

Then on ingest it should remove the _type and rename the timeStamp field to @timestamp

AnneHermann · March 31, 2021, 2:16pm

It works now! Thank you so much for your help and your patience.

Now I wanted to try "Anomaly detection with Machine Learning" (Observability -> Logs -> Anomalies) and I got a new problem:

I get the impression that using Log4net.ElasticSearch leads to a couple of problems. I will take a look on the ECS Compliant Logger.

stephenb · March 31, 2021, 2:39pm

Cool,

Yes that is an issue too....

I would look at the ECS logger

event.dataset is required but can you fix that reql quick by adding

event.datasetas a keyword type in your mapping and then adding a set processor

{
  "set": {
    "field": "event.dataset",
    "value": "scs-log"
  }
}

that will probably fix that...

stephenb · March 31, 2021, 3:11pm

Hi @AnneHermann

I just found this, not sure why I did not see it earlier. I think we just released some new docs!

Hope this helps!

system · April 28, 2021, 3:12pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana not showing the data due to time stamp field issue Kibana	9	6294	July 6, 2017
Logstash Log Stream Kibana	4	3680	April 7, 2022
Kibana graph now showing on a selected Index Kibana	13	448	August 4, 2020
Default Index pattern not found? Kibana	10	2702	June 15, 2018
Logs are indexed into logstash indices but not showing in kibana dashboard? Kibana	9	4158	August 19, 2020

Kibana (7.11.1) isn't showing logs

Related topics