Kibana (7.11.1) isn't showing logs

Hi,

I have an index, that is containing log data, but the data can't be seen under Observability -> Logs. I have also created an index pattern and choose a time field.
I adjusted the settings of Observability -> Logs to "Log indices: {my-index-name}*."
But I just get " There are no log messages to display. Try adjusting your filter."
Apparently there is an Internal Server Error with the following message:

[query_shard_exception] No mapping found for [@timestamp] in order to sort on, with { index_uuid="BhrWkvASSDakWeMZJByc6w" & index="scs-client-log" }

The name of the time field is in the index pattern is "timeStamp". Is there a problem because of the "@"?

Hi @AnneHermann Welcome to the community.

Also I am not sure where that screenshot is from, a little more context helps us help you. :slight_smile: Which App / Visualization is that?

When you created the index pattern what did you select at the Time field?

If you you are not using the default @timestamp then you should select the field that your timestamp is in

Thanks for the quick reply :slight_smile:

In the screenshot above you can see my time field in my index pattern.

I have chosen "timeStamp" as my time field when creating the index pattern:

First just to be clear so I don't make assumptions when you showed setting the indices in log viewer you showed

{my-index-name}*.

You should put in the index pattern name like

scs-client-logs-*

2nd thing I notice is that you have a field

properties.@timestamp

Which is highly unusual so let's back up a bit and take a look perhaps at the indices and mapping.

In Kibana / Dev Tools can you run

GET _cat/indices?v

And post the results?

If there are indices with documents can you post

GET /<indexname>/

That should show the mapping.

Finally you could also post

GET /<indexname>/_search

And post a couple docs

This should help us diagnos

GET _cat/indices?v

GET /scs-client-log/
{
  "scs-client-log" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "className" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "exception" : {
          "type" : "object"
        },
        "fileName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "fix" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "fullInfo" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "hostName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "level" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "lineNumber" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "messageObject" : {
          "type" : "object"
        },
        "methodName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "properties" : {
          "properties" : {
            "@timestamp" : {
              "type" : "date"
            },
            "log4net:HostName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "tab" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "timeStamp" : {
          "type" : "date"
        }
      }
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "scs-client-log",
        "creation_date" : "1614264757339",
        "number_of_replicas" : "1",
        "uuid" : "BhrWkvASSDakWeMZJByc6w",
        "version" : {
          "created" : "7110199"
        }
      }
    }
  }
}
GET /scs-client-log/_search
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1317,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "scs-client-log",
        "_type" : "logEvent",
        "_id" : "ySWv2XcBYgpFxvFAgAvO",
        "_score" : 1.0,
        "_source" : {
          "timeStamp" : "2021-02-25T11:55:33.3922395Z",
          "message" : "SCS.Client (3.4.0.297) - LogManager initialisiert",
          "messageObject" : { },
          "exception" : { },
          "loggerName" : null,
          "domain" : null,
          "identity" : null,
          "level" : "INFO",
          "className" : "LogManager",
          "fileName" : "",
          "lineNumber" : "0",
          "fullInfo" : "LogManager.(:0)",
          "methodName" : "",
          "fix" : "All",
          "properties" : {
            "log4net:UserName" : null,
            "log4net:Identity" : null,
            "tab" : """	""",
            "log4net:HostName" : "DE106YDX",
            "@timestamp" : "2021-02-25T11:55:33.3922395Z"
          },
          "userName" : null,
          "threadName" : null,
          "hostName" : "DE106YDX"
        }
      },

I hope that this helps. I've tried using "scs-client-logs-*" in the index pattern, but there is the same error.

In general looks pretty good, there are a few things I might fix, but those should not keep you from seeing the logs. We can come back to those.

Lets try this.

Go delete the existing index pattern.

Recreate the index pattern for now just use scs-client-log no * or anything.

Use the timeStamp field as the time field.

Go to Discover

Select index pattern scs-client-log and in the time picker select last 90 days and lets see what you get...

The log data appears under Discover, but not under Logs.

Hi @AnneHermann

I should have recognized this earlier, but it has been recently deprecated. The logviewier uses ECS (Elastic Common Schema) fields so it is based on the @timestamp field which is the correct ECS field.

In the Logs / Settings you will see this

What we really need to do is fix your timestamp in the ingest process.

This is what I was referring

I just did not realize we have deprecated setting in the log viewer, so it needs to be fixed first.

You want the either rename this field to @timestamp (yes the @ is correct)

or move this field to the root level

Get that fixed and delete and recreate the index patterns and you should be good to go.

and of course you need to add your index pattern here in the logs settings

okay and we have no other option because the function you mentioned is deprecated? I have to change the name of the timeStamp field (or move the other to the root level)?
Because I am using Log4Net.ElasticSearch to log messages to the ElasticSearch document database and I think the name of the field is set in this library.

Yes that is correct, and what you really should do anyways.

Best practice is the timestamp in the @timestamp field in the root level, that complies with ECS

How are you ingesting the data? Logstash? Beats?

Let me know if you need help with that.

@AnneHermann

Sorry I just saw, that is the format from Log4Net.ElasticSearch someone should really write an issue to get that fixed @timestamp should be the default timestamp field :slight_smile: It also looks that this logger is becoming legacy it is still setting _type which legacy / deprecated and will not longer be supported going forward.

This is deprecated see here

        "_type" : "logEvent",

I believe there is an ECS Compliant Logger Here

But that may not be an option.

So what you can do is use an ingest pipeline to rename the field and we can get ride of that _type too

PUT _ingest/pipeline/dotnet-logger-pipeline
{
  "processors": [
    {
      "rename": {
        "field": "timeStamp",
        "target_field": "@timestamp"
      }
    },
    {
      "remove": {
        "field": "_type"
      }
    }
  ]
}

You can create a _template and name that pipeline as the index default pipeline

PUT _template/scs-client-log
{
  "index_patterns": [
    "scs-client-log*"
  ],
  "settings": {
    "default_pipeline": "dotnet-logger-pipeline"
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
     ..... <- The rest of your mapping 
    }
  }
}

Then on ingest it should remove the _type and rename the timeStamp field to @timestamp

It works now! Thank you so much for your help and your patience.

Now I wanted to try "Anomaly detection with Machine Learning" (Observability -> Logs -> Anomalies) and I got a new problem:

I get the impression that using Log4net.ElasticSearch leads to a couple of problems. I will take a look on the ECS Compliant Logger.

Cool,

Yes that is an issue too....

I would look at the ECS logger

event.dataset is required but can you fix that reql quick by adding

event.datasetas a keyword type in your mapping and then adding a set processor

{
  "set": {
    "field": "event.dataset",
    "value": "scs-log"
  }
}

that will probably fix that...

1 Like

Hi @AnneHermann

I just found this, not sure why I did not see it earlier. I think we just released some new docs! :slight_smile:

Hope this helps!

1 Like