Kibana (7.11.1) isn't showing logs

Elastic Stack Kibana
1

Hi,

I have an index, that is containing log data, but the data can't be seen under Observability -> Logs. I have also created an index pattern and choose a time field.
I adjusted the settings of Observability -> Logs to "Log indices: {my-index-name}*."
But I just get " There are no log messages to display. Try adjusting your filter."
Apparently there is an Internal Server Error with the following message:

[query_shard_exception] No mapping found for [@timestamp] in order to sort on, with { index_uuid="BhrWkvASSDakWeMZJByc6w" & index="scs-client-log" }

The name of the time field is in the index pattern is "timeStamp". Is there a problem because of the "@"?

grafik
grafik1190×61 2.1 KB

2

Hi @AnneHermann Welcome to the community.

Also I am not sure where that screenshot is from, a little more context helps us help you. :slight_smile: Which App / Visualization is that?

When you created the index pattern what did you select at the Time field?

If you you are not using the default @timestamp then you should select the field that your timestamp is in

Screen Shot 2021-03-25 at 11.13.59 AM
Screen Shot 2021-03-25 at 11.13.59 AM3478×1320 406 KB

3

Thanks for the quick reply :slight_smile:

In the screenshot above you can see my time field in my index pattern.

I have chosen "timeStamp" as my time field when creating the index pattern:

grafik
grafik1241×566 34.2 KB

4

First just to be clear so I don't make assumptions when you showed setting the indices in log viewer you showed

{my-index-name}*.

You should put in the index pattern name like

scs-client-logs-*

2nd thing I notice is that you have a field

properties.@timestamp

Which is highly unusual so let's back up a bit and take a look perhaps at the indices and mapping.

In Kibana / Dev Tools can you run

GET _cat/indices?v

And post the results?

If there are indices with documents can you post

GET /<indexname>/

That should show the mapping.

Finally you could also post

GET /<indexname>/_search

And post a couple docs

This should help us diagnos

5 
GET _cat/indices?v

grafik
grafik1107×259 25 KB 

GET /scs-client-log/

{
  "scs-client-log" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "className" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "exception" : {
          "type" : "object"
        },
        "fileName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "fix" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "fullInfo" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "hostName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "level" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "lineNumber" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "messageObject" : {
          "type" : "object"
        },
        "methodName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "properties" : {
          "properties" : {
            "@timestamp" : {
              "type" : "date"
            },
            "log4net:HostName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "tab" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "timeStamp" : {
          "type" : "date"
        }
      }
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "scs-client-log",
        "creation_date" : "1614264757339",
        "number_of_replicas" : "1",
        "uuid" : "BhrWkvASSDakWeMZJByc6w",
        "version" : {
          "created" : "7110199"
        }
      }
    }
  }
}

GET /scs-client-log/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1317,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "scs-client-log",
        "_type" : "logEvent",
        "_id" : "ySWv2XcBYgpFxvFAgAvO",
        "_score" : 1.0,
        "_source" : {
          "timeStamp" : "2021-02-25T11:55:33.3922395Z",
          "message" : "SCS.Client (3.4.0.297) - LogManager initialisiert",
          "messageObject" : { },
          "exception" : { },
          "loggerName" : null,
          "domain" : null,
          "identity" : null,
          "level" : "INFO",
          "className" : "LogManager",
          "fileName" : "",
          "lineNumber" : "0",
          "fullInfo" : "LogManager.(:0)",
          "methodName" : "",
          "fix" : "All",
          "properties" : {
            "log4net:UserName" : null,
            "log4net:Identity" : null,
            "tab" : """	""",
            "log4net:HostName" : "DE106YDX",
            "@timestamp" : "2021-02-25T11:55:33.3922395Z"
          },
          "userName" : null,
          "threadName" : null,
          "hostName" : "DE106YDX"
        }
      },

I hope that this helps. I've tried using "scs-client-logs-*" in the index pattern, but there is the same error.

6

In general looks pretty good, there are a few things I might fix, but those should not keep you from seeing the logs. We can come back to those.

Lets try this.

Go delete the existing index pattern.

Recreate the index pattern for now just use scs-client-log no * or anything.

Use the timeStamp field as the time field.

Go to Discover

Select index pattern scs-client-log and in the time picker select last 90 days and lets see what you get...

Screen Shot 2021-03-26 at 1.28.44 PM
Screen Shot 2021-03-26 at 1.28.44 PM3166×1090 367 KB

7

The log data appears under Discover, but not under Logs.

grafik
grafik1666×537 61.9 KB

grafik
grafik1660×795 64.1 KB

8

Hi @AnneHermann

I should have recognized this earlier, but it has been recently deprecated. The logviewier uses ECS (Elastic Common Schema) fields so it is based on the @timestamp field which is the correct ECS field.

In the Logs / Settings you will see this

Screen Shot 2021-03-29 at 7.23.51 AM
Screen Shot 2021-03-29 at 7.23.51 AM2816×2074 343 KB

What we really need to do is fix your timestamp in the ingest process.

This is what I was referring

I just did not realize we have deprecated setting in the log viewer, so it needs to be fixed first.

You want the either rename this field to @timestamp (yes the @ is correct)

or move this field to the root level

Get that fixed and delete and recreate the index patterns and you should be good to go.

and of course you need to add your index pattern here in the logs settings

Screen Shot 2021-03-29 at 7.48.13 AM
Screen Shot 2021-03-29 at 7.48.13 AM2866×1082 152 KB

9

okay and we have no other option because the function you mentioned is deprecated? I have to change the name of the timeStamp field (or move the other to the root level)?
Because I am using Log4Net.ElasticSearch to log messages to the ElasticSearch document database and I think the name of the field is set in this library.

10

Yes that is correct, and what you really should do anyways.

Best practice is the timestamp in the @timestamp field in the root level, that complies with ECS

How are you ingesting the data? Logstash? Beats?

Let me know if you need help with that.

11

@AnneHermann

Sorry I just saw, that is the format from Log4Net.ElasticSearch someone should really write an issue to get that fixed @timestamp should be the default timestamp field :slight_smile: It also looks that this logger is becoming legacy it is still setting _type which legacy / deprecated and will not longer be supported going forward.

This is deprecated see here

        "_type" : "logEvent",

I believe there is an ECS Compliant Logger Here

But that may not be an option.

So what you can do is use an ingest pipeline to rename the field and we can get ride of that _type too

PUT _ingest/pipeline/dotnet-logger-pipeline
{
  "processors": [
    {
      "rename": {
        "field": "timeStamp",
        "target_field": "@timestamp"
      }
    },
    {
      "remove": {
        "field": "_type"
      }
    }
  ]
}

You can create a _template and name that pipeline as the index default pipeline

PUT _template/scs-client-log
{
  "index_patterns": [
    "scs-client-log*"
  ],
  "settings": {
    "default_pipeline": "dotnet-logger-pipeline"
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      }
     ..... <- The rest of your mapping 
    }
  }
}

Then on ingest it should remove the _type and rename the timeStamp field to @timestamp

12

It works now! Thank you so much for your help and your patience.

Now I wanted to try "Anomaly detection with Machine Learning" (Observability -> Logs -> Anomalies) and I got a new problem:

grafik
grafik833×811 60.5 KB

I get the impression that using Log4net.ElasticSearch leads to a couple of problems. I will take a look on the ECS Compliant Logger.

13

Cool,

Yes that is an issue too....

I would look at the ECS logger

event.dataset is required but can you fix that reql quick by adding

event.datasetas a keyword type in your mapping and then adding a set processor

{
  "set": {
    "field": "event.dataset",
    "value": "scs-log"
  }
}

that will probably fix that...

1 Like
14

Hi @AnneHermann

I just found this, not sure why I did not see it earlier. I think we just released some new docs! :slight_smile:

Hope this helps!

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.