# Observability Overview - Logs not shown as log source

**URL:** <https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183>\
**Category:** Elastic Observability\
**Tags:** ecs-elastic-common-schema\
**Created:** [October 3, 2024, 10:13am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183 "2024-10-03T10:13:25Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 3, 2024, 10:13am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/1 "2024-10-03T10:13:25Z")

</div>

Hi

Running Kibana version: 8.14.3

I'm trying to show my "log sources" via the Observability Overview screen. It contains a via view: Log Events \> Logs rate per minute

I assume this is the required config:

> **[Logs app fields | Elastic Observability \[8.15\] | Elastic](https://www.elastic.co/guide/en/observability/current/logs-app-fields.html)**

Although I have the below fields, it is still not shown:

- @timestamp
- message
- event.dataset
- host.hostname
- host.name
- container.id
- service.name

Are there any requirements on field types? I converted already most of them from text to keyword.

Logs are popping up fine via stream & explorer (both under Observability \> Logs).

Anyone who has an idea on how to show my log sources in the Overview screen?

Thanks!  
Christof

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 3, 2024, 10:57am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/2 "2024-10-03T10:57:12Z")

</div>

Extra info: Logs are neither shown as "unknown". So really just absent in the overview screen..

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c44fa9edbddb34b57674d402dc5c06d924ac316.png)

Sample log & in the background you see +20.000 logs available but not shown above in the overview.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9cb5fce44f8b4519541046e8e5c8122e661b0985.jpeg)

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 7, 2024, 6:25am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/3 "2024-10-07T06:25:49Z")

</div>

No one who knows how this works?

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 9, 2024, 10:02am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/4 "2024-10-09T10:02:55Z")

</div>

I set up a separate stack via Docker for testing purposes:  
It seems that the field responsible for "count" is giving issues:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/5/759224749326f521a70ceb446036384bbc26c51a.png)

Still need to proceed my testing what is causing this behavior.. 🙂

REMARK: Strange thing is that even if I only configure my jboss index in the original set up, it does not show the "unknown" source

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 10, 2024, 12:47pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/5 "2024-10-10T12:47:43Z")

</div>

I just found out that the time window also plays a role:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/2/227def73c7b9bb9911cb54bbeba3b59d3ea06c0d.png)  
Same "Jboss" source, but counted:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3b71377c9c4221a3c6cc60d75ca5541c369dcf46.png)  
And using a different time filter, I can show the bar but I'm missing the count on top:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/5/15bebc2ce0b312691c6d1c84d2bcf8f9ce56b166.png)

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 10, 2024, 12:57pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/6 "2024-10-10T12:57:03Z")

</div>

New observation:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/2/920663a2cd52a4341a30d33dd1403c2ca4ab72d1.png)

Events:

- Oct 10, 2024 @ 14:43:11.279
- Oct 10, 2024 @ 14:44:13.355  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/b/dbe12d631b0acecb7c166c614f23c9864409b712.png)

Event - additional host fields:

- Oct 10, 2024 @ 14:45:31.227  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fbe8f6dfaf6d70532decb68f406e46cbeafa3eca.png)

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 10, 2024, 1:06pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/7 "2024-10-10T13:06:25Z")

</div>

Got an even more bizar scenario..

Only 1 event:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9072beb3d785e0f94affdc564b909ab763b6824.png)

But "Log Events" is able to count "2", bar is missing again:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9fac4002d0715ed74bd647cb2e412bbc157b1e20.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 11, 2024, 12:05am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/8 "2024-10-11T00:05:28Z")

</div>

Hi @chouben

I am not following all this but I can say some things are not correct. I am not sure how you are ingesting the logs, how if / you create index templates

I think you need to read this to understand data stream naming conventions even of you are not using agent these concepts still hold.

> **[Data streams | Fleet and Elastic Agent Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/fleet/current/data-streams.html)**

When I look below your data stream is

`logs-debug-test`

Which should be

```auto
type: logs
datastream: debug
namespace: test 

```

But some reason you have set the values differently and that is not good / best practice.

 ![Screenshot 2024-10-10 at 4.14.49 PM](https://us1.discourse-cdn.com/elastic/original/3X/9/0/904ad428d407a632337298104ca32c31fb3926d0.jpeg)

So how are you ingest? A

> [@chouben](#):
>
> - @timestamp
> - message
> - event.dataset
> - host.hostname
> - host.name
> - container.id
> - service.name
> 
> Are there any requirements on field types? I converted already most of them from text to keyword.

These are all ECS fields and the mapping should be automatically mapped for you

if you run the commands below you should see the fields you want to be mapped correctly so I am not sure how you are ingesting

```auto
DELETE _data_stream/logs-debug-test/

GET logs-debug-test/

POST logs-debug-test/_doc
{
  "@timestamp": "2024-10-10T23:40:04.279Z",
  "message" : "my log message",
  "data_stream" : {
    "dataset" : "debug",
    "namespace" : "test"
  },
  "event" : {
    "dataset" : "mycustomlogs"
  },
  "host" : {
    "name" : "myhost",
    "hostname" : "myhostname"
  },
  "container" : {
    "id" : "12345-345876"
  },
  "service" :{ "name" : "myservice"}
}

GET logs-debug-test/_search

GET logs-debug-test/

```

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 11, 2024, 6:21am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/9 "2024-10-11T06:21:33Z")

</div>

Hi @stephenb

Good catch, but I'm simply sending in the test cases with below code via dev console:

```auto
POST /logs-debug-test/_doc
{
    "message": "Starting up Elasticsearch",
    "event": {
      "dataset": "jboss"
    },
    "level": "debug",
    "host": {
      "hostname": "xxxxxxmon01",
      "name": "xxxxxxmon01"
    }
  }

```

So not sure why elastic is mapping to an incorrect datastream field 🤔 It seems that I "broke" it during my testing.. Might be related to a different message I recovered from the operational stack, where I forgot to remove the data\_stream part.

After dropping the existing datastream, this is how my message looks like:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8cbed4096b21cef68df135c32c9a21632a9adcdd.png)

Last 15 minutes:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/9/7943c09bc67036f9e33cb40fa1e1d6f6877d00d1.png)

8:14-8:20

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/5/35d87578dc4a4cf86a2272a2c0f6e35bb730f4b8.png)

So there is still some issue with the counting. I'll get back to your sample case now..

Best regards  
Christof

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 11, 2024, 6:28am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/10 "2024-10-11T06:28:30Z")

</div>

Running your test case in 18.4.3:

I removed the datastream so everything is created from scratch.  
I also removed the "@timestamp" to have recent & unique timestamps.

```auto
POST logs-debug-test/_doc
{
  "message" : "my log message",
  "data_stream" : {
    "dataset" : "debug",
    "namespace" : "test"
  },
  "event" : {
    "dataset" : "mycustomlogs"
  },
  "host" : {
    "name" : "myhost",
    "hostname" : "myhostname"
  },
  "container" : {
    "id" : "12345-345876"
  },
  "service" :{ "name" : "myservice"}
}

```

After sending in, it look like:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00390faa53c6f6204e2ccc30340227f8a8283f13.png)

Result is still the same @stephenb :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/b/0b0803a9f2be826b214d543701aaf86765716abc.png)

Data stream with default logs index template:

> GET /\_data\_stream/logs-debug-test/

```auto
{
  "data_streams": [
    {
      "name": "logs-debug-test",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-logs-debug-test-2024.10.11-000001",
          "index_uuid": "bloEyFieQUyiu6a5t5IYLQ",
          "prefer_ilm": true,
          "ilm_policy": "logs",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 1,
      "_meta": {
        "managed": true,
        "description": "default logs template installed by x-pack"
      },
      "status": "YELLOW",
      "template": "logs",
      "ilm_policy": "logs",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}

```

Index

> GET logs-debug-test/

```auto
{
  ".ds-logs-debug-test-2024.10.11-000001": {
    "aliases": {},
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "dynamic_templates": [
        {
          "ecs_timestamp": {
            "match": "@timestamp",
            "mapping": {
              "ignore_malformed": false,
              "type": "date"
            }
          }
        },
        {
          "ecs_message_match_only_text": {
            "path_match": [
              "message",
              "*.message"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "ecs_non_indexed_keyword": {
            "path_match": "event.original",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "keyword"
            }
          }
        },
        {
          "ecs_non_indexed_long": {
            "path_match": "*.x509.public_key_exponent",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "long"
            }
          }
        },
        {
          "ecs_ip": {
            "path_match": [
              "ip",
              "*.ip",
              "*_ip"
            ],
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "ecs_wildcard": {
            "path_match": [
              "*.io.text",
              "*.message_id",
              "*registry.data.strings",
              "*url.path"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_wildcard_and_match_only_text": {
            "path_match": [
              "*.body.content",
              "*url.full",
              "*url.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_match_wildcard_and_match_only_text": {
            "match": [
              "*command_line",
              "*stack_trace"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "*.title",
              "*.executable",
              "*.name",
              "*.working_directory",
              "*.full_name",
              "*file.path",
              "*file.target_path",
              "*os.full",
              "email.subject",
              "vulnerability.description",
              "user_agent.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        },
        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        },
        {
          "ecs_path_match_float": {
            "path_match": [
              "*.score.*",
              "*_score*"
            ],
            "path_unmatch": "*.version",
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "float"
            }
          }
        },
        {
          "ecs_usage_double_scaled_float": {
            "path_match": "*.usage",
            "match_mapping_type": [
              "double",
              "long",
              "string"
            ],
            "mapping": {
              "scaling_factor": 1000,
              "type": "scaled_float"
            }
          }
        },
        {
          "ecs_geo_point": {
            "path_match": [
              "location",
              "*.location"
            ],
            "mapping": {
              "type": "geo_point"
            }
          }
        },
        {
          "ecs_flattened": {
            "path_match": [
              "*structured_data",
              "*exports",
              "*imports"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
        {
          "all_strings_to_keywords": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date",
          "ignore_malformed": false
        },
        "container": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword",
              "value": "debug"
            },
            "namespace": {
              "type": "constant_keyword",
              "value": "test"
            },
            "type": {
              "type": "constant_keyword",
              "value": "logs"
            }
          }
        },
        "event": {
          "properties": {
            "dataset": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "host": {
          "properties": {
            "hostname": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            }
          }
        },
        "message": {
          "type": "match_only_text"
        },
        "service": {
          "properties": {
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "lifecycle": {
          "name": "logs"
        },
        "codec": "best_compression",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "mapping": {
          "total_fields": {
            "ignore_dynamic_beyond_limit": "true"
          },
          "ignore_malformed": "true"
        },
        "hidden": "true",
        "number_of_shards": "1",
        "provided_name": ".ds-logs-debug-test-2024.10.11-000001",
        "default_pipeline": "logs@default-pipeline",
        "creation_date": "1728627754279",
        "number_of_replicas": "1",
        "uuid": "bloEyFieQUyiu6a5t5IYLQ",
        "version": {
          "created": "8505000"
        }
      }
    },
    "data_stream": "logs-debug-test"
  }
}

```

Best regards  
Christof

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 11, 2024, 6:34am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/11 "2024-10-11T06:34:23Z")

</div>

Using my 8.15.2 stack:

Same message:

```auto
POST logs-debug-test/_doc
{
  "message" : "my log message",
  "data_stream" : {
    "dataset" : "debug",
    "namespace" : "test"
  },
  "event" : {
    "dataset" : "mycustomlogs"
  },
  "host" : {
    "name" : "myhost",
    "hostname" : "myhostname"
  },
  "container" : {
    "id" : "12345-345876"
  },
  "service" :{ "name" : "myservice"}
}

```

Which looks like:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/1/11f27d4f336e125c6af0fb09deab7cc1f44196b0.png)

Result is the same:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/6/06d9edb315b6db2123d2b4eddc2cd5922c389234.png)

Data stream with logs template:

> GET /\_data\_stream/logs-debug-test/

```auto
{
  "data_streams": [
    {
      "name": "logs-debug-test",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-logs-debug-test-2024.10.11-000001",
          "index_uuid": "ApyItpxBQPSTnqwXWok9Dg",
          "prefer_ilm": true,
          "ilm_policy": "logs",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 1,
      "_meta": {
        "description": "default logs template installed by x-pack",
        "managed": true
      },
      "status": "YELLOW",
      "template": "logs",
      "ilm_policy": "logs",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "rollover_on_write": false
    }
  ]
}

```

Index:

> GET logs-debug-test/

```auto
{
  ".ds-logs-debug-test-2024.10.11-000001": {
    "aliases": {},
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "dynamic_templates": [
        {
          "ecs_timestamp": {
            "match": "@timestamp",
            "mapping": {
              "ignore_malformed": false,
              "type": "date"
            }
          }
        },
        {
          "ecs_message_match_only_text": {
            "path_match": [
              "message",
              "*.message"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "ecs_non_indexed_keyword": {
            "path_match": "event.original",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "keyword"
            }
          }
        },
        {
          "ecs_non_indexed_long": {
            "path_match": "*.x509.public_key_exponent",
            "mapping": {
              "doc_values": false,
              "index": false,
              "type": "long"
            }
          }
        },
        {
          "ecs_ip": {
            "path_match": [
              "ip",
              "*.ip",
              "*_ip"
            ],
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "ecs_wildcard": {
            "path_match": [
              "*.io.text",
              "*.message_id",
              "*registry.data.strings",
              "*url.path"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_wildcard_and_match_only_text": {
            "path_match": [
              "*.body.content",
              "*url.full",
              "*url.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_match_wildcard_and_match_only_text": {
            "match": [
              "*command_line",
              "*stack_trace"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "wildcard"
            }
          }
        },
        {
          "ecs_path_match_keyword_and_match_only_text": {
            "path_match": [
              "*.title",
              "*.executable",
              "*.name",
              "*.working_directory",
              "*.full_name",
              "*file.path",
              "*file.target_path",
              "*os.full",
              "email.subject",
              "vulnerability.description",
              "user_agent.original"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              },
              "type": "keyword"
            }
          }
        },
        {
          "ecs_date": {
            "path_match": [
              "*.timestamp",
              "*_timestamp",
              "*.not_after",
              "*.not_before",
              "*.accessed",
              "created",
              "*.created",
              "*.installed",
              "*.creation_date",
              "*.ctime",
              "*.mtime",
              "ingested",
              "*.ingested",
              "*.start",
              "*.end",
              "*.indicator.first_seen",
              "*.indicator.last_seen",
              "*.indicator.modified_at",
              "*threat.enrichments.matched.occurred"
            ],
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "date"
            }
          }
        },
        {
          "ecs_path_match_float": {
            "path_match": [
              "*.score.*",
              "*_score*"
            ],
            "path_unmatch": "*.version",
            "unmatch_mapping_type": "object",
            "mapping": {
              "type": "float"
            }
          }
        },
        {
          "ecs_usage_double_scaled_float": {
            "path_match": "*.usage",
            "match_mapping_type": [
              "double",
              "long",
              "string"
            ],
            "mapping": {
              "scaling_factor": 1000,
              "type": "scaled_float"
            }
          }
        },
        {
          "ecs_geo_point": {
            "path_match": "*.geo.location",
            "mapping": {
              "type": "geo_point"
            }
          }
        },
        {
          "ecs_flattened": {
            "path_match": [
              "*structured_data",
              "*exports",
              "*imports"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
        {
          "all_strings_to_keywords": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date",
          "ignore_malformed": false
        },
        "container": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword",
              "value": "debug"
            },
            "namespace": {
              "type": "constant_keyword",
              "value": "test"
            },
            "type": {
              "type": "constant_keyword",
              "value": "logs"
            }
          }
        },
        "event": {
          "properties": {
            "dataset": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "host": {
          "properties": {
            "hostname": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            }
          }
        },
        "message": {
          "type": "match_only_text"
        },
        "service": {
          "properties": {
            "name": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "ignore_dynamic_beyond_limit": "true"
          },
          "ignore_malformed": "true"
        },
        "hidden": "true",
        "provided_name": ".ds-logs-debug-test-2024.10.11-000001",
        "creation_date": "1728628274014",
        "number_of_replicas": "1",
        "uuid": "ApyItpxBQPSTnqwXWok9Dg",
        "version": {
          "created": "8512000"
        },
        "lifecycle": {
          "name": "logs"
        },
        "mode": "standard",
        "codec": "best_compression",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "number_of_shards": "1",
        "default_pipeline": "logs@default-pipeline"
      }
    },
    "data_stream": "logs-debug-test"
  }
}

```

UPDATE:  
Firing in 4 exact the same messages (timestamp is taken at ingestion) extra, gives bars but counter is still incorrect (5 messages now).

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/8/58571f81d3f2b68698f4a612aa7ad3ffabb6d3f7.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 11, 2024, 7:02am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/12 "2024-10-11T07:02:23Z")

</div>

@chouben

Those charts are rates... I do not think just putting a couple of logs is going to work...

I just loaded logs with Python with the same basic data and it works as expected

```auto
DATA_STREAM_TYPE = "logs"
DATA_STREAM_DATASET = "debug"
DATA_STREAM_NAMESPACE= "test"
SERVICE_NAME= "pii-generator"

with open(LOG_FILE, "r") as f:
  for line in f:
    
    # Assuming each line represents a single document
    # subtract random time to spread out the logs
    d = timedelta(seconds=(random.randint(0,300)))
    timestamp = (now-d).isoformat()
    document = {
        "@timestamp" : timestamp,
        "message" : line.strip(),
        "service" : {"name": SERVICE_NAME},
        "data_stream": {"dataset" : DATA_STREAM_DATASET, "namespace" : DATA_STREAM_NAMESPACE},
        "run.id" : run_id,
        "file.name" : LOG_FILE,
        "event" : {
         "dataset" : "mycustomlogs"},
        }
    action = {
      "_index": data_stream,
      "_op_type": "create",
      "_source": document
      }
    data.append(action)
    count = count + 1
...

```

And it loaded and worked fine...

 ![Screenshot 2024-10-10 at 11.59.36 PM](https://us1.discourse-cdn.com/elastic/original/3X/6/8/680fc0a0a8bbb4f611c2fa6a7477e3a88588ed03.png)

 ![Screenshot 2024-10-10 at 11.59.50 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/1/312d3d39408bed23c6b2e827ee6658334ff496ac.jpeg)

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 11, 2024, 9:26am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/13 "2024-10-11T09:26:49Z")

</div>

Hi Stephen

Thanks for your test!!

> [@stephenb](#):
>
> Those charts are rates... I do not think just putting a couple of logs is going to work...
> 
> ...
> 
> And it loaded and worked fine...
> 
> ![Screenshot 2024-10-10 at 11.59.36 PM](https://us1.discourse-cdn.com/elastic/original/3X/6/8/680fc0a0a8bbb4f611c2fa6a7477e3a88588ed03.png)

What I understand from above testcase: You have 15 buckets of 1 minutes, where the average is 667. That means you loaded ~10.000 logs (that seems to match visually with your screenshot).

Visualizing the same for my latest test (10 buckets of 1 minute with 5 documents):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/3/73a6e600ad134d9e5bfb585989d2fb273dd2e16a.png)  
Log rate would be 0.5 but is rounded to 1, cfr. below test.

Extra test to decide on required fields (thrice same fields for all documents, but a different timestamp):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/1/71fd5d7431675c9eaf06e63468c9ecd90923cd34.png)  
3 documents / 3 buckets = 1 log rate  
_buckets = minute 5, 6, 7_  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a08c46702f82b071cd0c46ddbac751fb8da04690.png)  
3 documents / 2 buckets = 2 log rate (1.5 rounded to 2)  
_buckets = minute 6, 7_  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/6/16638a1a9ecee239e83361c3dcf0c7cbb90267a4.png)

@stephenb Would you be able to confirm that:

- At least 2 buckets are required for visualization to work correctly
- Number on top is the average over all buckets, rounded to a whole number
- Based on [Logs app fields | Elastic Observability [8.14] | Elastic](https://www.elastic.co/guide/en/observability/8.14/logs-app-fields.html) there are a lot of required fields. But it seems only "@timestamp" & "event.dataset" are required, although documentation indicates multiple other fields as required. Correct?

Thanks!  
Christof

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 11, 2024, 3:22pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/14 "2024-10-11T15:22:20Z")

</div>

Hi @chouben The goals of this UI is to show overall Rate Trends ...  
So, I am not really sure what you are trying to figure out.  
I can say this UI is used widely as Top Level Metrics by 1000s of users, and in general, we believe it to be accurate.

> [@chouben](#):
>
> - At least 2 buckets are required for visualization to work correctly
> 
> You are going to need to look at the code yourself if you want that level of detail.
> 
> - Number on top is the average over all buckets, rounded to a whole number
> 
> Yes that is my understanding
> 
> - Based on [Logs app fields | Elastic Observability [8.14] | Elastic](https://www.elastic.co/guide/en/observability/8.14/logs-app-fields.html) there are a lot of required fields. But it seems only "@timestamp" & "event.dataset" are required, although documentation indicates multiple other fields as required. Correct?

In general, the expectation is that you are ingestings ECS style logs if you want to use the curated UIs. And this UI is

If `event.dataset` is not set, this screen (should still work or at least it does in 8.15.2) you will get unknown those unknown

 ![Screenshot 2024-10-11 at 8.10.29 AM](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fba4bfdbfb29d488ef992448ff00378559761b47.png)

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 14, 2024, 7:56am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/15 "2024-10-14T07:56:02Z")

</div>

Hi @stephenb

I'm not trying to indicate that this view does not work. I did use it as well on my last project. Now I'm trying to enable it as well on my new project, but it is acting strange.

As you have read in my second comment, I did send in +20.000 logs (migration of old elasticsearch). But the overview is not showing them. So first target is to understand this view, to be able to understand what is wrong with my jboss-fat index.

The overview in below screenshot (8.14.3) contains 60 minutes (= 60 buckets), so that means that system.auth consists of only 2400 logs. It does not make sense that my logs for jboss-fat are not shown (+20.000 logs).

> [@chouben](#):
>
> Extra info: Logs are neither shown as "unknown". So really just absent in the overview screen..
> 
> ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c44fa9edbddb34b57674d402dc5c06d924ac316.png)
> 
> Sample log & in the background you see +20.000 logs available but not shown above in the overview.
> 
> ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9cb5fce44f8b4519541046e8e5c8122e661b0985.jpeg)

Do you see any reason why they are not shown?

Thanks  
Christof

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 14, 2024, 12:30pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/16 "2024-10-14T12:30:33Z")

</div>

**event.dataset: jboss-fat**  
7162 documents:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/1/117355aafe5307959a94de800d56afb22de8a896.png)

Distrubution over the full day: 1,350,276

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/0/60c03a43e2da61d7e91d0416c56fc09726d75116.png)

**event.dataset: system.auth**  
2145 documents:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a291b4fcbba7a877bf50c1ac051f4a9bf3d38ca.png)

Distrubution over the full day: 51,391

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/e/fe4b6878f9a858e90bdf712dbdba1315f773cd15.png)

**Log rate per minute**  
Same timeframe: 00:00-01:00

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/b/0beded552ca08e4112100356a20a0cece6bf3ca2.png)

And the "Logs rate per minute" view uses the same dataview as above "discover":

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af470e54ab79e659e6a9dd4229211b4b8deb020e.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 14, 2024, 3:01pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/17 "2024-10-14T15:01:06Z")

</div>

No, I don't know why... Yes, I would expect them to show assuming everything is configured correctly and ingested correctly\>

When you say migrate what does that mean?

If you go to discover, are they shown correctly there? Do you see the right number? I can't really tell ...

Do you have a timestamp or time zone issue? Are these logs in the past or the future?

Is the `@timestamp` correct? Is the `event.ingested` timestamp correct

There's something subtle going on I suspect.

Perhaps there's a minor bug between 8.14.3 and 8:15.2, which is what I'm using But I I'm not aware ofI'm not aware of anything related to that

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 14, 2024, 3:15pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/18 "2024-10-14T15:15:15Z")

</div>

Checking

What is the data stream name? And what is the dataset name? And what is the event.dataset values

Are they all consistent?

Take the `-` out of the `event.dataset` and cannot be part of `data_stream.dataset`

Try `jboss.fat` for both

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [October 16, 2024, 11:06am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/19 "2024-10-16T11:06:40Z")

</div>

Hi @stephenb

Sorry for the dealy, I didn't find time to do the requested test yesterday.

**Test**  
I did the migration again for the specific datastream (Logstash pipeline below - Fetch data from elasticsearch 7.16 and load it in 8.14.3):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b14ee069a09a5eede968c7d9a712c995629f539a.png)

Same result:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/d/fde3c64a0a5ce77ffeed59a8b332e8f3aa0915d4.png)

`event.dataset` does now contain a `.` instead of a `-`. Below a sample document:  
PS: a colleague of you used `-` in this sample ["Unknown" logs in observability overview - #2 by felixbarny](https://discuss.elastic.co/t/unknown-logs-in-observability-overview/273240/2). That's where I got it from.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/3/93e6d98cf87577d423597ebc450e4682198cfcc7.png)

All below fields are on the screenshot above as before (if you prefer it in a different format because of readability, please do request):

> [@stephenb](#):
>
> What is the data stream name? And what is the dataset name? And what is the event.dataset values

```auto
    "data_stream": {
      "namespace": "jboss",
      "type": "logs",
      "dataset": "info"
    },
...
    "event.dataset": "jboss.fat"

```

> [@stephenb](#):
>
> Do you have a timestamp or time zone issue? Are these logs in the past or the future?

I don't think we are facing timezone issues:

```auto
{
  "_index": ".ds-logs-info-jboss-2024.10.16-000001",
  "_id": "GbtLlJIBP4HuDLbvLht9",
  "_version": 1,
  "_score": 0,
  "_source": {
...
    "@timestamp": "2024-09-28T22:00:05.852Z",

```

To avoid timezone issues I also started loading bigger chunks of consecutive days that even if there would be shifts in hours, there would still be enough data available (as you see below) - 28/9 to 1/10 and we are looking at 29/9:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a5ed2503b6fa46fa90f8062a1999f4be4f5f6d7.png)

**Migration pipeline**

Logstash file 1:

```auto
#Only pipeline size 500 & scroll 5m
#Other running pipeline size 200 & scroll 5m
input {
 elasticsearch {
    hosts => "localhost:9200"
    index => "jboss-fat-2024.09*"
    query => '{ }'
    size => 200
    scroll => "5m"
    docinfo => true
  }
}

filter {
#Parse data via new logic (remove deducted fields)
      mutate {
        remove_field => ["loglevel", "thread", "logtime", "class", "logmessage", "context"]
      }

#ID is generated below, old tags are removed first
      mutate {
        remove_tag => ["idParsed", "idParsingFailed", "dateparsed", "idParsed"]
      }

#key is required for bug: https://github.com/logstash-plugins/logstash-filter-fingerprint/issues/46
    fingerprint {
      source => "message"
      target => "[@metadata][fingerprint]"
      method => "MD5"
      key => "XXX"
    }
    ruby {
      code => "event.set('[@metadata][tsEpochMilliPrefix]', (1000*event.get('@timestamp').to_f).round(0))" 
    }

    if [@metadata][tsEpochMilliPrefix] and [@metadata][fingerprint] {
        mutate {
#Document ID is set in the elasticsearch output plugin
# add_field => { document_id => "%{[@metadata][tsEpochMilliPrefix]}%{[@metadata][fingerprint]}"}
            add_tag => ["idParsed"]
        }
    } else {
        mutate {
            add_tag => ["idParsingFailed"]
        }
    }
}

output {
	if [fields][type] == "jboss" {
	  pipeline { send_to => "jboss-input" }
	} else if [fields][type] == "cassandra" {
	  pipeline { send_to => "cassandra-input" }
	} else if [fields][type] == "kpi" {
	  pipeline { send_to => kpi }
	} else if [fields][type] == "monitoring" {
	  pipeline { send_to => monitoring }
	}
}

```

Logstash file 2:

```auto
input { pipeline { address => "jboss-input" } }

filter {
       grok {
          patterns_dir => ["/etc/logstash/patterns"]
          match => ["message", "^%{TIMESTAMP_ISO8601:[log][time]}%{SPACE}%{SLOGLEVEL:[log][level]}%{SPACE}\[%{ENDCONTEXT:[log][context]}\]%{SPACE}\(%{NOTBRACKET:[log][thread]}\)%{SPACE}%{GREEDYDATA:[log][content]}$"] 
        }
        mutate {
            convert => ["pid", "integer"]
            remove_field => ["offset", "[prospector][type]"]
        }
        date {
            match => ["[log][time]" , "yyyy-MM-dd HH:mm:ss,SSS" ]
            timezone => "Europe/Brussels"
            add_tag => ["dateparsed"]
        }
	#https://www.elastic.co/guide/en/observability/current/logs-app-fields.html
	#https://discuss.elastic.co/t/log-source-unknown-in-observability-overview/262568
	#Required to have the source in Observability - Logs view
      mutate {
        add_field => { "event.dataset" => "%{[fields][type]}.%{[fields][env]}" }
		add_field => { "service.name" => "jboss" }
		add_field => { "host.hostname" => "%{[host][name]}" }
		add_field => { "container.id" => "jboss-%{[host][name]}" }
		add_field => { "log.file.path" => "%{[source]}" }
		#rename => { "[host][name]" => "[host][hostname]" }
      }			  
}

```

I hope I answered all requests.. ☺

Best regards  
Christof

PS:

> [@stephenb](#):
>
> Perhaps there's a minor bug between 8.14.3 and 8:15.2, which is what I'm using But I I'm not aware ofI'm not aware of anything related to that

I assume you do advise now to upgrade to the latest version. I'm still in development phase, so that would be perfectly feasible.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 16, 2024, 4:43pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/20 "2024-10-16T16:43:56Z")

</div>

Not Really sure what to tell you at this point.

Perhaps upgrade to 8.15.2

I can't debug your logstash...

Compare Discover use breakdown `event.dataset` by 1 minute interval

 ![Screenshot 2024-10-16 at 9.11.51 AM](https://us1.discourse-cdn.com/elastic/original/3X/3/c/3c3689ea1d23e4525dfa576897e7ab995b4028e7.jpeg)

Then Observability Overview for 30 Mins

 ![Screenshot 2024-10-16 at 9.11.28 AM](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0ed11c4d4f26c949cac2c65b1a2c4708e190bfe7.png)

They Looks the same for me, if they don't for you I am not sure what to tell you..

I would load some data from another method...  
Then do the comparison if that works then I would look at your migration logstash pipelines.

[Next page](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183.md?page=2)