# Observability Overview - Logs not shown as log source **URL:** \ **Category:** Elastic Observability\ **Tags:** ecs-elastic-common-schema\ **Created:** [October 3, 2024, 10:13am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183 "2024-10-03T10:13:25Z")\ **Posts on this page:** 4\ **Page:** 2
**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\ **Post date:** [October 17, 2024, 8:53am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/21 "2024-10-17T08:53:16Z")
Hi Stephen We upgraded to 8.15.2 - Same result 😓 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/8/28216f701e7568f36d2cd16d75f0b04e8e35fab4.png) Could it be related to `event.dataset` typing? What I noticed now is that `event.dataset` is a `String` field with `keyword` subfield. Grouping options via discover: ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/7/874c5b47990c9e633b2422edc418ffe4e658294d.png) Although still shown via discover as `keyword` field: ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/c/6c3085d604fcc28c843ae76dfb25c65477d15504.png) `keyword` subfield: ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/b/eb0abb775f57b95dcf9d031e0d4ee17477192863.png) `String` subfield: > Not possible to group on String field I'll get back to you once tested with correct typed fields.. Best regards Christof ---
**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\ **Post date:** [October 17, 2024, 9:39am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/22 "2024-10-17T09:39:18Z")
Hi @stephenb It seems your last suggestion to debug via Discover did point me in the right direction! Checking an item further in the past (dataset finally appears): ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f3ed31d44a18d2e0c25e955b1544067d1b13b6af.png) Timeframe we were checking (dataset is shown too together with other sources) 🤩 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/a/0a5cdca9363aae5d396395e21e171e4ed54d787a.png) For completeness, my updated `Component Template` (I'm disabling `all_strings_to_keywords` because `keyword` fields make searches harder) ```auto PUT _component_template/logs@custom { "template": { "settings": { "index": { "number_of_replicas": "0", "default_pipeline": "calculate_lag", "codec": "best_compression" } }, "mappings": { "dynamic": true, "date_detection": true, "dynamic_date_formats": [ "strict_date_optional_time", "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z", "yyyy-MM-dd HH:mm:ss,SSS" ], "dynamic_templates": [ { "all_strings_to_keywords": { "unmatch" : "*", "match_mapping_type": "string", "mapping": { "ignore_above": 1024, "type": "keyword" } } } ], "properties": { "@timestamp": { "type": "date" }, "data_stream.namespace": { "type": "constant_keyword" }, "data_stream.dataset": { "type": "constant_keyword" }, "data_stream.type": { "type": "constant_keyword", "value": "logs" }, "log.level": { "type": "keyword" }, "service.name": { "type": "keyword" }, "event.dataset": { "type": "keyword" }, "container.id": { "type": "keyword" }, "host.hostname": { "type": "keyword" }, "host.name": { "type": "keyword" } } } }, "_meta": { "managed": true, "description": "default mappings for the logs index template installed by x-pack" } } ``` **CONCLUSION** - `event.dataset` field was shown via Discover as `Keyword` field, although it was only a subtype on my system (primary type `String`) - Due to incorrect (primary) type, `Log rate per minute` tile on the `Observability Overview` screen did not pick up the `event.dataset` field - Same solution would have probably worked on `8.14.3`, but system was already upgraded to `8.15.2` Thanks a lot for your persistence! Christof ---
**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\ **Post date:** [October 17, 2024, 11:57am UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/23 "2024-10-17T11:57:57Z")
Hi, I am just scanning this issue quickly but I noticed that your file has beat.version: 6.5.1, which would imply that it's almost 6 years out of date from current, and would have very limited support for Elastic Common Schema, which could be the cause of some of the field type conflicts that you're experiencing. Can you confirm that your beats version is up to date? If not, then you can download current beats from: [Download Beats: Data Shippers for Elasticsearch | Elastic](https://www.elastic.co/downloads/beats) ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/1/61716a685b233bf309f31a4ac5783a64a24db8bd.png) ---
**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\ **Post date:** [October 17, 2024, 12:18pm UTC](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183/24 "2024-10-17T12:18:55Z")
Hi @Mike_Paquette Thanks for stopping by! I'm currently in the process of upgrading an old standalone 7.16 stack to a new RPM installed 8.15.2 stack. The logs are old logs, which have to be migrated. Issue has been resolved since this morning. Below my conclusions: > [@chouben](#): > > **CONCLUSION** > > - `event.dataset` field was shown via Discover as `Keyword` field, although it was only a subtype on my system (primary type `String`) > - Due to incorrect (primary) type, `Log rate per minute` tile on the `Observability Overview` screen did not pick up the `event.dataset` field > - Same solution would have probably worked on `8.14.3`, but system was already upgraded to `8.15.2` With an import remark, why we didn't have the keyword mapping: > [@chouben](#): > > (I'm disabling `all_strings_to_keywords` because `keyword` fields make searches harder) > > ```auto > PUT _component_template/logs@custom > { > "template": { > ... > "mappings": { > ... > "dynamic_templates": [ > { > "all_strings_to_keywords": { > "unmatch" : "*", > "match_mapping_type": "string", > "mapping": { > "ignore_above": 1024, > "type": "keyword" > } > } > } > ... > > ``` Those logs for the migration are fetched, cleaned (file 1) & enriched (file 2) with new fields, cfr. my migration pipeline: > [@chouben](#): > > **Migration pipeline** > > Logstash file 1: > > ```auto > #Only pipeline size 500 & scroll 5m > #Other running pipeline size 200 & scroll 5m > input { > elasticsearch { > hosts => "localhost:9200" > index => "jboss-fat-2024.09*" > query => '{ }' > size => 200 > scroll => "5m" > docinfo => true > } > } > > filter { > #Parse data via new logic (remove deducted fields) > mutate { > remove_field => ["loglevel", "thread", "logtime", "class", "logmessage", "context"] > } > > #ID is generated below, old tags are removed first > mutate { > remove_tag => ["idParsed", "idParsingFailed", "dateparsed", "idParsed"] > } > > #key is required for bug: https://github.com/logstash-plugins/logstash-filter-fingerprint/issues/46 > fingerprint { > source => "message" > target => "[@metadata][fingerprint]" > method => "MD5" > key => "XXX" > } > ruby { > code => "event.set('[@metadata][tsEpochMilliPrefix]', (1000*event.get('@timestamp').to_f).round(0))" > } > > if [@metadata][tsEpochMilliPrefix] and [@metadata][fingerprint] { > mutate { > #Document ID is set in the elasticsearch output plugin > # add_field => { document_id => "%{[@metadata][tsEpochMilliPrefix]}%{[@metadata][fingerprint]}"} > add_tag => ["idParsed"] > } > } else { > mutate { > add_tag => ["idParsingFailed"] > } > } > } > > output { > if [fields][type] == "jboss" { > pipeline { send_to => "jboss-input" } > } else if [fields][type] == "cassandra" { > pipeline { send_to => "cassandra-input" } > } else if [fields][type] == "kpi" { > pipeline { send_to => kpi } > } else if [fields][type] == "monitoring" { > pipeline { send_to => monitoring } > } > } > > ``` > > Logstash file 2: > > ```auto > input { pipeline { address => "jboss-input" } } > > filter { > grok { > patterns_dir => ["/etc/logstash/patterns"] > match => ["message", "^%{TIMESTAMP_ISO8601:[log][time]}%{SPACE}%{SLOGLEVEL:[log][level]}%{SPACE}\[%{ENDCONTEXT:[log][context]}\]%{SPACE}\(%{NOTBRACKET:[log][thread]}\)%{SPACE}%{GREEDYDATA:[log][content]}$"] > } > mutate { > convert => ["pid", "integer"] > remove_field => ["offset", "[prospector][type]"] > } > date { > match => ["[log][time]" , "yyyy-MM-dd HH:mm:ss,SSS" ] > timezone => "Europe/Brussels" > add_tag => ["dateparsed"] > } > #https://www.elastic.co/guide/en/observability/current/logs-app-fields.html > #https://discuss.elastic.co/t/log-source-unknown-in-observability-overview/262568 > #Required to have the source in Observability - Logs view > mutate { > add_field => { "event.dataset" => "%{[fields][type]}.%{[fields][env]}" } > add_field => { "service.name" => "jboss" } > add_field => { "host.hostname" => "%{[host][name]}" } > add_field => { "container.id" => "jboss-%{[host][name]}" } > add_field => { "log.file.path" => "%{[source]}" } > #rename => { "[host][name]" => "[host][hostname]" } > } > } > > ``` FYI: - I was planning to move all beats to an Elastic Agent. In the process however I noted that support for custom log formats is not yet that great on Elastic Agent. So it was still advised to use Filebeat. - All components for the new Elastic Stack are in sync and are updated to 8.15.2 since this morning: Elastic Agent, Filebeat, Logstash, Elasticsearch with Fleet Server, Kibana Best regards Christof [Previous page](https://discuss.elastic.co/t/observability-overview-logs-not-shown-as-log-source/368183.md?page=1)