Hi,
I have this log file /opt/apigee/var/log/edge-message-processor/messagelogging/apigee-dac-training/test/sf-response-parameters/6/log-api/elk.log which is seen in the kibana dashboard. I am searching for a specific string "interactionId":"b014415c-1795-46e5-8585-e65ed31c5a81" in the elk.log file.
Please guide me how do I search for any error exceptions or any specific string as mentioned above in the elk.log file.
Thanks in advance.
Best Regards,
Kaushal
Hi,
I will appreciate it if someone can pitch in for my earlier post to this forum.
Thanks in advance.
Best Regards,
Kaushal
Hi,
# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)
# rpm -qa | grep -i filebeat
filebeat-8.10.4-1.x86_64
# rpm -qa | grep elasticsearch
elasticsearch-8.10.4-1.x86_64
# rpm -qa | grep kibana
kibana-8.10.4-1.x86_64
# rpm -qa | grep logstash
logstash-8.10.4-1.x86_64
#
cat /etc/logstash/conf.d/beats.conf
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://192.168.0.109:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
filebeat yml file :- https://termbin.com/sjq5
Logstatsh yml file :- https://termbin.com/xzcq
Elasticsearch yml file :- https://termbin.com/fszw5
Kibana yml file :- https://termbin.com/afbj9
I am attaching all the config files for your reference. Please guide me.
Thanks in advance.
Best Regards,
KaushalDo you see your logs in Discover?
Can you share the JSON of one of the documents that has been ingested?
How you search will depend on how / if the data is parsed?
@stephenb Thank you for your response. I am attaching the screenshot for your reference.
Please refer to the below JSON output for one of the document.
{
"_index": "filebeat-8.10.4-2023.11.09",
"_id": "jreasYsBWPdMhzgU9hTW",
"_version": 1,
"_score": 0,
"_ignored": [
"event.original.keyword",
"message.keyword"
],
"_source": {
"agent": {
"name": "apigeeapigatewaytraining",
"id": "25105342-6cf3-417a-9b9a-faca5db07260",
"ephemeral_id": "c580c801-b50b-4d23-90d0-eae5e4db4abb",
"type": "filebeat",
"version": "8.10.4"
},
"ecs": {
"version": "8.0.0"
},
"tags": [
"beats_input_codec_plain_applied"
],
"@version": "1",
"input": {
"type": "log"
},
"event": {
"original": "Nov 9 06:33:22 apigeeapigatewaytraining filebeat: {\"log.level\":\"info\",\"@timestamp\":\"2023-11-09T06:33:22.699+0530\",\"log.logger\":\"monitoring\",\"log.origin\":{\"file.name\":\"log/log.go\",\"file.line\":187},\"message\":\"Non-zero metrics in the last 30s\",\"service.name\":\"filebeat\",\"monitoring\":{\"metrics\":{\"beat\":{\"cpu\":{\"system\":{\"ticks\":81770,\"time\":{\"ms\":10}},\"total\":{\"ticks\":283760,\"time\":{\"ms\":30},\"value\":283760},\"user\":{\"ticks\":201990,\"time\":{\"ms\":20}}},\"handles\":{\"limit\":{\"hard\":4096,\"soft\":4096},\"open\":12},\"info\":{\"ephemeral_id\":\"c580c801-b50b-4d23-90d0-eae5e4db4abb\",\"uptime\":{\"ms\":224190773},\"version\":\"8.10.4\"},\"memstats\":{\"gc_next\":36543608,\"memory_alloc\":22993264,\"memory_total\":13519746040,\"rss\":45096960},\"runtime\":{\"goroutines\":45}},\"filebeat\":{\"events\":{\"active\":0,\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":1}},\"output\":{\"events\":{\"acked\":1,\"active\":0,\"batches\":1,\"total\":1},\"read\":{\"bytes\":6},\"write\":{\"bytes\":1133}},\"pipeline\":{\"clients\":3,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":13,\"update\":1},\"writes\":{\"success\":1,\"total\":1}},\"system\":{\"load\":{\"1\":0.04,\"15\":0.05,\"5\":0.04,\"norm\":{\"1\":0.01,\"15\":0.0125,\"5\":0.01}}}},\"ecs.version\":\"1.6.0\"}}"
},
"@timestamp": "2023-11-09T01:03:23.776Z",
"message": "Nov 9 06:33:22 apigeeapigatewaytraining filebeat: {\"log.level\":\"info\",\"@timestamp\":\"2023-11-09T06:33:22.699+0530\",\"log.logger\":\"monitoring\",\"log.origin\":{\"file.name\":\"log/log.go\",\"file.line\":187},\"message\":\"Non-zero metrics in the last 30s\",\"service.name\":\"filebeat\",\"monitoring\":{\"metrics\":{\"beat\":{\"cpu\":{\"system\":{\"ticks\":81770,\"time\":{\"ms\":10}},\"total\":{\"ticks\":283760,\"time\":{\"ms\":30},\"value\":283760},\"user\":{\"ticks\":201990,\"time\":{\"ms\":20}}},\"handles\":{\"limit\":{\"hard\":4096,\"soft\":4096},\"open\":12},\"info\":{\"ephemeral_id\":\"c580c801-b50b-4d23-90d0-eae5e4db4abb\",\"uptime\":{\"ms\":224190773},\"version\":\"8.10.4\"},\"memstats\":{\"gc_next\":36543608,\"memory_alloc\":22993264,\"memory_total\":13519746040,\"rss\":45096960},\"runtime\":{\"goroutines\":45}},\"filebeat\":{\"events\":{\"active\":0,\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":1}},\"output\":{\"events\":{\"acked\":1,\"active\":0,\"batches\":1,\"total\":1},\"read\":{\"bytes\":6},\"write\":{\"bytes\":1133}},\"pipeline\":{\"clients\":3,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":13,\"update\":1},\"writes\":{\"success\":1,\"total\":1}},\"system\":{\"load\":{\"1\":0.04,\"15\":0.05,\"5\":0.04,\"norm\":{\"1\":0.01,\"15\":0.0125,\"5\":0.01}}}},\"ecs.version\":\"1.6.0\"}}",
"log": {
"file": {
"path": "/var/log/messages"
},
"offset": 16031480
},
"host": {
"hostname": "apigeeapigatewaytraining",
"id": "5398f52b66bc457b99db318a519bb25d",
"name": "apigeeapigatewaytraining",
"architecture": "x86_64",
"containerized": false,
"ip": [
"192.168.0.146"
],
"mac": [
"52-54-00-E2-46-31"
],
"os": {
"version": "7 (Core)",
"name": "CentOS Linux",
"kernel": "3.10.0-1160.102.1.el7.x86_64",
"type": "linux",
"family": "redhat",
"codename": "Core",
"platform": "centos"
}
}
},
"fields": {
"agent.version.keyword": [
"8.10.4"
],
"host.architecture.keyword": [
"x86_64"
],
"host.name.keyword": [
"apigeeapigatewaytraining"
],
"host.hostname": [
"apigeeapigatewaytraining"
],
"host.mac": [
"52-54-00-E2-46-31"
],
"ecs.version.keyword": [
"8.0.0"
],
"host.ip.keyword": [
"192.168.0.146"
],
"host.os.version": [
"7 (Core)"
],
"host.os.name": [
"CentOS Linux"
],
"agent.name": [
"apigeeapigatewaytraining"
],
"host.id.keyword": [
"5398f52b66bc457b99db318a519bb25d"
],
"host.name": [
"apigeeapigatewaytraining"
],
"host.os.version.keyword": [
"7 (Core)"
],
"event.original": [
"Nov 9 06:33:22 apigeeapigatewaytraining filebeat: {\"log.level\":\"info\",\"@timestamp\":\"2023-11-09T06:33:22.699+0530\",\"log.logger\":\"monitoring\",\"log.origin\":{\"file.name\":\"log/log.go\",\"file.line\":187},\"message\":\"Non-zero metrics in the last 30s\",\"service.name\":\"filebeat\",\"monitoring\":{\"metrics\":{\"beat\":{\"cpu\":{\"system\":{\"ticks\":81770,\"time\":{\"ms\":10}},\"total\":{\"ticks\":283760,\"time\":{\"ms\":30},\"value\":283760},\"user\":{\"ticks\":201990,\"time\":{\"ms\":20}}},\"handles\":{\"limit\":{\"hard\":4096,\"soft\":4096},\"open\":12},\"info\":{\"ephemeral_id\":\"c580c801-b50b-4d23-90d0-eae5e4db4abb\",\"uptime\":{\"ms\":224190773},\"version\":\"8.10.4\"},\"memstats\":{\"gc_next\":36543608,\"memory_alloc\":22993264,\"memory_total\":13519746040,\"rss\":45096960},\"runtime\":{\"goroutines\":45}},\"filebeat\":{\"events\":{\"active\":0,\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":1}},\"output\":{\"events\":{\"acked\":1,\"active\":0,\"batches\":1,\"total\":1},\"read\":{\"bytes\":6},\"write\":{\"bytes\":1133}},\"pipeline\":{\"clients\":3,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":13,\"update\":1},\"writes\":{\"success\":1,\"total\":1}},\"system\":{\"load\":{\"1\":0.04,\"15\":0.05,\"5\":0.04,\"norm\":{\"1\":0.01,\"15\":0.0125,\"5\":0.01}}}},\"ecs.version\":\"1.6.0\"}}"
],
"host.os.type": [
"linux"
],
"agent.id.keyword": [
"25105342-6cf3-417a-9b9a-faca5db07260"
],
"@version.keyword": [
"1"
],
"input.type": [
"log"
],
"log.offset": [
16031480
],
"tags": [
"beats_input_codec_plain_applied"
],
"host.architecture": [
"x86_64"
],
"agent.id": [
"25105342-6cf3-417a-9b9a-faca5db07260"
],
"ecs.version": [
"8.0.0"
],
"host.containerized": [
false
],
"host.hostname.keyword": [
"apigeeapigatewaytraining"
],
"agent.version": [
"8.10.4"
],
"host.os.family": [
"redhat"
],
"input.type.keyword": [
"log"
],
"tags.keyword": [
"beats_input_codec_plain_applied"
],
"host.ip": [
"192.168.0.146"
],
"agent.type": [
"filebeat"
],
"host.os.kernel.keyword": [
"3.10.0-1160.102.1.el7.x86_64"
],
"host.os.kernel": [
"3.10.0-1160.102.1.el7.x86_64"
],
"@version": [
"1"
],
"host.os.name.keyword": [
"CentOS Linux"
],
"host.id": [
"5398f52b66bc457b99db318a519bb25d"
],
"log.file.path.keyword": [
"/var/log/messages"
],
"agent.type.keyword": [
"filebeat"
],
"agent.ephemeral_id.keyword": [
"c580c801-b50b-4d23-90d0-eae5e4db4abb"
],
"host.os.codename.keyword": [
"Core"
],
"host.mac.keyword": [
"52-54-00-E2-46-31"
],
"agent.name.keyword": [
"apigeeapigatewaytraining"
],
"host.os.codename": [
"Core"
],
"message": [
"Nov 9 06:33:22 apigeeapigatewaytraining filebeat: {\"log.level\":\"info\",\"@timestamp\":\"2023-11-09T06:33:22.699+0530\",\"log.logger\":\"monitoring\",\"log.origin\":{\"file.name\":\"log/log.go\",\"file.line\":187},\"message\":\"Non-zero metrics in the last 30s\",\"service.name\":\"filebeat\",\"monitoring\":{\"metrics\":{\"beat\":{\"cpu\":{\"system\":{\"ticks\":81770,\"time\":{\"ms\":10}},\"total\":{\"ticks\":283760,\"time\":{\"ms\":30},\"value\":283760},\"user\":{\"ticks\":201990,\"time\":{\"ms\":20}}},\"handles\":{\"limit\":{\"hard\":4096,\"soft\":4096},\"open\":12},\"info\":{\"ephemeral_id\":\"c580c801-b50b-4d23-90d0-eae5e4db4abb\",\"uptime\":{\"ms\":224190773},\"version\":\"8.10.4\"},\"memstats\":{\"gc_next\":36543608,\"memory_alloc\":22993264,\"memory_total\":13519746040,\"rss\":45096960},\"runtime\":{\"goroutines\":45}},\"filebeat\":{\"events\":{\"active\":0,\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":1}},\"output\":{\"events\":{\"acked\":1,\"active\":0,\"batches\":1,\"total\":1},\"read\":{\"bytes\":6},\"write\":{\"bytes\":1133}},\"pipeline\":{\"clients\":3,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":13,\"update\":1},\"writes\":{\"success\":1,\"total\":1}},\"system\":{\"load\":{\"1\":0.04,\"15\":0.05,\"5\":0.04,\"norm\":{\"1\":0.01,\"15\":0.0125,\"5\":0.01}}}},\"ecs.version\":\"1.6.0\"}}"
],
"host.os.family.keyword": [
"redhat"
],
"@timestamp": [
"2023-11-09T01:03:23.776Z"
],
"host.os.type.keyword": [
"linux"
],
"host.os.platform": [
"centos"
],
"host.os.platform.keyword": [
"centos"
],
"log.file.path": [
"/var/log/messages"
],
"agent.ephemeral_id": [
"c580c801-b50b-4d23-90d0-eae5e4db4abb"
]
},
"ignored_field_values": {
"message.keyword": [
"Nov 9 06:33:22 apigeeapigatewaytraining filebeat: {\"log.level\":\"info\",\"@timestamp\":\"2023-11-09T06:33:22.699+0530\",\"log.logger\":\"monitoring\",\"log.origin\":{\"file.name\":\"log/log.go\",\"file.line\":187},\"message\":\"Non-zero metrics in the last 30s\",\"service.name\":\"filebeat\",\"monitoring\":{\"metrics\":{\"beat\":{\"cpu\":{\"system\":{\"ticks\":81770,\"time\":{\"ms\":10}},\"total\":{\"ticks\":283760,\"time\":{\"ms\":30},\"value\":283760},\"user\":{\"ticks\":201990,\"time\":{\"ms\":20}}},\"handles\":{\"limit\":{\"hard\":4096,\"soft\":4096},\"open\":12},\"info\":{\"ephemeral_id\":\"c580c801-b50b-4d23-90d0-eae5e4db4abb\",\"uptime\":{\"ms\":224190773},\"version\":\"8.10.4\"},\"memstats\":{\"gc_next\":36543608,\"memory_alloc\":22993264,\"memory_total\":13519746040,\"rss\":45096960},\"runtime\":{\"goroutines\":45}},\"filebeat\":{\"events\":{\"active\":0,\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":1}},\"output\":{\"events\":{\"acked\":1,\"active\":0,\"batches\":1,\"total\":1},\"read\":{\"bytes\":6},\"write\":{\"bytes\":1133}},\"pipeline\":{\"clients\":3,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":13,\"update\":1},\"writes\":{\"success\":1,\"total\":1}},\"system\":{\"load\":{\"1\":0.04,\"15\":0.05,\"5\":0.04,\"norm\":{\"1\":0.01,\"15\":0.0125,\"5\":0.01}}}},\"ecs.version\":\"1.6.0\"}}"
],
"event.original.keyword": [
"Nov 9 06:33:22 apigeeapigatewaytraining filebeat: {\"log.level\":\"info\",\"@timestamp\":\"2023-11-09T06:33:22.699+0530\",\"log.logger\":\"monitoring\",\"log.origin\":{\"file.name\":\"log/log.go\",\"file.line\":187},\"message\":\"Non-zero metrics in the last 30s\",\"service.name\":\"filebeat\",\"monitoring\":{\"metrics\":{\"beat\":{\"cpu\":{\"system\":{\"ticks\":81770,\"time\":{\"ms\":10}},\"total\":{\"ticks\":283760,\"time\":{\"ms\":30},\"value\":283760},\"user\":{\"ticks\":201990,\"time\":{\"ms\":20}}},\"handles\":{\"limit\":{\"hard\":4096,\"soft\":4096},\"open\":12},\"info\":{\"ephemeral_id\":\"c580c801-b50b-4d23-90d0-eae5e4db4abb\",\"uptime\":{\"ms\":224190773},\"version\":\"8.10.4\"},\"memstats\":{\"gc_next\":36543608,\"memory_alloc\":22993264,\"memory_total\":13519746040,\"rss\":45096960},\"runtime\":{\"goroutines\":45}},\"filebeat\":{\"events\":{\"active\":0,\"added\":1,\"done\":1},\"harvester\":{\"open_files\":1,\"running\":1}},\"libbeat\":{\"config\":{\"module\":{\"running\":1}},\"output\":{\"events\":{\"acked\":1,\"active\":0,\"batches\":1,\"total\":1},\"read\":{\"bytes\":6},\"write\":{\"bytes\":1133}},\"pipeline\":{\"clients\":3,\"events\":{\"active\":0,\"published\":1,\"total\":1},\"queue\":{\"acked\":1}}},\"registrar\":{\"states\":{\"current\":13,\"update\":1},\"writes\":{\"success\":1,\"total\":1}},\"system\":{\"load\":{\"1\":0.04,\"15\":0.05,\"5\":0.04,\"norm\":{\"1\":0.01,\"15\":0.0125,\"5\":0.01}}}},\"ecs.version\":\"1.6.0\"}}"
]
}
}
Please guide me. Thanks in advance.
Best Regards,
Kaushal
Hi @kaushalshriyan Good information
BUT the sample Document you gave looks like filebeat log so that field / text does not exist.
The other things is that you have not parse your logs so the entire log ends up in a field called message
In general, you would parse your logs with 1 of 3 Ways
They are of a known type that a filebeat module support
You would parse with logstash filters Grok etc
You would parse with an ingest pipeline
Otherwise, you whole log will just stay in the message field
You also need to learn the difference between keyword field (exact match) and a text field which is full text search
If the message field is a type text or match_only_text
In Dev Tools run
GET filebeat-8.10.4-2023.11.09_mapping/field/message
In discover try
message : "b014415c-1795-46e5-8585-e65ed31c5a81"
@stephenb Thanks a lot for sharing with me an approach to this specific use case. I will appreciate it if you can share with me some working examples to understand it better for the below-mentioned options.
Please guide me. Once again thanks in advance.
Best Regards,
Kaushal
Perhaps you should look at our documents and nanny resources available to you
Adding Data
Many getting started
https://www.elastic.co/getting-started
Free Training
https://www.elastic.co/training/free
There is much content...
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.