Thanks
You mean this flag button?
1 Like
Hi @leandrojmp,
Made the changes as above and can see logs are appearing in order. (commented pipeline.workers : 1)
Oct 21 21:47:07 <hostname> -api: {"level":"info","time":1634842027575,"pid":105690,"hostname":"<hostname>","config":{"appName":"-api","appHost":"https://dev.com","port":"88","tenantName":"b","sepc":"next","bankingApi":"https://api.com","consentsAuth":"https://login.com","apidocsWebsite":"https://dev.com","tenantFintechName":"onefintech","datafeedServerCallback":"https://data-feed..orufin/callback","pispWebsiteCallback":"https://isp..orufin/buy","enableSwaggerPreview":true,"pdUri":"dap://pd.com:13","pdBindDn":"cn= manager","pdBindPassword":"***","pdBaseDn":"dc=bank,dc=com","pdBankTppDn":"TP","pdBankOauthClientsDn":"Clients","passwordChangeFailureAttempts":"***","passwordChangeFailureLockTime":"***","validation":{"usernameMinLength":"6","usernameMaxLength":"48","passwordMinLength":"***","passwordMaxLength":"***"},"pic":{"baseUri":"https://.consumer/-api","gatewayHost":"gw.com","catalog":"next","organization":"DEV","client":{"clientId":"consumer-kit-app","clientSecret":"***"},"apiPlan":"default-plan","apm":{"baseUri":false,"realm":false,"adin":{"login":false,"password":"***"}}},"ping":{"restApiUri":"https://login.com/pf-ws/rest","revokeToken":"https://login.com/as/revoke_token.oauth2","refreshToken":"https://login.com/as/token.oauth2","exchangeToken":"https://login.com/as/token.oauth2","createDisabled":false,"credentials":{"oauthAdmin":{"login":"devportal","password":"***"},"devPortal":{"login":"dev-portal","password":"***"}}},"mail":{"options":{"host":"smtp.","port":"2"},"contact":{"to":"Open@DEV.com","from":"prtal-Test@DEV.com","subject":"DEV Contact","template":"<p><strong>[[name]]</strong> <[[email]]></p><p>[[message]]</p>"},"events":{"tracking":"account:registration, account:activation","to":"Open@DEV.com","from":"DEV-Test@DEV.com","subject":"DEV [[eventName]]","templates":{"default":"<p><strong>[[eventName]]</strong></p><p>[[username]]</p>","account:registration":"<p><strong>[[eventName]]</strong></p><p>New Account registered with [[username]] / [[organization]]</p>","account:activation":"<p><strong>[[eventName]]</strong></p><p>Account registered with [[username]] / [[organization]] has been activated</p>"}}},"recaptcha {"enabled":false,"verifyUri":"https://www.google.com/recaptcha/api/siteverify","secret":"***"},"statics":"../data/public","PRODUCTION":true},"msg":"server config","v":1}
Oct 21 21:47:07 <hostname> -api: {"level":"info","time":1634842027583,"pid":105690,"hostname":"<hostname>","msg":"server started on port 88","v":1}
Oct 21 21:47:08 <hostname> -api: {"level":"info","time":1634842028085,"pid":105690,"hostname":"<hostname>","msg":"ldap connecting dap://pd.com:13","v":1}
Oct 21 21:47:08 <hostname> -api: {"level":"info","time":1634842028095,"pid":105690,"hostname":"<hostname>","msg":"ldap client connected","v":1}
Oct 21 21:47:08 <hostname> -api: {"level":"info","time":1634842028440,"pid":105690,"hostname":"<hostname>","msg":"ldap credentials bound","v":1}
Thanks.
only left thing is the big message is not getting parsed even when json filter is used to parse json data.
Thanks,
I'm sorry, but it is confusing now.
If you applied the filters I suggested, got no errors, and the messages are in order, then your message field is being parsed as the time field relies on the parsed message.
If your message are not being parsed, then you would have a _dateparsefailure on your messages as the date filter needs the time field that will only exist if you parse the message.
It is not clear what is the issue now, you need to provide more information.
What is not being parsed? Show your full document, do not share screenshots.
What is output message and what is your expected output message?
I will provide more info/output tomorrow. As you can see above server.log has total 5 log events but kibana only showing four i.e the big json message is not getting parsed it looks.
Thanks,
It is still no clear what the issue is.
A message not being parsed means that it was ingested, but your filters didn't worked on that message, and it will still appear in Kibana with some tags like _grokparsefailure or _jsonparsefailure, if your message does not appear in Kibana, it is a completely different error as this means that your message was not indexed.
First share your original messages using the Preformatted button, to make each line appear in one line, the first message you shared makes very hard to read it as it is not shared in this way.
Share your full pipeline, and tell what is your expected output, also share the output you are getting in Kibana. You are providing only limited information, avoid sharing screenshots as they are hard to read an some people may not be able to see it.
Also share logstash logs from the momento you indexed those file.
Yep 
1
Hi @leandrojmp,
Some messages are not getting indexed nad having below error.
I tried accessing application so that it will generate more logs.
I have searched error logs for App_Server2 for particular application logs dev-api in logstash logs.
(I was avoiding preformatting as need to scroll right side)
below logs has error like,
....:response=>{"index"=>{"_index"=>"filebeat-7.14.0-2021.10", "_type"=>"_doc", "_id"=>"P_FptnwBOor-K-JUWRUC", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [response.body] of type [text] in document with id 'P_FptnwBOor-K-JUWRUC'. Preview of field's value: '{email=<email_id>}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1932"}}}}}
.....msg\":\"external request\",\"v\":1}"}], :response=>{"index"=>{"_index"=>"dev-api-000001", "_type"=>"_doc", "_id"=>"Ie28tnwB6LeW4Sv3KavF", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [request.body] tried to parse field [body] as object, but found a concrete value"}}}}
(In above, not sure why its going to two diff index, it should only go to dev-api-000001. (pipeline config output is below)
complete logs -
[root@<logstash_server1> ~]# cat /var/log/logstash/logstash-plain.log |grep '<App_server2_IP>'| grep 'dev-api'| grep error -i
[2021-10-25T10:45:29,885][WARN ][logstash.outputs.elasticsearch][main][d5b2f6fb514512917f52a7710cddd4830e9af6db81a3651f173122b6e693d132] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.14.0-2021.10", :routing=>nil}, {"type"=>"dev-api_app_server2", "host"=>{"os"=>{"type"=>"linux", "version"=>"7.9 (Maipo)", "codename"=>"Maipo", "name"=>"Red Hat Enterprise Linux Server", "platform"=>"rhel", "kernel"=>"3.10.0-1160.45.1.el7.x86_64", "family"=>"redhat"}, "ip"=>["<App_server2_IP>", "fe80::250:56ff:fbbe:5990"], "mac"=>["00:52:50:be:52:96"], "containerized"=>false, "architecture"=>"x86_64", "name"=>"app_server2", "id"=>"b4a4", "hostname"=>"<App_server2>"}, "program"=>"dev-api", "message"=>"Oct 25 10:45:23 <App_server2> dev-api: {\"level\":\"info\",\"time\":1635147923598,\"pid\":105690,\"hostname\":\"<App_server2>\",\"response\":{\"uri\":\"[https://api-consumer-/con-api/me?fields=email\](https://api-consumer-/con-api/me?fields=email%5C)",\"method\":\"GET\",\"statusCode\":200,\"statusMessage\":\"OK\",\"headers\":{\"date\":\"Mon, 25 Oct 2021 07:45:23 GMT\",\"content-type\":\"application/json; charset=utf-8\",\"content-length\":\"51\",\"connection\":\"close\",\"strict-transport-security\":\"max-age=31536000; includeSubDomains\",\"x-request-id\":\"e5b6023c96e35c340b554f58d947a4fe\",\"etag\":\"W/\\\"33-hjv8/5wxs9RZaSBsQh3LGMGO1ms\\\"\",\"vary\":\"Accept-Encoding\"},\"body\":{\"email\":\"<email_id>\"}},\"msg\":\"external response\",\"v\":1}", "level"=>"info", "pid"=>105690, "agent"=>{"type"=>"filebeat", "version"=>"7.14.0", "ephemeral_id"=>"e88a0591-8183-4b84-ac75-8338e28a73df", "name"=>"app_server2", "id"=>"250b578c-a719-4fd0-8f37-1dc6a1c3b346", "hostname"=>"<App_server2>"}, "msg"=>"external response", "time"=>1635147923598, "logsource"=>"<App_server2>", "hostname"=>"<App_server2>", "log_type"=>"dev-api_app_server2", "tags"=>["beats_input_codec_plain_applied"], "ecs"=>{"version"=>"1.10.0"}, "v"=>1, "app_id"=>"node", "input"=>{"type"=>"log"}, "@timestamp"=>2021-10-25T07:45:23.598Z, "log"=>{"offset"=>6307, "file"=>{"path"=>"/var/log/dev-api/server.log"}}, "timestamp"=>"Oct 25 10:45:23", "response"=>{"statusMessage"=>"OK", "body"=>{"email"=>"<email_id>"}, "headers"=>{"connection"=>"close", "date"=>"Mon, 25 Oct 2021 07:45:23 GMT", "strict-transport-security"=>"max-age=31536000; includeSubDomains", "x-request-id"=>"e5b6023c96e35c340b554f58d947a4fe", "content-type"=>"application/json; charset=utf-8", "content-length"=>"51", "etag"=>"W/\"33-hjv8/5wxs9RZaSBsQh3LGMGO1ms\"", "vary"=>"Accept-Encoding"}, "method"=>"GET", "statusCode"=>200, "uri"=>"https://api-consumer-/con-api/me?fields=email"}, "@version"=>"1", "json_message"=>"{\"level\":\"info\",\"time\":1635147923598,\"pid\":105690,\"hostname\":\"<App_server2>\",\"response\":{\"uri\":\"[https://api-consumer-/con-api/me?fields=email\](https://api-consumer-/con-api/me?fields=email%5C)",\"method\":\"GET\",\"statusCode\":200,\"statusMessage\":\"OK\",\"headers\":{\"date\":\"Mon, 25 Oct 2021 07:45:23 GMT\",\"content-type\":\"application/json; charset=utf-8\",\"content-length\":\"51\",\"connection\":\"close\",\"strict-transport-security\":\"max-age=31536000; includeSubDomains\",\"x-request-id\":\"e5b6023c96e35c340b554f58d947a4fe\",\"etag\":\"W/\\\"33-hjv8/5ws\\\"\",\"vary\":\"Accept-Encoding\"},\"body\":{\"email\":\"<email_id>\"}},\"msg\":\"external response\",\"v\":1}"}], :response=>{"index"=>{"_index"=>"filebeat-7.14.0-2021.10", "_type"=>"_doc", "_id"=>"P_FptnwBOor-K-JUWRUC", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [response.body] of type [text] in document with id 'P_FptnwBOor-K-JUWRUC'. Preview of field's value: '{email=<email_id>}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1932"}}}}}
[2021-10-25T12:15:57,257][WARN ][logstash.outputs.elasticsearch][main][482] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dev-api", :routing=>nil}, {"type"=>"dev-api_app_server2", "host"=>{"os"=>{"type"=>"linux", "version"=>"7.9 (Maipo)", "codename"=>"Maipo", "name"=>"Red Hat Enterprise Linux Server", "platform"=>"rhel", "kernel"=>"3.10.0-1160.45.1.el7.x86_64", "family"=>"redhat"}, "hostname"=>"<App_server2>", "containerized"=>false, "mac"=>["00:52:50:be:52:96"], "architecture"=>"x86_64", "id"=>"b4964fefa8d845fb81b0dc1cc071a9a4", "name"=>"app_server2", "ip"=>["<App_server2_IP>", "fe80::250:56ff:fbbe:5990"]}, "program"=>"dev-api", "message"=>"Oct 25 12:15:55 <App_server2> dev-api: {\"level\":\"info\",\"time\":1635153355844,\"pid\":105690,\"hostname\":\"<App_server2>\",\"request\":{\"method\":\"POST\",\"uri\":\"[https://login./as/token.oauth2\](https://login./as/token.oauth2%5C)",\"headers\":{\"accept\":\"application/json\",\"authorization\":\"Basic ***\",\"content-type\":\"application/x-www-form-urlencoded\"},\"body\":\"grant_type=authorization_code&code=VU0vbP-My-6AW&redirect_uri=https%3A%2F%2Fdev.com%2Fauth%2Fcallback\"},\"msg\":\"external request\",\"v\":1}", "request"=>{"body"=>"grant_type=authorization_code&code=VU0vbP-My-69AAW&redirect_uri=https%3A%2F%2Fdev.com%2Fauth%2Fcallback", "headers"=>{"authorization"=>"Basic ***", "content-type"=>"application/x-www-form-urlencoded", "accept"=>"application/json"}, "method"=>"POST", "uri"=>"https://login./as/token.oauth2"}, "level"=>"info", "pid"=>105690, "agent"=>{"version"=>"7.14.0", "ephemeral_id"=>"e88a0591-8183-4b84-ac75-8338e28a73df", "type"=>"filebeat", "name"=>"app_server2", "id"=>"250b578c-a719-4fd0-8f37-1dc6a1c3b346", "hostname"=>"<App_server2>"}, "msg"=>"external request", "time"=>1635153355844, "logsource"=>"<App_server2>", "hostname"=>"<App_server2>", "log_type"=>"dev-api_app_server2", "tags"=>["beats_input_codec_plain_applied"], "ecs"=>{"version"=>"1.10.0"}, "v"=>1, "app_id"=>"node", "input"=>{"type"=>"log"}, "@timestamp"=>2021-10-25T09:15:55.844Z, "log"=>{"offset"=>30936, "file"=>{"path"=>"/var/log/dev-api/server.log"}}, "timestamp"=>"Oct 25 12:15:55", "@version"=>"1", "json_message"=>"{\"level\":\"info\",\"time\":1635153355844,\"pid\":105690,\"hostname\":\"<App_server2>\",\"request\":{\"method\":\"POST\",\"uri\":\"[https://login./as/token.oauth2\](https://login./as/token.oauth2%5C)",\"headers\":{\"accept\":\"application/json\",\"authorization\":\"Basic ***\",\"content-type\":\"application/x-www-form-urlencoded\"},\"body\":\"grant_type=authorization_code&code=VU0vbP-My-69AW&redirect_uri=https%3A%2F%2Fdev.com%2Fauth%2Fcallback\"},\"msg\":\"external request\",\"v\":1}"}], :response=>{"index"=>{"_index"=>"dev-api-000001", "_type"=>"_doc", "_id"=>"Ie28tnwB6LeW4Sv3KavF", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [request.body] tried to parse field [body] as object, but found a concrete value"}}}}
[2021-10-25T12:15:57,262][WARN ][logstash.outputs.elasticsearch][main][d2] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.14.0-2021.10", :routing=>nil}, {"type"=>"dev-api_app_server2", "host"=>{"hostname"=>"<App_server2>", "os"=>{"type"=>"linux", "version"=>"7.9 (Maipo)", "codename"=>"Maipo", "name"=>"Red Hat Enterprise Linux Server", "platform"=>"rhel", "kernel"=>"3.10.0-1160.45.1.el7.x86_64", "family"=>"redhat"}, "containerized"=>false, "mac"=>["00:52:50:be:52:96"], "architecture"=>"x86_64", "name"=>"app_server2", "id"=>"b4964fefa8d845fb81b0dc1cc071a9a4", "ip"=>["<App_server2_IP>", "fe80::250:56ff:fbbe:5990"]}, "program"=>"dev-api", "message"=>"Oct 25 12:15:55 <App_server2> dev-api: {\"level\":\"info\",\"time\":1635153355917,\"pid\":105690,\"hostname\":\"<App_server2>\",\"response\":{\"uri\":\"[https://login./as/token.oauth2\](https://login./as/token.oauth2%5C)",\"method\":\"POST\",\"statusCode\":200,\"statusMessage\":\"OK\",\"headers\":{\"connection\":\"close\",\"date\":\"Mon, 25 Oct 2021 09:15:55 GMT\",\"x-frame-options\":\"SAMEORIGIN\",\"referrer-policy\":\"origin\",\"cache-control\":\"no-cache, no-store\",\"pragma\":\"no-cache\",\"expires\":\"Thu, 01 Jan 1970 00:00:00 GMT\",\"content-type\":\"application/json;charset=utf-8\",\"set-cookie\":[\"PF=gotwYr5fVVE9ECabx8Ywbu;Path=/;Secure;HttpOnly\"]},\"body\":{\"access_token\":\"eyJhbGciOimxdly\",\"scope\":\"openid dev\",\"id_token\":\"eyJU1hA\",\"token_type\":\"Bearer\",\"expires_in\":7775999}},\"msg\":\"external response\",\"v\":1}", "level"=>"info", "pid"=>105690, "agent"=>{"ephemeral_id"=>"e88a0591-8183-4b84-ac75-8338e28a73df", "type"=>"filebeat", "version"=>"7.14.0", "name"=>"app_server2", "id"=>"250b578c-a719-4fd0-8f37-1dc6a1c3b346", "hostname"=>"<App_server2>"}, "msg"=>"external response", "time"=>1635153355917, "logsource"=>"<App_server2>", "hostname"=>"<App_server2>", "log_type"=>"dev-api_app_server2", "tags"=>["beats_input_codec_plain_applied"], "ecs"=>{"version"=>"1.10.0"}, "v"=>1, "app_id"=>"node", "input"=>{"type"=>"log"}, "@timestamp"=>2021-10-25T09:15:55.917Z, "log"=>{"offset"=>31456, "file"=>{"path"=>"/var/log/dev-api/server.log"}}, "timestamp"=>"Oct 25 12:15:55", "response"=>{"statusMessage"=>"OK", "body"=>{"access_token"=>"ely", "scope"=>"openid dev", "token_type"=>"Bearer", "id_token"=>"eyJhbGciOF1ZCI6ImRldCI_pVSv84-7H1H_GhA", "expires_in"=>7775999}, "headers"=>{"connection"=>"close", "date"=>"Mon, 25 Oct 2021 09:15:55 GMT", "x-frame-options"=>"SAMEORIGIN", "cache-control"=>"no-cache, no-store", "referrer-policy"=>"origin", "expires"=>"Thu, 01 Jan 1970 00:00:00 GMT", "content-type"=>"application/json;charset=utf-8", "pragma"=>"no-cache", "set-cookie"=>["PF=gotwYr5fVVE9ECabx8Ywbu;Path=/;Secure;HttpOnly"]}, "method"=>"POST", "statusCode"=>200, "uri"=>"https://login./as/token.oauth2"}, "@version"=>"1", "json_message"=>"{\"level\":\"info\",\"time\":1635153355917,\"pid\":105690,\"hostname\":\"<App_server2>\",\"response\":{\"uri\":\"[https://login./as/token.oauth2\](https://login./as/token.oauth2%5C)",\"method\":\"POST\",\"statusCode\":200,\"statusMessage\":\"OK\",\"headers\":{\"connection\":\"close\",\"date\":\"Mon, 25 Oct 2021 09:15:55 GMT\",\"x-frame-options\":\"SAMEORIGIN\",\"referrer-policy\":\"origin\",\"cache-control\":\"no-cache, no-store\",\"pragma\":\"no-cache\",\"expires\":\"Thu, 01 Jan 1970 00:00:00 GMT\",\"content-type\":\"application/json;charset=utf-8\",\"set-cookie\":[\"PF=gotwYr5fVVE9ECabx8Ywbu;Path=/;Secure;HttpOnly\"]},\"body\":{\"access_token\":\"eyJ9wmxdly\",\"scope\":\"openid dev\",\"id_token\":\"eyJhbGcZBdwpjlBJEsnQvNzCkizWUy7QRvgRotXLE3rztdZYyE8lfvtrf4cOU1hA\",\"token_type\":\"Bearer\",\"expires_in\":7775999}},\"msg\":\"external response\",\"v\":1}"}], :response=>{"index"=>{"_index"=>"filebeat-7.14.0-2021.10", "_type"=>"_doc", "_id"=>"B368tnwBTpJ_5jDaKejJ", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [response.body] of type [text] in document with id 'B368tnwBTpJ_5jDaKejJ'. Preview of field's value: '{access_token=eyJhbGciOiJSUzUxMiIsImtpZCI6IU1hA, token_type=Bearer, expires_in=7775999}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:3521"}}}}}
I would suggest that you open another topic, this is a completely different error.
"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [request.body] tried to parse field [body] as object, but found a concrete value"}}}}
This means that you are trying to index the field request.body as a json object, but this field was already indexed before as a text field, it is a mapping error.
2
These are the original log message from server log file. (searching with timestamp, found in above error log )
[root@<App_server2> ~]# cat /var/log/dev-api/server.log |grep 1635147923598
Oct 25 10:45:23 <App_server2> dev-api: {"level":"info","time":1635147923598,"pid":105690,"hostname":"<App_server2>","response":{"uri":"https://api-consumer-/con-api/me?fields=email","method":"GET","statusCode":200,"statusMessage":"OK","headers":{"date":"Mon, 25 Oct 2021 07:45:23 GMT","content-type":"application/json; charset=utf-8","content-length":"51","connection":"close","strict-transport-security":"max-age=31536000; includeSubDomains","x-request-id":"e5b6023c96e35c340b554f58d947a4fe","etag":"W/\"33-hjv8/5wxs9RZaSBsQh3LGMGO1ms\"","vary":"Accept-Encoding"},"body":{"email":"<user_email_id>"}},"msg":"external response","v":1}
[root@<App_server2> ~]#
[root@<App_server2> ~]# cat /var/log/dev-api/server.log |grep 1635147927632
Oct 25 10:45:27 <App_server2> dev-api: {"level":"info","time":1635147927632,"pid":105690,"hostname":"<App_server2>","response":{"uri":"https://api-consumer-/con-api/me?fields=email","method":"GET","statusCode":200,"statusMessage":"OK","headers":{"date":"Mon, 25 Oct 2021 07:45:27 GMT","content-type":"application/json; charset=utf-8","content-length":"51","connection":"close","strict-transport-security":"max-age=31536000; includeSubDomains","x-request-id":"752ce54789fe634b6c1bfccc7d00ba5a","etag":"W/\"33-hjv8/5wxs9RZaSBsQh3LGMGO1ms\"","vary":"Accept-Encoding"},"body":{"email":"<user_email_id>"}},"msg":"external response","v":1}
[root@<App_server2> ~]#
[root@<App_server2> ~]# cat /var/log/dev-api/server.log |grep 1635151484138
Oct 25 11:44:44 <App_server2> dev-api: {"level":"info","time":1635151484138,"pid":105690,"hostname":"<App_server2>","response":{"uri":"https://api-consumer-/con-api/me?fields=email","method":"GET","statusCode":200,"statusMessage":"OK","headers":{"date":"Mon, 25 Oct 2021 08:44:44 GMT","content-type":"application/json; charset=utf-8","content-length":"51","connection":"close","strict-transport-security":"max-age=31536000; includeSubDomains","x-request-id":"9373c1038b140e96b1f0fa1acaf60466","etag":"W/\"33-hjv8/5wxs9RZaSBsQh3LGMGO1ms\"","vary":"Accept-Encoding"},"body":{"email":"<user_email_id>"}},"msg":"external response","v":1}
[root@<App_server2> ~]#
[root@<App_server2> ~]# cat /var/log/dev-api/server.log |grep 1635153355844
Oct 25 12:15:55 <App_server2> dev-api: {"level":"info","time":1635153355844,"pid":105690,"hostname":"<App_server2>","request":{"method":"POST","uri":"https://login.com/as/token.oauth2","headers":{"accept":"application/json","authorization":"Basic ***","content-type":"application/x-www-form-urlencoded"},"body":"grant_type=authorization_code&code=VU0vbP-My-69Xa2I6B57fCrVykgqsDWLqVMAAAAW&redirect_uri=https%3A%2F%2Fdev.com%2Fauth%2Fcallback"},"msg":"external request","v":1}
[root@<App_server2> ~]#
[root@<App_server2> ~]# cat /var/log/dev-api/server.log |grep 1635153355917
Oct 25 12:15:55 <App_server2> dev-api: {"level":"info","time":1635153355917,"pid":105690,"hostname":"<App_server2>","response":{"uri":"https://login.com/as/token.oauth2","method":"POST","statusCode":200,"statusMessage":"OK","headers":{"connection":"close","date":"Mon, 25 Oct 2021 09:15:55 GMT","x-frame-options":"SAMEORIGIN","referrer-policy":"origin","cache-control":"no-cache, no-store","pragma":"no-cache","expires":"Thu, 01 Jan 1970 00:00:00 GMT","content-type":"application/json;charset=utf-8","set-cookie":["PF=gotwYr5fVVE9ECabx8Ywbu;Path=/;Secure;HttpOnly"]},"body":{"access_token":"eyJhbGhxM8hzqA-Qt-WE_4slpPWUy7QRvgRotXLE3rztdZYyE8lfvtrf4cOU1hA","token_type":"Bearer","expires_in":7775999}},"msg":"external response","v":1}
[root@<App_server2> ~]#
[root@<App_server2> ~]# cat /var/log/dev-api/server.log |grep 1635153361592
Oct 25 12:16:01 <App_server2> dev-api: {"level":"info","time":1635153361592,"pid":105690,"hostname":"<App_server2>","response":{"uri":"https://api-consumer-/con-api/orgs/<user_email_id>/apps","method":"GET","statusCode":200,"statusMessage":"OK","headers":{"date":"Mon, 25 Oct 2021 09:16:01 GMT","content-type":"application/json; charset=utf-8","content-length":"4158","connection":"close","vary":"Accept-Encoding, Accept-Encoding","strict-transport-security":"max-age=31536000; includeSubDomains","x-request-id":"c3a0381d0a05eef0877f46a340a7f9b8","etag":"W/\"103e-5GdaWQahpJN4S9laVMlXOXAb4E8\""},"body":{"total_results":4,"results":[{"type":"app","api_version":"2.0.0","id":"e9a8e347-ba02-4fc8-a32c-bd231deaccde","name":"ais1","title":"ais1","summary":"{\"product\":{\"label\":\"NextPS2 ais\",\"productId\":\"3d010c25-a411-43e1-9cb8-eec63e7db46c\"}}","state":"enabled","lifecycle_state":"production","app_credential_urls":["https://api-consumer-/con-api/apps/3d0e7bfc-845f-4682-9ea5-4f4680d64e52/e9a8e347-ba02-4fc8-a32c-bd231deaccde/credentials/4d092a4d-2650-486b-9fb1-cc1e3a7853cd"],"created_at":"2021-07-12T12:14:37.144Z","updated_at":"2021-07-12T12:14:45.049Z","org_url":"https://api-consumer-/con-api/orgs/3d0e7bfc-845f-4682-9ea5-4f4680d64e52","url":"https://api-consumer-/con-api/apps/3d0e7bfc-845f-4682-9ea5-4f4680d64e52/e9a8e347-ba02-4fc8-a32c-bd231deaccde"},{"type":"app","api_version":"2.0.0","id":"52d6c947-93f4-476b-acb7-299ba329d3a9","name":"ais2","title":"ais2","summary":"{\"product\":{\"label\":\"NextPS2 ais\",\"productId\":\"3d010c25-a411-43e1-9cb8-eec63e7db46c\"}}","state":"enabled","lifecycle_state":"production","app_credential_urls":["https://api-consumer-/con-api/apps/3d0e7bfc-845f-4682-9ea5-4f4680d64e52/52d6c947-93f4-476b-acb7-299ba329d3a9/credentials/8ed65e5f-d9da-47ba-8f0d-7ee871841207"],"created_at":"2021-09-28T17:01:06.101Z","updated_at":"2021-09-28T17:01:06.890Z","org_url":"https://api-consumer-/con-api/orgs/3d0e7bfc-845f-4682-9ea5-4f4680d64e52","url":"https://api-consumer-/con-api/apps/35-4b-acb7-299ba329d3a9"},{"type":"app","api_version":"2.0.0","id":"fe62e961-a0c9-48a1-ab96-f67613d62432","name":"pip","title":"pip","summary":"{\"product\":{\"label\":\"NextPS2 pip\",\"productId\":\"c2f8c891-9265-4543-964b-9de0cdf25cc6\"}}","state":"enabled","lifecycle_state":"production","app_credential_urls":["https://api-consumer-/con-api/apps/3d0e7bfc-845f-4682-9ea5-4f4680d64e52/fe62e961-a0c9-48a1-ab96-f67613d62432/credentials/88229947-f158-4e5f-9cb0-970dc50918f2"],"created_at":"2021-08-04T19:16:06.820Z","updated_at":"2021-08-04T19:16:09.570Z","org_url":"https://api-consumer-/con-api/orgs/3d0e7bfc-845f-4682-9ea5-4f4680d64e52","url":"https://api-consumer-/con-api/apps/32"},{"type":"app","api_version":"2.0.0","id":"d1183d8e-3277-485a-a8f1-784313643dbe","name":"pis1","title":"pis1","summary":"{\"product\":{\"label\":\"NextPS2 pis\",\"productId\":\"1a\"}}","state":"enabled","lifecycle_state":"production","app_credential_urls":["https://api-consumer-/con-api/apps/3e/credentials/1b"],"created_at":"2021-07-30T12:30:43.165Z","updated_at":"2021-07-30T12:30:49.591Z","org_url":"https://api-consumer-/con-api/orgs/32","url":"https://api-consumer-/con-api/apps/3d0e7bfc-845f-4682-9ea5-4fe"}]}},"msg":"external response","v":1}
[root@<App_server2> ~]#
[root@<App_server2> ~]# cat /var/log/dev-api/server.log |grep 1635153651047
Oct 25 12:20:51 <App_server2> dev-api: {"level":"info","time":1635153651047,"pid":105690,"hostname":"<App_server2>","response":{"uri":"https://api-consumer-/con-api/me?fields=email","method":"GET","statusCode":200,"statusMessage":"OK","headers":{"date":"Mon, 25 Oct 2021 09:20:51 GMT","content-type":"application/json; charset=utf-8","content-length":"51","connection":"close","strict-transport-security":"max-age=31536000; includeSubDomains","x-request-id":"3ed","etag":"W/\"33-hjv8/5s\"","vary":"Accept-Encoding"},"body":{"email":"<user_email_id>"}},"msg":"external response","v":1}
If above error message are not indexed not sure why but able to see output of below search query by searching with above timestamp in dev-api-000001 index (except for 1635153355844).
GET /dev-api-000001/_search
{
"query": {
"match": {
"time": "1635147923598"
}
}
}
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "dev-api-000001",
"_type" : "_doc",
"_id" : "Me",
"_score" : 1.0,
"_ignored" : [
"message.keyword",
"json_message.keyword"
],
"_source" : {
"type" : "dev-api_app_server2",
"host" : {
"os" : {
"type" : "linux",
"version" : "7.9 (Maipo)",
"codename" : "Maipo",
"name" : "Red Hat Enterprise Linux Server",
"platform" : "rhel",
"kernel" : "3.10.0-1160.45.1.el7.x86_64",
"family" : "redhat"
},
"ip" : [
"<App_server2_IP>",
"fe80::250:56ff:fbbe:5994"
],
"mac" : [
"00:50:56:bb:60:94"
],
"containerized" : false,
"architecture" : "x86_64",
"name" : "app_server2",
"id" : "b4a4",
"hostname" : "<App_Server_2>"
},
"program" : "dev-api",
"message" : """Oct 25 10:45:23 <App_Server_2> dev-api: {"level":"info","time":1635147923598,"pid":105690,"hostname":"<App_Server_2>","response":{"uri":"https://apiman-con-/con-api/me?fields=email","method":"GET","statusCode":200,"statusMessage":"OK","headers":{"date":"Mon, 25 Oct 2021 07:45:23 GMT","content-type":"application/json; charset=utf-8","content-length":"51","connection":"close","strict-transport-security":"max-age=31536000; includeSubDomains","x-request-id":"e5d947a4fe","etag":"W/\"33-hjv8/5ws\"","vary":"Accept-Encoding"},"body":{"email":"<email_user_id>"}},"msg":"external response","v":1}""",
"level" : "info",
"pid" : 105690,
"agent" : {
"type" : "filebeat",
"version" : "7.14.0",
"ephemeral_id" : "e8f",
"name" : "app_server2",
"id" : "26",
"hostname" : "<App_Server_2>"
},
"msg" : "external response",
"time" : 1635147923598,
"logsource" : "<App_Server_2>",
"hostname" : "<App_Server_2>",
"log_type" : "dev-api_app_server2",
"tags" : [
"beats_input_codec_plain_applied"
],
"ecs" : {
"version" : "1.10.0"
},
"v" : 1,
"app_id" : "node",
"input" : {
"type" : "log"
},
"@timestamp" : "2021-10-25T07:45:23.598Z",
"log" : {
"offset" : 6307,
"file" : {
"path" : "/var/log/dev-api/server.log"
}
},
"timestamp" : "Oct 25 10:45:23",
"response" : {
"statusMessage" : "OK",
"body" : {
"email" : "<email_user_id>"
},
"headers" : {
"connection" : "close",
"date" : "Mon, 25 Oct 2021 07:45:23 GMT",
"strict-transport-security" : "max-age=31536000; includeSubDomains",
"x-request-id" : "e5e",
"content-type" : "application/json; charset=utf-8",
"content-length" : "51",
"etag" : "W/\"33-hjv8/5s\"",
"vary" : "Accept-Encoding"
},
"method" : "GET",
"statusCode" : 200,
"uri" : "https://apiman-con-/con-api/me?fields=email"
},
"@version" : "1",
"json_message" : """{"level":"info","time":1635147923598,"pid":105690,"hostname":"<App_Server_2>","response":{"uri":"https://apiman-con-/con-api/me?fields=email","method":"GET","statusCode":200,"statusMessage":"OK","headers":{"date":"Mon, 25 Oct 2021 07:45:23 GMT","content-type":"application/json; charset=utf-8","content-length":"51","connection":"close","strict-transport-security":"max-age=31536000; includeSubDomains","x-request-id":"e8d947a4fe","etag":"W/\"33-hjv8/5ws\"","vary":"Accept-Encoding"},"body":{"email":"<email_user_id>"}},"msg":"external response","v":1}"""
}
}
]
}
}
GET /dev-api-000001/_search
{
"query": {
"match": {
"time": "1635153355844"
}
}
}
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
3
Below is the mapping for request, response field.
"request" : {
"properties" : {
"body" : {
"properties" : {
"client" : {
"properties" : {
"bypassApprovalPage" : {
"type" : "boolean"
},
"response" : {
"properties" : {
"body" : {
"properties" : {
"access_token" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
This is the pipeline configuration for dev-api application.
input {
beats {
port => 5044
}
}
filter {
if [log_type] == "dev-api_server1" and [app_id] == "node"
{
grok { match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:json_message}" } } json { source => "json_message" }
date { match => ["time", "UNIX_MS"]
}
mutate {
replace => {
"[type]" => "dev-api_server1"
}
}
}
if [log_type] == "dev-api_server2" and [app_id] == "node"
{
grok { match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:json_message}" } } json { source => "json_message" }
date { match => ["time", "UNIX_MS"]
}
mutate {
replace => {
"[type]" => "dev-api_server2"
}
}
}
output {
if [log_type] == "dev-api_server1" {
elasticsearch {
hosts => ['http://es_1:<es_port>', 'http://es_2:<es_port>', 'http://es_3:<es_port>']
index => "dev-api"
template_name => "dev-api"
template_overwrite => "false"
user => elastic
password => "${es_pwd}"
}
}
if [log_type] == "dev-api_server2" {
elasticsearch {
hosts => ['http://es_1:<es_port>', 'http://es_2:<es_port', 'http://es_3:<es_port>']
index => "dev-api"
template_name => "dev-api"
template_overwrite => "false"
user => elastic
password => "${es_pwd}"
}
}
Thanks,
Hi @leandrojmp,
Thanks. Opened. Can you please check and help here.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.