we have some logs. most of them are constructed, but for the traceback it was unconstructed and seperated in the log, such as
* 2023-05-30T20:52:15.545314-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: Traceback (most recent call last):
* 2023-05-30T20:52:15.545442-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.27 100.65.4.27 31/May/2023/00/52/15 HEAD /v1/AUTH_udc/data-lake-maglev-nvidia-cn/drivedata/lidar_metadata_partitioned/parquet/session_id%253D8ffe520c-baf
* f-11ed-bbc2-00044bf66018/part_301499165.parquet HTTP/1.0 200 - Hadoop%203.2.1%2C%20aws-sdk-java/1.11.655%20Linux/5.4.0-81-generic%20OpenJDK_64-Bit_Server_VM/25.252-b09%20java/1.8.0_252%20scala/2.12.10%20vendor/Oracle_Corporation udc%2Cs3
* - - - txf1dd02cc5db5458396258-0064769a3f - 0.0164 S3 - 1685494335.118332148 1685494335.134732485 0 - AUTH_udc data-lake-maglev-nvidia-cn drivedata/lidar_metadata_partitioned/parquet/session_id%253D8ffe520c-baff-11ed-bbc2-00044bf66018/part_301499165.parquet 200
* 2023-05-30T20:52:15.546315-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/swift/proxy/controllers/base.py", line 1366, in _get_response_parts_iter#012 'part_iter': part_iter}
* 2023-05-30T20:52:15.546482-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: GeneratorExit
* 2023-05-30T20:52:15.546614-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: During handling of the above exception, another exception occurred:
* 2023-05-30T20:52:15.546780-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.26 100.65.4.26 31/May/2023/00/52/15 HEAD /v1/AUTH_nvetl/nvetl-data-prod/cache/data/65/40/6540c922c21af02f65bac06596a7e753d5e57b9d HTTP/1.0 200 - NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/mark%20Lib/com.nvidia.corp.go.maglev.src.common.swiftclient%20Git/e3dd7d3c6%20WorkflowName/xniu-etl-janitor%20WorkflowRun/2023.05.11-0900-1dl2q2x93uasx%20WorkflowTask/mark-from-roots%20WorkflowJob/yiqiang-4d-mlmcf-v50-no-egomotion-3task-train/2023.01.10-0016-1ngxgf4b4qtaj nvetl%2CAUTH_tk2b6... - - - tx88ef2544a121443aa9f98-0064769a3f - 0.0452 - - 1685494335.078597307 1685494335.123748064 0 - AUTH_nvetl nvetl-data-prod cache/data/65/40/6540c922c21af02f65bac06596a7e753d5e57b9d 200
* 2023-05-30T20:52:15.547703-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: Traceback (most recent call last):
* 2023-05-30T20:52:15.547860-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.29 100.65.4.29 31/May/2023/00/52/15 GET /v1/AUTH_nvetl/nvetl-data-prod/cache/data/7c/1a/7c1a8f4f32b3343ec2497524d02770c331240e87 HTTP/1.0 206 - NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/etl-agent%20Lib/com.nvidia.corp.go.maglev.src.common.yarofs%20Git/90cfd2cf2%20WorkflowName/guanlin-gt-egomotion-posegraph-prod%20WorkflowRun/2023.05.28-0000-265p4zuavnaiz%20WorkflowTask/%20WorkflowJob/ nvetl%2Cs3 - 1898212 - tx8de2ecc454d948219dfeb-0064769a3f Range:%20bytes%3D4194304-6092515%0AX-Maglev-Workflow:%20guanlin-gt-egomotion-posegraph-prod/2023.05.28-0000-265p4zuavnaiz 0.0136 S3 - 1685494335.122107744 1685494335.135662556 0 0.004297494888305664 AUTH_nvetl nvetl-data-prod cache/data/7c/1a/7c1a8f4f32b3343ec2497524d02770c331240e87 206
* 2023-05-30T20:52:15.548709-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.29 100.65.4.29 31/May/2023/00/52/15 GET /nvetl-data-prod/cache/data/7c/1a/7c1a8f4f32b3343ec2497524d02770c331240e87 HTTP/1.0 206 - NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/etl-agent%20Lib/com.nvidia.corp.go.maglev.src.common.yarofs%20Git/90cfd2cf2%20WorkflowName/guanlin-gt-egomotion-posegraph-prod%20WorkflowRun/2023.05.28-0000-265p4zuavnaiz%20WorkflowTask/%20WorkflowJob/ - - 1898212 - tx8de2ecc454d948219dfeb-0064769a3f Range:%20bytes%3D4194304-6092515%0AX-Maglev-Workflow:%20guanlin-gt-egomotion-posegraph-prod/2023.05.28-0000-265p4zuavnaiz 0.0257 - - 1685494335.120033026 1685494335.145698786 0 0.007:
* 2023-05-30T20:52:15.549586-04:00 ssnode-proxy-1202-f09-2 proxy-server: info - - 31/May/2023/00/52/15 GET /v1/AUTH_nvetl/nvetl-data-prod%252Bsegments/cache/data/35/32/35321a5e08243c857bf6cdefe556814bbf631a88/OWU4NmQ4ZTctYmY3Ny00NWYzLWJhNTI
* tMDExMzJkZDlhM2Fj/37%3Fmultipart-manifest%3Dget HTTP/1.0 206 - NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/etl-agent%20Lib/com.nvidia.corp.go.maglev.src.common.yarofs%20Git/90cfd2cf2%20WorkflowName/yichengz-train-cn-
* v009-exp2-2023-05-29%20WorkflowRun/2023.05.29-1503-0u45k5ueqevnb%20WorkflowTask/%20WorkflowJob/%20%20SLO%20MultipartGET nvetl%2Cs3 - 8388608 - txbee16dfb5eec4c64ad8ec-0064769a3f Range:%20bytes%3D8388608- 0.0267 SLO - 1685494335.111910105
* 1685494335.138648510 0 0.005083322525024414 AUTH_nvetl nvetl-data-prod%252Bsegments cache/data/35/32/35321a5e08243c857bf6cdefe556814bbf631a88/OWU4NmQ4ZTctYmY3Ny00NWYzLWJhNTItMDExMzJkZDlhM2Fj/37 206
* 2023-05-30T20:52:15.550603-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.4 100.65.4.4 31/May/2023/00/52/14 HEAD /v1/AUTH_nvetl/nvetl-data-prod/cache/data/ee/21/ee2182abb96f55132afae5bc53d0b031940e4646 HTTP/1.0 200 - NVIDIA/1.0
* %20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/mark%20Lib/com.nvidia.corp.go.maglev.src.common.swiftclient%20Git/e3dd7d3c6%20WorkflowName/xniu-etl-janitor%20WorkflowRun/2023.05.11-0900-1dl2q2x93uasx%20WorkflowTask/mark-from-roots
* %20WorkflowJob/yiqiang-4d-mlmcf-v50-no-egomotion-3task-train/2023.01.10-0016-1ngxgf4b4qtaj nvetl%2CAUTH_tk2b6... - - - txad0a069efde241168051d-0064769a3e - 0.0482 - - 1685494334.645647526 1685494334.693838596 0 - AUTH_nvetl nvetl-data-pro
* d cache/data/ee/21/ee2182abb96f55132afae5bc53d0b031940e4646 200
* 2023-05-30T20:52:15.551551-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.30 100.65.4.30 31/May/2023/00/52/14 HEAD /v1/AUTH_nvetl/nvetl-data-prod/cache/data/01/da/01dac740ea18b1728590398db523314039422209 HTTP/1.0 200 - NVIDIA/1
* .0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/mark%20Lib/com.nvidia.corp.go.maglev.src.common.swiftclient%20Git/e3dd7d3c6%20WorkflowName/xniu-etl-janitor%20WorkflowRun/2023.05.11-0900-1dl2q2x93uasx%20WorkflowTask/mark-from-roo
* ts%20WorkflowJob/yiqiang-4d-mlmcf-v50-no-egomotion-3task-train/2023.01.10-0016-1ngxgf4b4qtaj nvetl%2CAUTH_tk2b6... - - - tx09dd2f1911ba4d848a0a5-0064769a3e - 0.0051 - - 1685494334.911721706 1685494334.916850328 0 - AUTH_nvetl nvetl-data-p
* rod cache/data/01/da/01dac740ea18b1728590398db523314039422209 200
* 2023-05-30T20:52:15.552481-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/logging/handlers.py", line 934, in emit#012 self.socket.send(msg)
* 2023-05-30T20:52:15.552616-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.29 100.65.4.29 31/May/2023/00/52/14 HEAD /v1/AUTH_nvetl/nvetl-data-prod/cache/data/c8/15/c8153550762b0dce5bbaf58b1df4993dd4395c86 HTTP/1.0 200 - NVIDIA/1
* .0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/mark%20Lib/com.nvidia.corp.go.maglev.src.common.swiftclient%20Git/e3dd7d3c6%20WorkflowName/xniu-etl-janitor%20WorkflowRun/2023.05.11-0900-1dl2q2x93uasx%20WorkflowTask/mark-from-roots%20WorkflowJob/yiqiang-4d-mlmcf-v50-no-egomotion-3task-train/2023.01.10-0016-1ngxgf4b4qtaj nvetl%2CAUTH_tk2b6... - - - txee5a3f24d5074c9094e3a-0064769a3e - 0.0245 - - 1685494334.682982445 1685494334.707516909 0 - AUTH_nvetl nvetl-data-prod cache/data/c8/15/c8153550762b0dce5bbaf58b1df4993dd4395c86 200
* 2023-05-30T20:52:15.553568-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/eventlet/greenio/base.py", line 401, in send#012 return self._send_loop(self.fd.send, data, flags)
* 2023-05-30T20:52:15.553700-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.6 100.65.4.6 31/May/2023/00/52/15 GET /v1/AUTH_udc/data-lake-maglev-nvidia-cn%3Fdelimiter%3D%252F%26limit%3D2%26prefix%3Ddrivedata%252Fgps%252Fparquet%252Fsession_id%253Df478b4dc-1d59-11ed-999a-00044bf65faf%252Fsensor_name%253Dgps_novatel_pp%252Fupload_date%253D2022-08-23%252F%26format%3Djson%26states%3Dlisting HTTP/1.0 200 - Hadoop%203.2.1%2C%20aws-sdk-java/1.11.655%20Linux/5.4.0-81-generic%20OpenJDK_64-Bit_Server_VM/25.252-b09%20java/1.8.0_252%20scala/2.12.10%20vendor/Oracle_Corporation udc%2Cs3 - 398 - tx8d356017ced0420989355-0064769a3f - 0.0121 S3 - 1685494335.139057398 1685494335.151184559 0 0.012038707733154297 AUTH_udc data-lake-maglev-nvidia-cn - 200
* 2023-05-30T20:52:15.554633-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.12 100.65.4.12 31/May/2023/00/52/14 HEAD /v1/AUTH_nvetl/nvetl-data-prod/cache/data/5b/f1/5bf1d9f731653d6e1cc23ebc31e32272d8e59a8c HTTP/1.0 200 - NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/mark%20Lib/com.nvidia.corp.go.maglev.src.common.swiftclient%20Git/e3dd7d3c6%20WorkflowName/xniu-etl-janitor%20WorkflowRun/2023.05.11-0900-1dl2q2x93uasx%20WorkflowTask/mark-from-roots%20WorkflowJob/yiqiang-4d-mlmcf-v50-no-egomotion-3task-train/2023.01.10-0016-1ngxgf4b4qtaj nvetl%2CAUTH_tk2b6... - - - txc5afda2d05a7466b82d92-0064769a3e - 0.0496 - - 1685494334.883673906 1685494334.933285236 0 - AUTH_nvetl nvetl-data-prod cache/data/5b/f1/5bf1d9f731653d6e1cc23ebc31e32272d8e59a8c 200
* 2023-05-30T20:52:15.555574-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.14 100.65.4.14 31/May/2023/00/52/15 HEAD /v1/AUTH_udc/data-lake-maglev-nvidia-cn/drivedata/gps/parquet/session_id%253Dee0725f8-1e0c-11ed-bc4f-00044bf66018/sensor_name%253Dgps_ublox_f9k/upload_date%253D2022-08-22 HTTP/1.0 404 - Hadoop%203.2.1%2C%20aws-sdk-java/1.11.655%20Linux/5.4.0-81-generic%20OpenJDK_64-Bit_Server_VM/11.0.10%2B9%20java/11.0.10%20scala/2.12.10%20vendor/Oracle_Corporation udc%2Cs3 - - - tx42faeb855aea44e284272-0064769a3f - 0.0164 S3 - 1685494335.131848097 1685494335.148231745 0 - AUTH_udc data-lake-maglev-nvidia-cn drivedata/gps/parquet/session_id%253Dee0725f8-1e0c-11ed-bc4f-00044bf66018/sensor_name%253Dgps_ublox_f9k/upload_date%253D2022-08-22 404
* 2023-05-30T20:52:15.556474-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/eventlet/greenio/base.py", line 396, in _send_loop#012 timeout_exc=_timeout_exc)
* 2023-05-30T20:52:15.556638-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.6 100.65.4.6 31/May/2023/00/52/15 GET /data-lake-maglev-nvidia-cn/%3Flist-type%3D2%26delimiter%3D%252F%26max-keys%3D1%26prefix%3Ddrivedata%252Fgps%252Fparquet%252Fsession_id%253Df478b4dc-1d59-11ed-999a-00044bf65faf%252Fsensor_name%253Dgps_novatel_pp%252Fupload_date%253D2022-08-23%252F%26fetch-owner%3Dfalse HTTP/1.0 200 - Hadoop%203.2.1%2C%20aws-sdk-java/1.11.655%20Linux/5.4.0-81-generic%20OpenJDK_64-Bit_Server_VM/25.252-b09%20java/1.8.0_252%20scala/2.12.10%20vendor/Oracle_Corporation - - 733 - tx8d356017ced0420989355-0064769a3f - 0.0176 - - 1685494335.135771275 1685494335.153389931 0 0.017529726028442383 AUTH_udc data-lake-maglev-nvidia-cn - 200
* 2023-05-30T20:52:15.557512-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.14 100.65.4.14 31/May/2023/00/52/15 HEAD /data-lake-maglev-nvidia-cn/drivedata/gps/parquet/session_id%253Dee0725f8-1e0c-11ed-bc4f-00044bf66018/sensor_name%253Dgps_ublox_f9k/upload_date%253D2022-08-22 HTTP/1.0 404 - Hadoop%203.2.1%2C%20aws-sdk-java/1.11.655%20Linux/5.4.0-81-generic%20OpenJDK_64-Bit_Server_VM/11.0.10%2B9%20java/11.0.10%20scala/2.12.10%20vendor/Oracle_Corporation - - - - tx42faeb855aea44e284272-0064769a3f - 0.0256 - - 1685494335.128760099 1685494335.154312849 0 - AUTH_udc data-lake-maglev-nvidia-cn drivedata/gps/parquet/session_id%253Dee0725f8-1e0c-11ed-bc4f-00044bf66018/sensor_name%253Dgps_ublox_f9k/upload_date%253D2022-08-22 404
* 2023-05-30T20:52:15.558338-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/eventlet/greenio/base.py", line 213, in _trampoline#012 mark_as_closed=self._mark_as_closed)
* 2023-05-30T20:52:15.558464-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/eventlet/hubs/__init__.py", line 138, in trampoline#012 assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
* 2023-05-30T20:52:15.558618-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: AssertionError: do not call blocking functions from the mainloop
* 2023-05-30T20:52:15.558788-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: Call stack:
* 2023-05-30T20:52:15.558925-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/eventlet/hubs/hub.py", line 365, in run#012 self.wait(sleep_time)
* 2023-05-30T20:52:15.559089-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/eventlet/hubs/poll.py", line 97, in wait#012 for fileno, event in presult:
* 2023-05-30T20:52:15.559267-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.19 100.65.4.19 31/May/2023/00/52/15 GET /nvetl-data-prod/cache/data/35/32/35321a5e08243c857bf6cdefe556814bbf631a88 HTTP/1.0 206 - NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/etl-agent%20Lib/com.nvidia.corp.go.maglev.src.common.yarofs%20Git/90cfd2cf2%20WorkflowName/yichengz-train-cn-v009-exp2-2023-05-29%20WorkflowRun/2023.05.29-1503-0u45k5ueqevnb%20WorkflowTask/%20WorkflowJob/ - - 8388608 - txbee16dfb5eec4c64ad8ec-0064769a3f Range:%20bytes%3D612368384-620756991%0AX-Maglev-Workflow:%20yichengz-train-cn-v009-exp2-2023-05-29/2023.05.29-1503-0u45k5ueqevnb 0.0540 - - 1685494335.093349457 1685494335.147353649 0 0.02495861053466797 AUTH_nvetl nvetl-data-prod cache/data/35/32/35321a5e08243c857bf6cdefe556814bbf631a88 206
* 2023-05-30T20:52:15.560240-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: File "/opt/ss/lib/python3.7/site-packages/swift/proxy/controllers/base.py", line 1395, in _get_response_parts_iter#012 self.path)
* 2023-05-30T20:52:15.560411-04:00 ssnode-proxy-1202-f09-2 proxy-server: err STDERR: Message: 'Client disconnected on read of %r'#012Arguments: ('/AUTH_udc/data-lake-maglev-nvidia-cn/drivedata/gps/parquet/session_id=d867d9bc-297a-11ed-a1cc-00044bf66018/sensor_name=gps_novatel_pp/upload_date=2022-09-09/sensor_357472160.parquet',)
* 2023-05-30T20:52:15.560543-04:00 ssnode-proxy-1202-f09-2 proxy-server: info 100.65.4.26 100.65.4.26 31/May/2023/00/52/15 HEAD /v1/AUTH_nvetl/nvetl-data-prod/cache/data/2d/43/2d437830f11f4a9e3d356900eba3553bb7c2b7ea HTTP/1.0 200 - NVIDIA/1.0%20%28Linux%3B%20arch:amd64%3B%20lang:go1.18.2%29%20Cmd/mark%20Lib/com.nvidia.corp.go.maglev.src.common.swiftclient%20Git/e3dd7d3c6%20WorkflowName/xniu-etl-janitor%20WorkflowRun/2023.05.11-0900-1dl2q2x93uasx%20WorkflowTask/mark-from-roots%20WorkflowJob/yiqiang-4d-mlmcf-v50-no-egomotion-3task-train/2023.01.10-0016-1ngxgf4b4qtaj nvetl%2CAUTH_tk2b6... - - - tx9b248926097a4aa4b6ba2-0064769a3f - 0.0618 - - 1685494335.081943274 1685494335.143737793 0 - AUTH_nvetl nvetl-data-prod cache/data/2d/43/2d437830f11f4a9e3d356900eba3553bb7c2b7ea 200
it is possible to extract the traceback as a new field?
I have tried to use multiline, but the traceback is logged together