[Python] How to send logs to Elasticsearch?

Hi Guys, I got some questions. Here is my use case to be solved.

Python version: 3.11.4
Elastic version: 8.11.3
Python packages:
fastapi = "0.103.1"
elastic-apm = "6.23.0"

I am maintaining a Python FastAPI server which is using APM package "elastic-apm 6.23.0" to trace API spans.
Before, we had used standard library "logging.FileHandler" to record logs in a log file locally then transfer logs from local to Elasticsearch by Elastic Agent. Because we mark tracing information tailing the logs, we can directly see correlated logs under the logs tab.

And now we are trying to transfer logs without writing local file.
Thus, I tried "elasticapm.handlers.logging.LoggingHandler" to send logs to APM server. I also can see correlated logs on Kibana APM UI.

However, the function "capture" hardcoded "error" category, which is one of the distinguishable category of APM server, and cause NULL type error being showed on APM Error tab.


This confused me. I just want to make correlated logs showed alongside transaction/tracing tab.

I also tried other methods to achieve that but not really worked till now.
Like directly sending messages to Elasticsearch via HTTP calls, messages only can be found in Kibana Discover, couldn't be visualized in Kibana Logs and APM logs tab.

Does anyone have ideas about this?

I moved your question to the APM forum because I think it needs an answer from the elasticapm team.

HI @Shane_Chang , Welcome to the Elastic community.

A note from the doc -

Elastic Python APM Agent does not send the logs to Elasticsearch. It only injects correlation IDs and reformats the logs. You must use another ingestion strategy. We recommend Filebeat for that purpose.

trace.id, transcation.id & span.id is getting injected in your logs? If you can share the snippet.

Also you can give a try to structlog which will add above ids.

Once it visible in your file log you need to use elastic agent or filebeat to read the same logs and push to Elasticsearch.