Can elasticsearch/kibana/... export trace/span log as datasets?

Elastic Stack Elasticsearch
1

I'm a freshman here. I'm using skywalking+elasticsearch monitoring a microservice application in k8s. l know skywalking can collect trace/span logs from microservice and store the data in elasticsearch, and then display trace/span log on skywalking UI, but l don't know how to export the data stored in elasticsearch, is there anyway can accomplish that? thanks!
by the way, i want span log like this:

{
  "trace_id": "7bba9f33312b3dbb8b2c2c62bb7abe2d",
  "parent_id": "",
  "span_id": "086e83747d0e381e",
  "name": "/v1/sys/health",
  "start_time": "2021-10-22 16:04:01.209458162 +0000 UTC",
  "end_time": "2021-10-22 16:04:01.209514132 +0000 UTC",
  "status_code": "STATUS_CODE_OK",
  "status_message": "",
  "attributes": {
    "net.transport": "IP.TCP",
    "net.peer.ip": "172.17.0.1",
    "net.peer.port": "51820",
    "net.host.ip": "10.177.2.152",
    "net.host.port": "26040",
    "http.method": "GET",
    "http.target": "/v1/sys/health",
    "http.server_name": "mortar-gateway",
    "http.route": "/v1/sys/health",
    "http.user_agent": "Consul Health Check",
    "http.scheme": "http",
    "http.host": "10.177.2.152:26040",
    "http.flavor": "1.1"
  },
  "events": [
    {
      "name": "",
      "message": "OK",
      "timestamp": "2021-10-22 16:04:01.209512872 +0000 UTC"
    }
  ]
}

trace log like this:

{
    "name": "Hello-Greetings",
    "context": {
        "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2",
        "span_id": "0x5fb397be34d26b51",
    },
    "parent_id": "0x051581bf3cb55c13",
    "start_time": "2022-04-29T18:52:58.114304Z",
    "end_time": "2022-04-29T22:52:58.114561Z",
    "attributes": {
        "http.route": "some_route1"
    },
    "events": [
        {
            "name": "hey there!",
            "timestamp": "2022-04-29T18:52:58.114561Z",
            "attributes": {
                "event_attributes": 1
            }
        },
        {
            "name": "bye now!",
            "timestamp": "2022-04-29T18:52:58.114585Z",
            "attributes": {
                "event_attributes": 1
            }
        }
    ],
}
{
    "name": "Hello-Salutations",
    "context": {
        "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2",
        "span_id": "0x93564f51e1abe1c2",
    },
    "parent_id": "0x051581bf3cb55c13",
    "start_time": "2022-04-29T18:52:58.114492Z",
    "end_time": "2022-04-29T18:52:58.114631Z",
    "attributes": {
        "http.route": "some_route2"
    },
    "events": [
        {
            "name": "hey there!",
            "timestamp": "2022-04-29T18:52:58.114561Z",
            "attributes": {
                "event_attributes": 1
            }
        }
    ],
}
{
    "name": "Hello",
    "context": {
        "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2",
        "span_id": "0x051581bf3cb55c13",
    },
    "parent_id": null,
    "start_time": "2022-04-29T18:52:58.114201Z",
    "end_time": "2022-04-29T18:52:58.114687Z",
    "attributes": {
        "http.route": "some_route3"
    },
    "events": [
        {
            "name": "Guten Tag!",
            "timestamp": "2022-04-29T18:52:58.114561Z",
            "attributes": {
                "event_attributes": 1
            }
        }
    ],
}
2

Export the data in what format?. for what purpose?

Elasticsearch just has a REST endpoint so you certainly can GET or query the documents they will be in JSON.

And there are tools to help export (language clients, logstash etc ) depending on what you want to do... So what do you want to do / accomplish.

3

Here is my consideration:
let suppose the data in JSON format. I want to collect and export the trace/span log to form a dataset during the operation of a microservice, with skywalking to monitor and elastic to store. Then the dataset can be used to analysis, like root cause analysis/anomaly analysis/data mining/...

4

Apologies I still don't understand... If you put the data in elasticsearch it can accessed and returned in JSON 1 document or 1Billion documents they are all there..

Perhaps leave the in elasticsearch and do analysis on elasticsearch.. if you use OTEL or the free elasticsearch APM Agents the built in APM app in Kibana will work!

Or even with your current agents or others elasticsearch is a powerful analytic engine.

The analytics traces, aggregations, min, max, averages,Top N bottom N and many many more are available.

Otherwise just use the REST endpoints and query the data you wish.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.