SQL query on indices imported by OTEL Collector and Elastic APM

Elastic Stack Elasticsearch
1

When OTEL collector sends traces, logs and metrics to elastic APM
It got stored with indices names like

GET _cat/indices

yellow open .ds-logs-apm.app-default-2023.02.27-000001         ciiMrei8TLmJ1YilCaZy_A 1 1    96 0  69.1kb  69.1kb
yellow open .ds-traces-apm-default-2023.02.27-000001           dXXs_8ZNTD6k6G7OoaKDvQ 1 1   308 0 211.8kb 211.8kb
yellow open .ds-metrics-apm.internal-default-2023.02.27-000001 JAjp-LvFRU-SIT52Sz_97Q 1 1     7 0  23.1kb  23.1kb

Elasticsearch has generated Data Streams on these indices

GET /_data_stream

This gives me following streams corresponding to above indices
traces-apm-default
logs-apm.app-default
metrics-apm.internal-default

If I try to execute SQL query on these indices and data streams, I get "parsing_exception":

GET /_sql
{
  "query": "select * from .ds-logs-apm.app-default-2023.02.27-000001" 
}

GET /_sql
{
  "query": "select * from traces-apm-default" 
}

Question here is could we use SQL syntax on these indices? or only EQL is allowed? :question:

Thanks!!

2

Welcome to our community! :smiley:

Can you post the entire response you are getting from Elasticsearch?

3

Thank you!!

Please refer following:

_sql query on log/trace index

GET /_sql?format=txt
{
  "query": "select * from .ds-logs-apm.app-default-2023.02.27-000001" 
}

response:

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "line 1:15: extraneous input '.' expecting {'(', 'ANALYZE', 'ANALYZED', 'CATALOGS', 'COLUMNS', 'CURRENT_DATE', 'CURRENT_TIME', 'CURRENT_TIMESTAMP', 'DAY', 'DEBUG', 'EXECUTABLE', 'EXPLAIN', 'FIRST', 'FORMAT', 'FROZEN', 'FULL', 'FUNCTIONS', 'GRAPHVIZ', 'HOUR', 'INTERVAL', 'LAST', 'LIMIT', 'MAPPED', 'MINUTE', 'MONTH', 'OPTIMIZED', 'PARSED', 'PHYSICAL', 'PIVOT', 'PLAN', 'RLIKE', 'QUERY', 'SCHEMAS', 'SECOND', 'SHOW', 'SYS', 'TABLES', 'TEXT', 'TOP', 'TYPE', 'TYPES', 'VERIFY', 'YEAR', IDENTIFIER, DIGIT_IDENTIFIER, TABLE_IDENTIFIER, QUOTED_IDENTIFIER, BACKQUOTED_IDENTIFIER}"
      }
    ],
    "type": "parsing_exception",
    "reason": "line 1:15: extraneous input '.' expecting {'(', 'ANALYZE', 'ANALYZED', 'CATALOGS', 'COLUMNS', 'CURRENT_DATE', 'CURRENT_TIME', 'CURRENT_TIMESTAMP', 'DAY', 'DEBUG', 'EXECUTABLE', 'EXPLAIN', 'FIRST', 'FORMAT', 'FROZEN', 'FULL', 'FUNCTIONS', 'GRAPHVIZ', 'HOUR', 'INTERVAL', 'LAST', 'LIMIT', 'MAPPED', 'MINUTE', 'MONTH', 'OPTIMIZED', 'PARSED', 'PHYSICAL', 'PIVOT', 'PLAN', 'RLIKE', 'QUERY', 'SCHEMAS', 'SECOND', 'SHOW', 'SYS', 'TABLES', 'TEXT', 'TOP', 'TYPE', 'TYPES', 'VERIFY', 'YEAR', IDENTIFIER, DIGIT_IDENTIFIER, TABLE_IDENTIFIER, QUOTED_IDENTIFIER, BACKQUOTED_IDENTIFIER}"
  },
  "status": 400
}

_sql query on log/trace data-stream

GET /_sql?format=txt
{
  "query": "select * from traces-apm-default" 
}

response:

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "line 1:21: mismatched input '-' expecting {<EOF>, ',', 'ANALYZE', 'ANALYZED', 'AS', 'CATALOGS', 'COLUMNS', 'CURRENT_DATE', 'CURRENT_TIME', 'CURRENT_TIMESTAMP', 'DAY', 'DEBUG', 'EXECUTABLE', 'EXPLAIN', 'FIRST', 'FORMAT', 'FULL', 'FUNCTIONS', 'GRAPHVIZ', 'GROUP', 'HAVING', 'HOUR', 'INNER', 'INTERVAL', 'JOIN', 'LAST', 'LEFT', 'LIMIT', 'MAPPED', 'MINUTE', 'MONTH', 'NATURAL', 'OPTIMIZED', 'ORDER', 'PARSED', 'PHYSICAL', 'PIVOT', 'PLAN', 'RIGHT', 'RLIKE', 'QUERY', 'SCHEMAS', 'SECOND', 'SHOW', 'SYS', 'TABLES', 'TEXT', 'TOP', 'TYPE', 'TYPES', 'VERIFY', 'WHERE', 'YEAR', LIMIT_ESC, IDENTIFIER, DIGIT_IDENTIFIER, QUOTED_IDENTIFIER, BACKQUOTED_IDENTIFIER}"
      }
    ],
    "type": "parsing_exception",
    "reason": "line 1:21: mismatched input '-' expecting {<EOF>, ',', 'ANALYZE', 'ANALYZED', 'AS', 'CATALOGS', 'COLUMNS', 'CURRENT_DATE', 'CURRENT_TIME', 'CURRENT_TIMESTAMP', 'DAY', 'DEBUG', 'EXECUTABLE', 'EXPLAIN', 'FIRST', 'FORMAT', 'FULL', 'FUNCTIONS', 'GRAPHVIZ', 'GROUP', 'HAVING', 'HOUR', 'INNER', 'INTERVAL', 'JOIN', 'LAST', 'LEFT', 'LIMIT', 'MAPPED', 'MINUTE', 'MONTH', 'NATURAL', 'OPTIMIZED', 'ORDER', 'PARSED', 'PHYSICAL', 'PIVOT', 'PLAN', 'RIGHT', 'RLIKE', 'QUERY', 'SCHEMAS', 'SECOND', 'SHOW', 'SYS', 'TABLES', 'TEXT', 'TOP', 'TYPE', 'TYPES', 'VERIFY', 'WHERE', 'YEAR', LIMIT_ESC, IDENTIFIER, DIGIT_IDENTIFIER, QUOTED_IDENTIFIER, BACKQUOTED_IDENTIFIER}",
    "caused_by": {
      "type": "input_mismatch_exception",
      "reason": null
    }
  },
  "status": 400
}
1 Like
4

It complaint on chars like '.' and '-'.
Generally in SQL we add quotes if we have table name with '-' --> I had tried adding different quotes around index / data-stream name but no luck.

5

@warkolm I have provided response from elastic. Could you please suggest.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.