Elasticsearch 8.12 Error when tryng to generate a report using curl and an URL given by Discovery tool

Elastic Stack Elasticsearch
1

Hi guys.
I am tryng to generate a csv report using the following command

curl -uelastic:xxxxxx -X POST \
>   -H "kbn-xsrf: true" \
        "from": "now-12h"
      },
>   -H "Content-Type: application/json" \
  --cacert /etc/kibana/elasticsearch_accusys_com_ar.crt \
>   -d '{
>     "jobParams": {
>       {
>   "objectType": "url",
>   "relativeUrls": [
>     {
>       "url": "/app/r",
>       "timeRange": {
>         "to": "now",
>         "from": "now-12h"
>       },
>       "queryString": "l=DISCOVER_APP_LOCATOR&v=8.12.0&lz=N4IgjgrgpgTgniAXKSsGJCANCANgQwDsBzCfYqJEAa2nhAF8cBnAexgBckBtbkAAQ4BLALZRmHfCIAO2EABNxAYxABdVTiWtcEEYWY8QhVvNZyARvmaUWsllFKFTanEKdQAHlQBMAFgCsAIwA7PIAnPIAtPLyAGa%2Bkb5hYbGRAByx5lF%2Bwd5QAMzyaVAADPhhcm4csABu%2BLhU%2BBAcZjixQrjVMAaI3BoK%2BJIAakJQAO4AkvI%2BASHhUTHxicmpGVmROXmFxWUVOMJiAEpEFEigsTCsIlTGY5GB3gAWci03rGOMODBQF%2BKPE4QunUGsgQMDoEhfCVoSUcNImtYkLF6tYGAwgA%3D"
>     }
>   ]
> }
>   }' \
>   --cacert /etc/kibana/elasticsearch_accusys_com_ar.crt \
>   "https://elasticsearch.accusys.com.ar:5601/api/reporting/generate/csv"

And get the following error

{"statusCode":400,"error":"Bad Request","message":"Invalid request payload JSON format"}

When I try to generate it using Kibana, it freezes.
Please some help with this issue.
Thanks in advance.

2

I'm not sure but this looks weird:

  -H "kbn-xsrf: true" \
    "from": "now-12h"
  },
3

Hi dadoonet
I changed a little bit the command as you can see

curl -u administrador:xxxx -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "jobParams": {
        "objectType": "url",
        "timeRange": {
          "from": "2023-09-20"
          "to":"  "2023-09-20"
        },
        "queryString": "l=DISCOVER_APP_LOCATOR&v=8.12.0&lz=N4IgjgrgpgTgniAXKSsGJCANCANgQwDsBzCfYqJEAa2nhAF8cBnAexgBckBtbkAAQ4BLALZRmHfCIAO2EABNxAYxABdVTiWtcEEYWY8NIIYUUAPKgCYALAFYAjAHZ5ATnkBaefIBm199ZcXb3cADm8AIw8bR0soAGZ5EKgABnwXORMOWAA3fFwqfAgOVjlvIVwsmANEbiN5fEkANSEoAHcASXkrOydXDy9ff0DgsMj3aNiEpNT0nGExACUiCiRQbxhWEStkyzj3ZJdx5IAVZLjEZOSL5IA6S%2BSALTli7d39w8t7U8tEOKvLu6XJ5MEAwKDrcQAC3ahEquXyyBA8OgSGs92SOGkhWYlEQ3jyOIYDCAA%3D%3D"
      }
    }' \
    --cacert /etc/kibana/elasticsearch_accusys_com_ar.crt \
    "https://elasticsearch.accusys.com.ar:5601/api/reporting/generate/csv"

I am searching data in a time range "2024-09-20".
I made sure that the user "administrador" has permission to generate csv report and the sentence is ommited

`-H "kbn-xsrf: true" \

But, still get the error

{"statusCode":400,"error":"Bad Request","message":"Invalid request payload JSON format"}

Please, another advice?
Thanks in advance

4

Hi guys.
I solved the issue developing a program with python.
Thanks any way.

5

Hi @RobertC1 I'm glad you solved the problem here. Just FYI: I see a missing comma at the end of the line starting with "from": "2023-09-20", turning this into invalid JSON. This is likely the cause.

1 Like
6

Hi Sander_Philipse
Thanks for your help, nevertheless still get same error.

Any other suggestions would be greatly appreciated.

curl -u administrador:xxxxxx -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "jobParams": {
        "objectType": "url",
        "timeRange": {
          "from": "2023-09-20",
          "to":"  "2023-09-20"
        },
        "queryString": "l=DISCOVER_APP_LOCATOR&v=8.12.0&lz=N4IgjgrgpgTgniAXKSsGJCANCANgQwDsBzCfYqJEAa2nhAF8cBnAexgBckBtbkAAQ4BLALZRmHfCIAO2EABNxAYxABdVTiWtcEEYWY8NIIYUUAPKgCYALAFYAjAHZ5ATnkBaefIBm199ZcXb3cADm8AIw8bR0soAGZ5EKgABnwXORMOWAA3fFwqfAgOVjlvIVwsmANEbiN5fEkANSEoAHcASXkrOydXDy9ff0DgsMj3aNiEpNT0nGExACUiCiRQbxhWEStkyzj3ZJdx5IAVZLjEZOSL5IA6S%2BSALTli7d39w8t7U8tEOKvLu6XJ5MEAwKDrcQAC3ahEquXyyBA8OgSGs92SOGkhWYlEQ3jyOIYDCAA%3D%3D"
      }
    }' \
    --cacert /etc/kibana/elasticsearch_accusys_com_ar.crt \
    "https://elasticsearch.accusys.com.ar:5601/api/reporting/generate/csv"


{"statusCode":400,"error":"Bad Request","message":"Invalid request payload JSON format"}
7

Could you share what you did to solve the problem?

8

Where did you get this json example from?

From the documentation, I can see:

curl \
-XPOST \ 
-u elastic \ 
-H 'kbn-xsrf: true' \ 
'http://0.0.0.0:5601/api/reporting/generate/csv?jobParams=...'

So no json provided here...

9

Hi dadoonet
Thanks for the advice, and I researched about the format of the command.
The json command suggested was executed, but with a correction

curl -XPOST \
-u elastic:xxxx \
-H 'kbn-xsrf: true' \
'http://172.28.194.109:5601/api/reporting/generate/csv?jobParams=https://elasticsearch.accusys.com.ar:5601/app/r?l=DISCOVER_APP_LOCATOR&v=8.12.0&lz=N4IgjgrgpgTgniAXKSsGJAOwPYBNuIAEAOiAAwCMphIANCADYCGmA5hE61EiANbTwQAX3oBnbDAAuSANoyQAAUkBLALZRRkpqoAOdELg0BjEAF1T9I9gYRVmUbKx5s%2B0XvoAjJqO4WQyzEMADx4AdiMPCIBWAE4yAFpQgDMADgA2eIoKKCN4jyyPeKTcACYyAGZyqCSyD1DuegDJWAA3JgYeAAt9JOUGZpgHRBk%2FXCYtADVlKAB3AElcMIjouMTUjKycvIKi0oqqmrqGkBV1ACUWLiRQJJhsVR4ykvL4shjXqIAVCjJEMl%2F%2FgA6f5kABa%2BkkLgwTxebw%2B3wofwBZGB%2F3BIhAMGqWNEnTmmAGbQ6yBAROgSDKIPoOiYEB8SEkMGgQiEQA'

And get the following error

curl: (52) Empty reply from server

Please, any advice?

About your question "Where did you get this json example from?", doing a research job.
Thanks

10

Hi dadoonet

The python

from elasticsearch import Elasticsearch
from datetime import datetime

# Elasticsearch URL
ELASTICSEARCH_URL = "http://localhost:9200"

# Elasticsearch username and password
USERNAME = "elastic"
PASSWORD = "xxxx"

# Create Elasticsearch client with basic authentication
es = Elasticsearch([ELASTICSEARCH_URL], basic_auth=(USERNAME, PASSWORD))
# Get the current date and time

# Check if the connection is successful
if not es.ping():
    print("Could not connect to Elasticsearch")
    exit()

start_date = "2023-09-05T00:00:00"
end_date = "2023-09-05T23:59:59"
# Define the date range
# Initial search query with date range
query = {
    "size": 10000,  # Number of documents to retrieve per scroll
    "query": {
        "bool": {
            "must": [
                {
                    "range": {
                        "@timestamp": {
                            "gte": start_date,
                            "lte": end_date
                        }
                    }
                }
            ]
        }
    },
    "sort": [
        {"nodo": {"order": "asc"}}
    ]
}
# Execute initial search request with scroll
response = es.search(index="trx_hours_history", body=query, scroll="1s")

# Store unique document identifiers to avoid duplicates
seen_ids = set()

#print(f'{fecha},"{nodo_str}",{base},{sp},{segundo_div:.4f},"{cant_format}"')
# Process initial search response
scroll_id = response["_scroll_id"]
hits = response["hits"]["hits"]
total_hits = response["hits"]["total"]["value"]

# Process the first batch of results
for hit in hits:
    source = hit["_source"]
    doc_id = hit["_id"]  # Get document ID
    # Check if the document ID has been seen before
    if doc_id not in seen_ids:
        seen_ids.add(doc_id)
        fecha = source.get("@timestamp")
        nodo = source.get("nodo")
        nodo_str = str(nodo).zfill(2)
        base = source.get("base")
        sp = source.get("sp")
        cant_trn = source.get("cant_trn")
        cant_format = "{:,}".format(cant_trn)
        segundo_prom = source.get("segundo_prom")
        if segundo_prom is not None:
            segundo_div = segundo_prom / 1000
        print(f'{fecha},"{nodo_str}",{base},{sp},{segundo_div:.4f},"{cant_format}"')
1 Like
11

Thanks for sharing your code. I think it could be super useful for the community. If you don't mind, I'd mark it as a solution for your post so people can get inspiration from it.

12

Do you remember which page did you find?

13

dadoonet

Thanks for your words and I am really glad to help the community.
Best regards.

14

dadoonet
I don't really remember, I believe it was a mix of a youtube video and chatgpt.
It was sunday and late and when I found the python solution, I didn't insist with the curl command.
Best Regards.