In Elasticsearch, data after a certain date is not coming through

Elastic Stack Elasticsearch
, ,
1

Problem:

My application is running on Docker, and I am using Elasticsearch. The index has been created. When I make a curl request, I get a response, but in my application, data after March 5th is not showing up. However, when I make a curl request, I can get a response.

Details:

The Elasticsearch container is up and running, and I sent a request using curl, receiving results for all dates. However, I am not getting a response through the application.

The logs show the following:

[Nest] 41 - 03/18/2025, 8:50:17 AM   DEBUG [new] Indexing has started!
[Nest] 41 - 03/18/2025, 8:50:17 AM    WARN [Elastic] Found indices!
[Nest] 41 - 03/18/2025, 8:50:17 AM    WARN [Elastic] Removing old indices..
[Nest] 41 - 03/18/2025, 8:50:18 AM     LOG [Elastic] Creating new Index: [index_name]
[Nest] 41 - 03/18/2025, 8:50:18 AM     LOG [Elastic] New Index: [index_name] created!
2

Hello!
Which language is your application written in, and which client are you using?

Also to be sure the problem is clear: you're calling the same Elasticsearch docker instance first using a client, then using curl, performing the same query and obtaining different results? Could we get more details on the query performed?

1 Like
3

hello , written in node js. ı can get data through elasticsearch by using curl but when ı send request from backend there are no datas after march.

4

yes exaclty during curl request everything ok but client side request nonew data after 5 march

5

Could you please show us your Node.js code that runs the query, as well as the curl request you're using? Thanks so much!

1 Like
6 
    async search(req: Request, limit: number) {
        const zoneResponse = await this.getZoneIDS(req.user)
        const zones = !zoneResponse ? [] : zoneResponse.map(zone => zone.zone_id)
        const input = req.query.q
        let body = {
            size: limit,
            from: 0,
            query: {
                bool: {
                    must: [
                        {
                            query_string: {
                                query: `*${input}*`,
                                fields: ["ilad", "ilcead", "mahad"],
                                default_operator: "AND",

                            },
                        },
                        // { terms: { gunid: "GECE" } }

                    ],
                    // "filter": {
                    //     "geo_bounding_box": {
                    //         "geom": {
                    //             "wkt": "BBOX (28.952045, 28.997099, 41.017083, 41.002053)"
                    //         }
                    //     }
                    // }
                },

            },

            sort: [{
                "kazatarihi": {
                    "order": "desc"
                }
            }] as any
        }
        if (zones.length >= 1) {
            body.query.bool.must.push({ terms: { zoneid: zones } } as any)
        }
        for (const key in req.query) {
            // let query = req.query[key]
            // switch (true) {
            //     case req.query[key] !== "" && key !== "q":
            //         console.log("burda");
            //         switch (key) {
            //             case "range":
            //                 body.query.bool.must.push({ range: { kazatarihi: { gte: req.query[key].toString().split(':')[0], lte: req.query[key].toString().split(':')[1] } } } as any)
            //                 break;
            //             case "arac_say" || "olu_say" || "yarali_say":
            //                 body.query.bool.must.push({ range: { [key]: { gte: req.query[key].toString().split(':')[0], lte: req.query[key].toString().split(':')[1] } } } as any)
            //             default:
            //                 body.query.bool.must.push({
            //                     match: {
            //                         [key]: {
            //                             query: req.query[key],
            //                             operator: "and"
            //                         }
            //                     }
            //                 } as any)
            //                 break;
            //         }
            //         break;
            //     default:
            //         throw new AgElasticError("Filteration Error")
            // }
            if (req.query[key] !== "" && key !== "q") {
                if (key === "range") {
                    body.query.bool.must.push({ range: { kazatarihi: { gte: req.query[key].toString().split(':')[0], lte: req.query[key].toString().split(':')[1] } } } as any)
                } else if (key === "arac_say" || key === "olu_say" || key === "yarali_say") {
                    body.query.bool.must.push({ range: { [key]: { gte: req.query[key].toString().split(':')[0], lte: req.query[key].toString().split(':')[1] } } } as any)
                    // } else if (key === "gunid") {
                    //     body.query.bool.must.push({ terms: { gunid: ["GECE", "ALACAKARANLIK"] } } as any)
                } else {
                    if (Array.isArray(req.query[key])) {
                        body.query.bool.must.push({ terms: { [key]: req.query[key] } } as any)
                    } else {
                        body.query.bool.must.push({
                            match: {
                                [key]: {
                                    query: req.query[key],
                                    operator: "and"
                                }
                            }
                        } as any)
                    }
                }
            }
        }
        for (const key in req.body) {
            if (key === "geo_shape") {
                body.query.bool["filter"] = { [key]: req.body[key] }
            }
            else if (key === "geo_distance") {
                body.sort.splice(0, 1)
                body.sort.push({
                    _geo_distance: {
                        geom: req.body[key].geom,
                        order: "asc",
                        unit: "km"
                    }
                })
                body.size = 1
                body.query.bool["filter"] = { [key]: req.body[key] }
            }
            else if (key === "geo_bounding_box") {
                body.query.bool["filter"] = { [key]: req.body[key] }

            } else {
                throw new AgElasticError("Lütfen filtre gövdenizi değiştiriniz!")
            }
        }

        //console.log(body.query.bool.must[1]["match"])
        return await client.search({
            index: 'kaza',
            body: body

        })
    }```





curl -k -u "elastic:trafik" "https://172.17.31.211:9200/kaza/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "range": {
      "kazatarihi": {
        "gte": "2025-03-21T00:00:00.000Z",
        "lte": "2025-03-24T23:59:59.999Z",
        "format": "strict_date_optional_time"
      }
    }
  },
  "size": 10
}'
7

There's quite a big difference between your curl query and your JavaScript one. The curl one is pretty easy to read in your post above, but one of many possible queries generated by your JS code looks more like this:

{
  "size": 10,
  "from": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "*${input}*",
            "fields": ["ilad", "ilcead", "mahad"],
            "default_operator": "AND",
          },
        },
        { "terms": { "zoneid": [] } },
        {
          "range": {
            "kazatarihi": {
              "gte": "...",
              "lte": "..."
            }
          }
        }
      ],
      "filter": {}
    },
  },
  "sort": [{ "kazatarihi": { "order": "desc" } }]
}

(That doesn't include actual zoneid values or filter clauses.)

My first assumption is that one or more of these query rule differences is filtering out documents you're expecting to see. I would try simplifying your query to more closely match the curl one, and slowly building it back up one query rule at a time to debug which rule is removing the documents you expect, and then inspecting those documents to understand what data they may be missing that filters them out of the results.

1 Like
8

is it possible to make a remote connection to my desktop

9

If you need direct support, you'll need to contact our support team rather than going through the community forum. This will require you to have a Platinum or Enterprise license if you do not have one already.

1 Like
10

The problem is still ongoing. The reason for the data not coming after March 5th is that the Docker services on the application server (Linux) were restarted during that period, and I was unaware of this. Later, I was informed that the data was not coming from the application. When I checked the Elasticsearch service on the database server, I noticed it had stopped, so I restarted it. However, after that, I couldn't retrieve new data from the application. I made requests using curl, and the index is there, data is coming. Even after disabling the cache and making requests, data is still coming. There were no changes in the code, so I don't think the issue is there. It seems like the problem is more likely on the DevOps side.

11

Thank you for the details! There's no data persistence by default with Docker, and as mentioned in our Docker docs you need to bind data volumes to ensure the data is retained. Could that be the issue?

12

yes but wouldn't i lose all my data this way?

13

The purpose of binding data volumes on a Docker container is to ensure data created within the container is persisted to a place on disk on the system where the container is running. You can read more about how Docker volumes work, and how to use them properly, on the Docker docs site.