Swiftype Document Expiration API does not seem to work

Elastic Search
1

I believe I may have found an issue with Swiftype's Document Expiration API: Document Indexing | Swiftype Documentation. I followed what the API docs said to do in regards to expires_after and precise_expiration,
but no matter what I try I always seem to get back a record that should have already expired.

See below for a minimal bash script that reproduces the issue (API KEY omitted intentionally). It creates a dummy document MY_DOC for MY_ENGINE_KEY with a randomized string and expires_after 60 seconds in the future, waits for 70 seconds, then runs a search query, with precise_expiration set, against the same randomized string. The expectation is that the record should have expired already, therefore no record should get return, however, I always seem to get 1 back.

The following test script reproduces the bug:

#! /bin/bash
export LC_CTYPE=C
export random_search_string=$(cat /dev/random | tr -dc "[:alpha:]" | head -c 32)
export ONE_MINUTE_LATER=$(date -v+1M +%s)
export MY_DOC=""
export MY_ENGINE_KEY=""
export MY_API_KEY=""

printf "Search string: $random_search_string\n"
printf "Creating '"$MY_DOC"' record with expire_after 60 seconds in the future\n"

curl -X POST 'https://api.swiftype.com/api/v1/engines/'"$MY_ENGINE_KEY"'/document_types/'"$MY_DOC"'/documents/async_bulk_create_or_update' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "'"$MY_API_KEY"'",
        "documents":
[
  {
    "expires_after": '"$ONE_MINUTE_LATER"',
    "external_id": "33333",
    "fields": [
      {
        "name": "external_id",
        "value": 33333,
        "type": "string"
      },
      {
        "name": "text",
        "value": "Dummy_title",
        "type": "string"
      },
      {
        "name": "slug",
        "value": "dummy-slug",
        "type": "string"
      },
      {
        "name": "lever_id",
        "value": "dummy_lever_id",
        "type": "string"
      },
      {
        "name": "location",
        "value": "Los Gatos, California",
        "type": "string"
      },
      {
        "name": "state",
        "value": "published",
        "type": "string"
      },
      {
        "name": "url",
        "value": "https://jobs.netflix.com/jobs/33333",
        "type": "enum"
      },
      {
        "name": "created_at",
        "value": "Wed Sep 05 2018 23:43:56 GMT+0000 (Coordinated Universal Time)",
        "type": "date"
      },
      {
        "name": "description",
        "value": "dummy description",
        "type": "text"
      },
      {
        "name": "lever_team",
        "value": "dummy lever team",
        "type": "string"
      },
      {
        "name": "search_text",
        "value": "'"$random_search_string"'",
        "type": "text"
      },
      {
        "name": "subteam",
        "value": "dummy subteam",
        "type": "string"
      },
      {
        "name": "organization",
        "value": "dummy org",
        "type": "string"
      },
      {
        "name": "team",
        "value": "dummy team",
        "type": "string"
      }
    ]
  }
]

      }'


printf "\nWait 70 seconds..."
sleep 70s


printf "\nCalling search with precise_expiration flag"


# Actual record_count:1, Expected: record_count:0
curl -XGET 'https://search-api.swiftype.com/api/v1/engines/'"$MY_ENGINE_KEY"'/document_types/'"$MY_DOC"'/search.json' \
  -H 'Content-Type: application/json' \
  -d '{
        "auth_token": "'"$MY_API_KEY"'",
        "q": "'"$random_search_string"'",
        "precise_expiration": {
          "'$MY_DOC'": true
        }
      }'

Here's a sample output I get when I run the script on my macbook:

Search string: GoeUwAJkYlKATKPrrPfJntygnyKCUkoy
Creating 'postings' record with expire_after 60 seconds in the future
{"batch_link":"https://api.swiftype.com/api/v1/document_receipts.json?ids=5e753ac882ecd375041dc72f","document_receipts":[{"id":"5e753ac882ecd375041dc72f","external_id":"33333","status":"pending","errors":[],"links":{"document_receipt":"https://api.swiftype.com/api/v1/document_receipts/5e753ac882ecd375041dc72f.json","document":null}}]}
Wait 70 seconds...
Calling search with precise_expiration flag{"record_count":1,"records":{"postings":[{"text":"Dummy_title","lever_id":"dummy_lever_id","team":"dummy team","slug":"dummy-slug","external_id":"33333","description":"dummy description","url":"https://jobs.netflix.com/jobs/33333","search_text":"GoeUwAJkYlKATKPrrPfJntygnyKCUkoy","state":"published","updated_at":"2020-03-20T21:51:04+00:00","created_at":"2018-09-05T23:43:56Z","location":"Los Gatos, California","organization":"dummy org","subteam":"dummy subteam","lever_team":"dummy lever team","_index":"5e1367e6c35bc9473d5fe272","_type":"5e74278682ecd314521dab28","_score":11.990819,"_version":null,"_explanation":null,"sort":null,"id":"5e74278d28ccbc113e741f21","highlight":{}}]},"info":{"postings":{"query":"GoeUwAJkYlKATKPrrPfJntygnyKCUkoy","current_page":1,"num_pages":1,"per_page":20,"total_result_count":1,"facets":{}}},"errors":{}}

The important part of this output is that record_count is 1 and not 0 .

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.