MongoServerError Path 'timestamp' needs to be indexed as token

[BlindChicken (ns)]

When running a $search query I’m getting the error Path 'timestamp' needs to be indexed as token. Where ‘timestamp’ can be any field, but the error seems to only occur when handling dates. The collection has dynamic index. The error occurs on some queries, but not on all queries, so it is really difficult to troubleshoot the error. (I’m generating the queries using an LLM). If I take the query and manually run in mongosh the query runs without any problems and returns the expected documents. But when it run in the application (nodeJS driver) it throws the error.

Here is an example of a query that throws the error:

const query = [
  {
    "$search":{
      "compound":{
        "must":[
          {"range":{
            "path":"timestamp",
            "gte": new Date("2024-01-01T00:00:00Z"),
            "lte": new Date("2024-12-31T23:59:59Z")
          }}
        ],
        "should":[
          {"text":{"query":"John Smith","path":"contact_name","fuzzy":{"maxEdits":1}}},
          {"text":{"query":"John Smith","path":"client_name","fuzzy":{"maxEdits":1}}}
        ],
        "minimumShouldMatch":1,
        "filter":[
          {
            "equals":{
              "value": new ObjectId("6763472e6e639d41342202d8"),
              "path":"firm_id"
            }
          }
        ]
      }
    }
  },{
    "$sort":{"score":-1,"_id":-1}
  },
  {"$limit":20},
  {
    "$project":{
      "score":{"$meta":"searchScore"},
      "_id":1,
      "business_number":1,
      "client_id":1,
      "client_name":1,
      "confirmed":1,
      "contact_name":1,
      "content_type":1,
      "doc_frequency":1,
      "doc_type":1,
      "email_address":1,
      "firm_id":1,
      "next_issue_id":1,
      "phone_number":1,
      "sha256":1,
      "timestamp":1,
      "next_date":1
    }
  }
]

Does anoyone have an idea of what could be causing the error, or what does this error message mean?

[BlindChicken (ns)]

TLDR: The dates were included in the query as strings and not as Date objects.

I found the bug. I have a script to parse the JSON response from the LLM, which converts date strings back to Date objects. The regex pattern I used required the milliseconds to be included in the date string to be detected as a date. So whenever the LLM omitted the milliseconds the date would be included in the query as a string, thus causing the error.

The bug was difficult to diagnose. To log the full query you needed to stringify the object so the dates are displayed as strings. Also, when I ran the query in the shell and I did the date conversion manually and so they were converted without any errors.