Self-invoke background function via POST requests

Support
1

Hello

I have an Express backend hosted in Netlify, coded in TS.

I have a background function that I need to self-invoke, not directly, but via a POST request. I’m doing that like this at the end of the function:

await axios.post(
    `${context.site.url}/.netlify/functions/my-function-background`
);

I’m using a background function because its 15-min limit is good for the operations I need to perform here, which are not normal req-res interactions from the frontend, and they might take a couple minutes every time.

When I deploy it, it seems to work ok, it completes the business logic, which involves a couple of database calls, and at the end it self-invokes.

After a few minutes of self invoking without showing errors, I start getting 508 errors like this:

{
  message: 'Request failed with status code 508',
  name: 'AxiosError',
  description: undefined,
  number: undefined,
  fileName: undefined,
  lineNumber: undefined,
  columnNumber: undefined,
  stack: 'AxiosError: Request failed with status code 508\n' +
    '    at settle (/var/task/node_modules/axios/dist/node/axios.cjs:2019:12)\n' +
    '    at IncomingMessage.handleStreamEnd (/var/task/node_modules/axios/dist/node/axios.cjs:3135:11)\n' +
    '    at IncomingMessage.emit (node:events:530:35)\n' +
    '    at endReadableNT (node:internal/streams/readable:1698:12)\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:90:21)\n' +
    '    at Axios.request (/var/task/node_modules/axios/dist/node/axios.cjs:4287:41)\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)',
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [ 'xhr', 'http', 'fetch' ],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function], Blob: [class Blob] },
    validateStatus: [Function: validateStatus],
    headers: Object [AxiosHeaders] {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json',
      Authorization: 'Bearer ...',
      'User-Agent': 'axios/1.7.7',
      'Content-Length': '2',
      'Accept-Encoding': 'gzip, compress, deflate, br'
    },
    method: 'post',
    url: '/.netlify/functions/my-function-background',
    data: '{}'
  },
  code: 'ERR_BAD_RESPONSE',
  status: 508
}

And, to make things more confusing, right after that error, and only some times, I see a log message saying:

c4022d8a INFO Unexpected Top Level Error: Error: Failed to get next invocation, error 403.

After getting those errors for a few more minutes of retrying (as in Netlify’s retry mechanism, not a retry in the code itself), it just stops retrying altogether and the function is never invoked again.

Is it not advisable to self-invoke background functions like this?
Are there technical limitations about them that I’m not aware of?
Should I await or not await them when self-invoking?
Is there any other recommended way of having recurrent processes in the background given Netlify’s lack of a server-full backend.

Thanks a lot

PS: I already went through their docs and I already tried the generative AI chatbot.

2

We prevent multiple executions after one-another as that’s a “loop”. So http 508 is a “loop detected” error code and we try to prevent it. I believe we enforce this after 9 or 10 self- invocations.

3

Is there any other recommended way of having recurrent processes in the background given Netlify’s lack of a server-full backend.

In server-full I could have timers running my logic after certain amount of time has gone by, but with serverless we can’t do this, hence the self-invocation approach.

I also tried using a scheduled function to invoke my background function on a cron basis, but this has two downsides:
1- it does not allow optimizing the requests to whenever I need them, instead they’re made periodically whether they’re needed or not, which unnecessarily increases my site’s request count (and potentially my billings)
2- Scheduled functions only run on published deploys. They don’t run on Deploy Previews or branch deploys., which is cumbersome at best, and near impossible to test them in branch deploys before pushing to prod at worst.

I understand this can’t be a rare use case.

Recommend an alternative within Netlify’s capabilities?

Thanks