Found that my relatively light project would throw the following error on builds:
Error: A Serverless Function has exceeded the unzipped maximum size of 250 MB. : https://vercel.link/serverless-function-size
Adding the following vercel.json file helped:
{ "functions": { "api/**": { "excludeFiles": "{.next,*.cache,node_modules,public,app}/**" } } }
I believe the build cache is being included somehow in the build.