Rework forward pass to remove old gradients#46

Arkay92 wants to merge 13 commits intoopenai:mainopenai/point-e:mainfrom

Arkay92:fix/residual-cross-attention-reworkArkay92/point-e:fix/residual-cross-attention-reworkCopy head branch name to clipboard