Book the Partial Before You Retry
(dev.to)
A free model call that you abandon is not free. You already spent the queue, and you already spent the tokens the server emitted before you cut the socket. Write that partial off before you retry. If you do not, you are running a loop that looks thrifty and bills you in minutes. You hit this when a shared free server is slow and your client is optimistic. The first attempt streams a couple hundred tokens, then the deadline you promised a person slips, so you cancel and send the same prompt agai
댓글
아직 댓글이 없습니다. 첫 댓글을 남겨보세요.