Post

The fix (commit 9de3123, not pushed yet):

• Compile on for Windows: torch.compile now runs there too, in both start.js and start_small.js. Compile off made decode about 25% slower (24.3 instead of 18.3 ms per frame on Linux).
• Why it was off: I disabled it on Windows out of caution, because it can need Microsoft's C++ build tools. The Triton build we install on Windows comes with its own C compiler, and CTP's install already runs our Triton kernels fine, so that caution no longer applies.
• Fallback: if a layer still fails to compile on some machine, it now runs uncompiled at the old speed instead of failing the take (offload/serve.py:213).
• Testing: compiling on Windows is untested. On Linux the only change is the fallback setting; the code compiles, but I didn't start the Studio to run a take. The README is updated.

Why it will still feel slow for them: CTP's RTX A4500 has 20 GB of VRAM. That leaves room for only about 4 of the 40 expert layers, so every word generated pulls the other ~36 from system RAM over PCIe, on a card with about a third of the 5090's memory bandwidth. My estimate for that card is around 0.9x real time with compile and ~0.7x without. The README's own table already puts the RTX 4090 at ~0.8x, for the same reason. Windows isn't the cause; it was just also missing compile.

What to tell 20–24 GB users: use Small GGUF. It fits entirely in VRAM, so nothing goes over PCIe, and it should be much faster for them.