I see this error fairly often, so I tried to narrow it down as far as I could with a few experiments:
I think the corrupt classifier checkpoint and the fallback retraining are probably the most useful place to start, rather than the CLIP inference itself.
Your log has this sequence:
classifier checkpoint cannot be loaded
→ corrupt file is moved aside
→ classifier is retrained on CPU for 60 epochs
→ Space finishes starting
→ first ZeroGPU request
→ ZeroGPU worker tries to initialize real CUDA
→ RuntimeError: No CUDA GPUs are available
The interesting part is that the CUDA error happens inside ZeroGPU’s worker_init, before your GPU function gets to the CLIP work.
I made a set of small ZeroGPU test Spaces and progressively removed pieces of the startup path. Eventually I could reproduce essentially the same worker-init failure with nothing more than a single CPU backward() in the long-lived Space process:
x = torch.tensor(2.0, requires_grad=True)
(x * x).backward()
Then, on the first later @spaces.GPU call, the worker failed at real CUDA initialization with:
spaces/zero/wrappers.py -> worker_init
torch.init(...)
torch.Tensor([0]).cuda()
torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available
The same backward() executed in a separate Python subprocess did not break the later ZeroGPU worker.
So my first two choices would be:
- Replace/regenerate the corrupt classifier checkpoint in the repository, so the Space does not retrain it during startup.
- If runtime retraining is intentional, run the training/backward pass in a separate Python process, let that process exit, then load the resulting checkpoint in the main app process.
For example, structurally:
if classifier_needs_training:
subprocess.run(
[sys.executable, "train_classifier.py"],
check=True,
)
load_classifier()
# Later:
# @spaces.GPU inference
I would try the valid-checkpoint route first because it is the smallest change.
Also, if the repaired checkpoint is only written to the Space’s runtime filesystem, remember that normal Space disk is ephemeral across restarts. If you want the fix to survive a rebuild/restart, replace the bad file in the repo or use persistent storage: Spaces storage documentation.
What I tested
I started with a known-good minimal ZeroGPU function and then added pieces back independently.
| Test |
Result |
What it ruled in/out |
Minimal @spaces.GPU CUDA tensor operation |
PASS |
Current ZeroGPU worker path itself works |
CUDA_VISIBLE_DEVICES="" before startup |
PASS |
That environment variable alone was not enough to reproduce it |
FAISS IndexFlatIP.add() with a TerraID-sized 490×512 index |
PASS |
FAISS add() alone was not enough |
| Transformers v5 import |
PASS |
Import alone was not enough |
| Explicit CUDA stream-state query in the parent |
FAIL |
Parent-side CUDA/runtime queries can produce the same later worker failure |
| TerraID-like complete startup |
FAIL |
The original startup pattern reproduced the failure |
| Classifier training only |
FAIL |
FAISS / Transformers / corrupt-file handling were not required |
CPU backward() only |
FAIL |
Optimizer, classifier architecture and 60 epochs were not required |
backward() with autograd multithreading disabled |
FAIL |
Disabling autograd worker threading did not avoid it |
Same backward() in a separate Python process |
PASS |
Process isolation prevented the later ZeroGPU failure |
The full startup reproduction was deliberately close to the sequence in your log:
- 490 examples
- 512-dimensional embeddings
- 8 classes
- FAISS
IndexFlatIP
- corrupt checkpoint load failure
- corrupt checkpoint moved aside
- classifier retraining
- SGD
- 60 epochs
- gradient clipping
All of that CPU-side work completed normally. The failure appeared only later, when the first ZeroGPU worker tried to initialize actual CUDA.
That distinction seems important: the training itself can look completely successful while leaving the long-lived parent process in a state that only becomes visible as a failure at the next GPU-worker startup.
Narrowing the training path
I then removed FAISS, Transformers and the checkpoint-loading path and kept only the CPU training loop.
It still failed.
Then I split the training loop further.
Model construction was fine. Importing torch._dynamo was fine. torch._dynamo.graph_break() was fine. Creating the SGD optimizer was fine.
The failure could be reduced to:
x = torch.tensor(2.0, requires_grad=True)
y = x * x
y.backward()
No CLIP, no FAISS, no optimizer, no classifier model.
That was enough to make the later ZeroGPU worker die during CUDA initialization.
A useful positive control
I also tested another parent-process operation that is already known to be troublesome in ZeroGPU-like conditions:
torch.cuda.is_current_stream_capturing()
Calling that while the ZeroGPU parent had no real GPU also led to the same later worker_init -> No CUDA GPUs are available failure.
There is a very similar real ZeroGPU case here:
Fix ZeroGPU CUDA initialization after CPU embedding
In that case, CPU-side Transformers inference indirectly called torch.cuda.is_current_stream_capturing(). The exception was caught, but the next ZeroGPU worker then failed with No CUDA GPUs are available.
I would not assume TerraID has exactly the same internal cause, but it demonstrates a useful ZeroGPU debugging rule:
The line where the GPU worker finally fails may be later than the operation that put the parent process into the problematic state.
Why CPU backward() and ZeroGPU can interact badly
ZeroGPU has an unusual process lifecycle compared with a normal dedicated-GPU Space.
The current ZeroGPU mechanism is described here:
How ZeroGPU works
In simplified form:
long-lived main web process
|
| startup / CPU work
|
+------ fork ------> short-lived GPU worker
|
+--> real CUDA initialization
+--> @spaces.GPU work
The main process does not own the real GPU. A cold GPU worker is forked from it when an @spaces.GPU request arrives, and real CUDA is initialized in the worker.
That makes the state of the parent process before the fork unusually important.
PyTorch itself has explicit machinery around this problem.
In the PyTorch 2.13 autograd engine source there is an in_bad_autograd_fork state specifically described as being true for children forked after the engine’s thread-pool initialization:
PyTorch 2.13 autograd engine
The engine also enumerates registered accelerator devices while initializing its device-side machinery.
On the CUDA side, PyTorch’s device-count implementation ultimately goes through CUDA device enumeration, and its device_count() result is initialized once and retained process-locally:
PyTorch 2.13 CUDAFunctions.cpp
So my current working model is roughly:
CPU backward() in the long-lived ZeroGPU parent
|
+--> autograd initializes accelerator/device-related machinery
|
+--> parent crosses a fork-sensitive runtime boundary
|
+--> ZeroGPU later forks a GPU worker
|
+--> real CUDA initialization
|
+--> "No CUDA GPUs are available"
I would treat that last internal mechanism as a working hypothesis, not a proven PyTorch root cause. I reproduced the boundary experimentally, but I did not instrument the exact C++/CUDA state inherited by the failed child.
The externally useful part is stronger:
- CPU
backward() in the ZeroGPU parent reproduced the failure.
- Turning off autograd multithreading did not fix it.
- Moving exactly the same
backward() into another Python process did fix it.
Why disabling autograd multithreading did not help
I also tried:
with torch.autograd.set_multithreading_enabled(False):
y.backward()
The ZeroGPU worker still failed afterward.
That is consistent with the autograd implementation: disabling execution multithreading does not necessarily prevent the engine from going through its device/thread-pool initialization path before graph execution.
So I would not rely on set_multithreading_enabled(False) as the workaround here.
Why the subprocess result is useful
The subprocess test looked conceptually like this:
ZeroGPU main process
|
+--> new Python child
| |
| +--> CPU backward()
| +--> save result/checkpoint
| +--> exit
|
+--> later ZeroGPU GPU worker
|
+--> CUDA initialization succeeds
That isolates whatever autograd/runtime state was created by training inside a process that is already dead by the time ZeroGPU forks its worker.
There is a related PyTorch precedent in:
PyTorch #83973 — Implement torch.cuda.device_count without poison
That issue discusses CUDA queries making a parent process unsafe for a later fork. One workaround used by Lightning was to run the CUDA query in a separate subprocess so the main process remained unpoisoned.
It is not the same exact call path as this backward() reproduction, but the process-isolation strategy is very similar.
What I would change in TerraID
1. Fix the classifier checkpoint first
Your current log already tells you that museum_classifier_v5.pt cannot be read:
PytorchStreamReader failed reading zip archive:
unsupported multidisk archive
The fallback training succeeds, but that fallback appears to be the thing that crosses the problematic backward() boundary.
So I would regenerate the classifier once, verify that it loads successfully, and put the valid checkpoint in the repository.
Then restart the Space and check that startup says something equivalent to:
classifier loaded successfully
rather than:
training classifier from existing memory...
If Analyze works after that, that is a very strong confirmation without requiring any architectural change.
2. If training at runtime is required, separate it
For example:
app.py
|
+--> detect missing/bad classifier
|
+--> run train_classifier.py as child process
| |
| +--> load embeddings
| +--> forward/backward/SGD
| +--> save checkpoint
| +--> exit
|
+--> load saved checkpoint
|
+--> serve Gradio / ZeroGPU inference
I would keep that child genuinely CPU-only and avoid importing the ZeroGPU application itself into it.
This is slightly more process plumbing, but it gives a clean boundary between “training code that uses autograd” and “long-lived process that ZeroGPU will later fork”.
3. I would not chase FAISS first
I initially considered FAISS because it is another native/threaded component running before the ZeroGPU fork.
However, a small IndexFlatIP reconstruction matching the approximate size in your log completed before the GPU request and the worker still initialized correctly.
So FAISS may have its own multiprocessing/OpenMP caveats in general, but it did not reproduce this particular failure by itself in my test.
4. CUDA_VISIBLE_DEVICES="" was not enough by itself either
I tested the empty visibility setting independently and the later ZeroGPU CUDA worker still worked.
So I would not call it the root cause of this failure.
I would still avoid manually controlling GPU visibility unless you specifically need it, since ZeroGPU already manages GPU visibility at the worker boundary and fewer moving parts make this easier to debug.
A separate ZeroGPU/CLIP point for after the worker issue is fixed
This does not explain the worker_init failure above, because that failure happens before your CLIP function actually starts.
But there is one cleanup I would make afterward.
Current ZeroGPU documentation recommends loading/placing the GPU model at module scope and leaving the actual computation inside @spaces.GPU. Lazy-loading the model or doing the initial .to("cuda") inside the decorated function is discouraged:
ZeroGPU documentation — model loading
The intended pattern is approximately:
import spaces
import torch
model = ...
model.to("cuda") # ZeroGPU emulation handles this at startup
@spaces.GPU
def analyze(...):
# actual model computation
...
ZeroGPU’s startup emulation captures that placement without requiring a real GPU in the main process, while the worker later receives the real CUDA-resident weights.
I would treat this as a second cleanup, not as the explanation for the traceback you posted.
Also, since your dependencies are unpinned, I would consider pinning the versions once the Space is working. My reproduction environment currently reported:
Python 3.12.12
torch 2.13.0+cu130
The original log does not print its resolved PyTorch version, so I would not assume it was exactly the same. The currently supported PyTorch versions are listed in the ZeroGPU docs.
About the `ValueError: Invalid file descriptor: -1`
I would treat this separately from the CUDA failure for now.
There is a Gradio-on-HF-Spaces issue reporting the same asyncio cleanup traceback:
Gradio #12699 — ValueError: Invalid file descriptor: -1 with Gradio 6 on HF Spaces
In that report the application otherwise continued to work.
In my failing ZeroGPU probes I also saw this message near the worker failure, but I do not have evidence that it causes the CUDA problem.
So I would first fix the backward() / worker-init path. If the file-descriptor traceback remains afterward, then it is worth treating it as its own Gradio/asyncio cleanup issue.
One final caution: the worker_init -> No CUDA GPUs are available stack is not unique to one root cause. Other ZeroGPU Spaces have produced a very similar stack from other mistakes, for example a GPU path that was not correctly decorated: MMAudio ZeroGPU discussion.
So I would not diagnose this from the stack trace alone.
What makes the startup-training explanation much more interesting here is that the posted log already shows an unexpected retraining pass immediately before the first GPU request, and a minimal CPU backward() was enough to reproduce the same later ZeroGPU worker failure in isolation.
For this particular Space, a valid classifier checkpoint that eliminates startup retraining is the lowest-cost test I would try first.
MUSEUM AI v5.2
Starting ZeroGPU-compatible Space
Imported 490 memory examples from museum_memory.json
FAISS memory rebuilt: 490 vectors
Could not load classifier: PytorchStreamReader failed reading zip archive: unsupported multidisk archive. This is an internal miniz error. If you are seeing this error, there is a high likelihood that your checkpoint file is corrupted. This can happen if the checkpoint was not saved properly, was transferred incorrectly, or the file was modified after saving.
Training classifier from existing memory…
(Node proxy → Python :7861)