AIDL — Fingerspelling

Fingerspelling is how ASL signers spell out names, technical terms and words that have no dedicated sign. It accounts for roughly 12–35% of all ASL signing, yet automatic recognition is still an open problem.

Classifying a static hand gesture maps one image to one letter. Fingerspelling is different: the model has to read a sequence of hand poses, including the transitions between letters. Those transitions vary with each signer and with signing speed, which averages around 65 words per minute.

Reliable recognition could make real-time communication possible where no interpreter is available: medical appointments, customer service, emergencies and everyday conversations.

After surveying the public ASL datasets, we chose the Google ASL Fingerspelling dataset from Kaggle (190 GB). It contains sequences of MediaPipe landmarks paired with text phrases. FSBoard is richer, but at 1.45 TB it didn’t fit our compute budget.

We kept only the 21 right-hand landmarks: x, y and z for each gives 63 position features. Adding frame-to-frame velocity doubles that to 126 features per frame. We split train and validation by participant, so no signer appears in both and the model is always validated on people it hasn’t seen.

To keep the problem tractable, we limited the vocabulary to the letters a–z and the space: 28 output classes including the CTC blank. Phrases with digits, punctuation and URLs were filtered out.

The dataset has no frame-level annotations: each sequence is labeled only with its full phrase. The model has to align a variable-length run of frames with a much shorter string of characters, without being told which frame belongs to which letter.

That made Connectionist Temporal Classification (CTC) the natural choice. It’s the same mechanism used in speech recognition and lip reading, and it learns the alignment on its own. Any approach that needed explicit frame-to-letter labels would have required manual annotation that doesn’t exist.

Preprocessing

1. Drop invalid frames where MediaPipe found no hand, and zero-fill partial detections.
2. Normalize each frame by centering on the wrist and scaling by the hand’s extent, so the model doesn’t depend on where the hand is or how close it is to the camera.
3. Add velocity features (frame-to-frame deltas) to capture motion.
4. Augment with temporal resampling at 0.8×–1.2× speed to simulate faster and slower signers.
5. Pad or truncate every sequence to 160 frames for batched training.

We started with a simple RNN baseline to validate the pipeline, then trained a BiLSTM, a Transformer and a Conformer under identical conditions: a 3k-sequence subset, 20 epochs, batch size 16 and learning rate 5e-4. All used CTC loss, the Adam optimizer and greedy decoding.

Scaling the BiLSTM

The biggest single gain came from scaling the training data from 3k to 50k sequences. Doubling the hidden size, a larger batch and a higher learning rate brought validation CER down to 0.38, and the average edit distance dropped from about 10 characters to about 5.

A further hyperparameter search (dropout 0.3, batch 32, hidden 256) reached a best validation CER of 0.372. It also showed that dropout is essential at full data scale: without it, CTC training collapsed.

On the held-out test set, the best checkpoint scored CER 0.42 against 0.38 on validation. On the supplemental set of natural sentences, which differs from the training data, CER rose to 0.52. The predictions show a clear pattern: the model reads the letters well but systematically drops spaces between words.

This matches what the research literature describes: word-boundary detection is an open problem in fingerspelling. It’s also an actionable finding. Beam-search decoding with a character-level language model is the most promising single fix.

Training moved through several environments as the experiments grew: Kaggle notebooks for prototyping, then Lightning AI, then dedicated Google Cloud GPUs once the runs needed more throughput. Weights & Biases tracked CER, loss and edit distance across every run.

On Google Cloud we set up a lightweight MLOps stack so the whole team could queue experiments on a single GPU machine:

• GPU VM: a g2-standard-12 with one NVIDIA L4, provisioned as Standard rather than Spot so long runs wouldn’t be preempted.
• ClearML queue: an agent running as a systemd service pulls jobs from a hosted ClearML server. The SDK captures parameters, metrics, logs and GPU stats for every run.
• Cost watchdog: a systemd timer checks GPU activity and the ClearML queue every 5 minutes, and shuts the VM down after 30 idle minutes.
• Checkpoint sync: a cron job syncs best and latest checkpoints to Google Cloud Storage every 30 minutes. Files are named by run and task ID, so they never overwrite each other.

• Beam search + language model to fix word boundaries, the dominant error.
• Richer features such as pairwise fingertip distances, to separate letters like “a” and “e”.
• Graph neural networks that model the hand skeleton directly.
• More data or pretraining to give Transformer-based models a fair chance.
• A larger vocabulary covering digits and punctuation.

The gap to state-of-the-art systems comes down mostly to data and compute rather than design. The foundation is solid and the failure modes are well understood.