EyeTAG: Eye Trajectory-Aware Gaze Estimation

This repository is the official PyTorch implementation of the BMVC 2026 paper "EyeTAG: Eye Trajectory-Aware Gaze Estimation".

Jungmin Lee, Niamat Ullah^†, Yoseob Han^†
Soongsil University, Seoul, Republic of Korea
_{^† Co-corresponding authors}

EyeTAG is a causal multi-frame gaze estimator built around an explicit first-order gaze prior: at every step the model differentiates its own recent predictions and feeds that differential trajectory back in as a compact kinematic token.

Most video gaze estimators do model time, but only implicitly — motion has to be recovered from high-dimensional appearance features that must simultaneously encode identity, illumination and head pose. EyeTAG instead carries the gaze trajectory as an explicit variable.

J fixation jitter (lower better), B signed saccade bias (closer to 0 better), M = (J + |B|) / 2.

Removing only the kinematic prior (same encoder, same everything else) restores a systematic saccade under-shoot: visual-only B = −0.46, normal prior B = −0.53. Reproduce with scripts/temporal_stability.sh.

Complexity of the released model: 37.91 M parameters, 121.57 GFLOPs and 19.97 ms per 48-frame window (RTX A6000, FP32, batch size 1). The GFLOPs figure is a no-caching upper bound — the visual encoders are per-frame, so a streaming deployment can cache them across the sliding window.

git clone https://github.com/peter8366/EyeTAG.git
cd EyeTAG
conda create -n eyetag python=3.11 -y && conda activate eyetag
pip install -r requirements.txt

decord and h5py are only needed for EVE. Gaze360 runs without them.

The face encoder is a ResNet-50 pretrained on VGGFace2. Download resnet50_ft_weight.pkl from the VGGFace2 pytorch release and place it at:

checkpoints/resnet50_ft_weight.pkl

Use the standard preprocessed Gaze360 layout (per-frame JPEG crops plus .label index files), as produced by GazeHub:

<GAZE360_ROOT>/
├── train.label  val.label  test.label
├── train/Face/<sid>/<sid>_<idx>.jpg
├── train/Left/...   train/Right/...
└── test/...

Each label line is Face Left Right Origin 3DGaze 2DGaze, where 3DGaze is gx,gy,gz (unit vector) and 2DGaze is gyaw,gpitch in radians.

The three evaluation subsets follow the original Gaze360 convention and are selected with --gaze-subset: full (|yaw| ≤ 180°), semi-front (≤ 90°), front (≤ 20°).

Use the official EVE release as-is (MP4 + HDF5):

<EVE_ROOT>/
├── train01/ ... val01/ ... test01/ ...
│   └── step008_image_MIT-i2263005350/
│       ├── webcam_c.mp4  webcam_c_eyes.mp4  webcam_c.h5

The paper uses the frontal webcam_c stream only, subsampled to 30 Hz.

--split test in eve/test.py selects participants val01..val05, i.e. the official EVE validation split. This is what the paper reports — the official test annotations are not publicly available.

# Gaze360 (main result)
bash scripts/train_gaze360.sh /path/to/gaze360 0

# EVE
bash scripts/train_eve.sh /path/to/eve_dataset 0

Both scripts spell out every hyper-parameter used for the paper numbers; see the docstring at the top of gaze360/train.py for the equivalent raw command.

Scheduled sampling. The gaze history fed to TAKE is sampled from the ground truth with probability α, otherwise from a cache of the model's own predictions that is rebuilt once per epoch (--cache-mode ar). α is annealed 1.0 → 0.0 over epochs 1–5, after which the history is fully autoregressive. Training remains fully parallel — no sequential rollout is needed, because the cache is built once per epoch rather than on the fly.

Key flags:

# Gaze360: autoregressive inference on all three yaw subsets
bash scripts/eval_gaze360.sh /path/to/gaze360 work_dir/eyetag_t48_delta/best.pth 0

# EVE
bash scripts/eval_eve.sh /path/to/eve_dataset work_dir/eyetag_eve_t48_delta/latest.pth 0

--autoregressive is the deployment setting reported in the paper: the gaze history is built from the model's own predictions, rolled out per subject track. Without it, the model is evaluated under teacher forcing.

bash scripts/temporal_stability.sh /path/to/gaze360 work_dir/eyetag_t48_delta/best.pth 0

This dumps frame-ordered predictions to analysis/data/*.npz and then computes J / B / M plus the saccade-level diagnostics (directional error, near-static prediction rate). Drop additional .npz dumps into the same directory to compare several models in one table.

eyetag/                  shared model package
├── geometry.py          pitchyaw <-> vector, angular error, differential gaze
└── models/
    ├── model.py         GazeEstimator: DSVE + TAKE + CCAF + GVP
    ├── face_vggface2.py face encoder (ResNet-50, VGGFace2)     [ours]
    ├── face_resnet.py   face encoder (ResNet-18, ImageNet)     [ablation]
    ├── eye_net.py       eye encoders: resnet18 [ours], efficientnet_b0, tsm, 3dcnn
    ├── prev_encoder.py  TAKE — differential gaze trajectory -> kinematic token
    ├── temporal_fusion.py  CCAF — cross-attention + causal Transformer decoder
    └── head.py          GVP — gaze regression head

gaze360/                 Gaze360 pipeline (train / test / dataset / utils)
eve/                     EVE pipeline (train / test / dataset / utils)
analysis/                temporal-stability analysis (paper Table 3)
scripts/                 one-command reproduction of every paper number
baselines/               patches + scripts used to re-train the baselines
docs/                    reproduction notes and baseline provenance

The two dataset pipelines are kept separate on purpose: they differ in media (JPEG frames vs MP4 + HDF5), label convention and sampling, and merging them would obscure the exact configuration behind each reported number. Everything they share — the model and the gaze-space math — lives in eyetag/.

All baselines in the comparison tables were re-trained by us under an identical protocol rather than quoted. The upstream repositories are not vendored here; docs/BASELINES.md lists each source commit together with the patches in baselines/patches/ and the driver scripts in baselines/scripts/ needed to reproduce our runs, including the backbone-controlled variants.

@inproceedings{eyetag2026,
  title     = {EyeTAG: Eye Trajectory-Aware Gaze Estimation},
  author    = {Lee, Jungmin and Ullah, Niamat and Han, Yoseob},
  booktitle = {British Machine Vision Conference (BMVC)},
  year      = {2026}
}

MIT — see LICENSE. Note that the Gaze360 and EVE datasets, and the VGGFace2 pretrained weights, carry their own licenses and must be obtained from their original providers.

The face encoder uses VGGFace2 pretrained weights; the eye-encoder ablations use TSM, C3D and EfficientNet-B0. We thank the authors of Gaze360, EVE, L2CS-Net, GazeTR, CrossGaze, STAGE and MCGaze for releasing their code.