Methods

How VitalsGraph measures — and what it cannot tell you

VitalsGraph is an autonomic-arousal and cognitive-load oscilloscope. It turns a webcam and microphone into six synchronized physiological waveforms, and reports how far each departs from the subject’s own calibrated baseline. It never classifies a statement as true or false.

1 · Architecture

camera ──rVFC──▶ VideoFrame ─┐                  ┌─ MediaPipe FaceLandmarker (GPU, OffscreenCanvas)
                              ├─▶ engine worker ─┤
mic ──AudioWorklet──▶ PCM ────┘   (no main-     └─ C++23 → WebAssembly DSP core (Clang/LLVM, wasm32-wasi)
       (MessagePort, zero-copy)    thread work)        frame clock · ROIs · LK flow · POS · YIN · fusion
                                                              │ 30 Hz rows (53 channels) + HUD block
main thread: React UI · WebGL 1 polygram · IndexedDB (idb) · MediaRecorder · Supabase sync (opt-in)

2 · Channels

CH1

Cardiovascular

RGB forehead & cheek ROIs

Expected signature: Pulse-amplitude constriction, shortened inter-beat intervals, RMSSD suppression.

  • Skin ROIs: an oriented forehead band and two cheek parallelograms derived from the 478-point mesh; pixels that are saturated or under-exposed are excluded.
  • Illumination rectification: per-channel NLMS (4 taps, μ = 0.02) against the two flattest background patches; textured patches (text, clocks, screens) are rejected and the canceller is bypassed whenever it does not reduce variance.
  • POS projection over 1.6 s windows with overlap-add; zero-phase 3rd-order Butterworth band-pass 0.75–3.0 Hz.
  • Heart rate: Welch PSD (8.5 s Hann segments, 12 s history) with parabolic peak interpolation; SNR = power within ±0.1 Hz of the fundamental and ±0.2 Hz of the harmonic vs. the rest of 0.6–4 Hz.
  • Beats: detected on an adaptive band (HR ± 0.6 Hz) with refractory logic; IBIs outside 0.33–1.5 s or > 30 % from the running median are rejected; RMSSD over 30 s; pulse amplitude = median of the last 5 peak-to-trough excursions.
  • Artefacts: > 4 % skin-luminance steps between consecutive 0.5 s means, or head motion > 25 % face-height/s, flag −1 s…+5 s as unscorable (ppg_art).
CH2

Thoracic respiration

Shoulder / clavicle optical flowVAD-muted while speaking

Expected signature: Flat-lining before speech (breath-hold) then shallow, fast recovery breaths.

  • Pyramidal Lucas–Kanade (3 levels, 15×15 window) on a 6×4 grid in each shoulder box anchored below the chin.
  • Displacement is measured against a keyframe refreshed every 6 s rather than integrated frame-to-frame, which would turn tracking noise into a 1/f² random walk inside the respiratory band.
  • Zero-phase 2nd-order Butterworth band-pass 0.15–0.5 Hz; rate from Welch PSD (20–32 s); amplitude = √2·RMS over 8 s, in ‰ of face height (scale-invariant). Falls back to head motion when the shoulders are not trackable (with hysteresis).
CH3

Ocular & cognitive load

Iris & eyelid landmarks

Expected signature: Blink suppression plateau followed by a dense flurry of blinks.

  • Eye aspect ratio (Soukupová & Čech) from six contour points per eye, relative to an adaptive open-eye level; blinks require EAR < 62 % of the open level or the eyeBlink blendshape > 0.55, lasting 40–600 ms.
  • Blink rate over 15 s; gaze angle from the iris centre's offset along the eye axis (eye width / eyeball radius ≈ 2.5); angular velocity with a 3-tap median.
CH4

Upper-face affect leakage

FACS AU1 · AU2 · AU4 · AU6

Expected signature: Transient brow raise / furrow micro-spikes right after a keyword.

  • AU1 = browInnerUp, AU2 = browOuterUp (L+R), AU4 = browDown (L+R), AU6 = cheekSquint (L+R).
  • Composite brow signal → derivative → 2–10 Hz band-pass → rectified attack/release envelope (20 ms / 250 ms).
  • Residual optical strain: a 12×12 LK grid over the face, rigid/affine motion removed by least squares, strain magnitude from finite differences — rendered as the replay heatmap.
CH5

Lower-face squelch & asymmetry

FACS AU12 · AU14 · AU17 · AU23/24VAD-muted while speaking

Expected signature: Post-answer lip clamping or unilateral mouth-corner tightening.

  • AU12 = mouthSmile, AU14 = mouthDimple, AU17 = mouthShrugLower, AU23/24 = max(mouthPress, mouthRoll).
  • Asymmetry = |L − R| differences; squelch = lip-press tension above a 20 s running baseline that adapts only during silence. The channel is muted (VAD gate) while the subject speaks.
CH6

Vocal micro-tremor & pitch

Microphone PCMscored only while speaking

Expected signature: Upward F0 drift and loss of the 8–12 Hz physiological micro-tremor.

  • Anti-aliased resampling to 16 kHz; 40 ms frames, 10 ms hop; noise floor by minimum tracking; VAD = loud + voiced onset (2 of 5 frames) with 300 ms hangover.
  • YIN F0 (60–400 Hz, threshold 0.12, parabolic refinement). Pitch-synchronous cycle tracker → local jitter and shimmer (%) within the current voiced run.
  • Tremor index = 8–12 Hz share of the 3–20 Hz modulation power of the F0 contour (1 s, detrended, in cents). Scored only while the subject speaks.

3 · Baseline & fusion statistics

4 · Validation

The DSP core ships with native unit tests (built with AddressSanitizer/UBSan) against synthetic ground truth: filter responses, zero-phase behaviour, Welch peak and SNR, POS heart rate and beats, respiratory rate, YIN F0, jitter, tremor, χ² and normal quantiles, Cholesky Mahalanobis, sub-pixel Lucas–Kanade and the frame clock. End to end, the full browser pipeline was run in Chrome on a frontal-webcam recording from the MCD-rPPG dataset with a synchronized reference PPG: median heart rate 74.3 bpm versus 75.1 bpm from the reference (10 s windows within ±3 bpm). That is one subject under laboratory lighting — a functional check, not a clinical validation.

5 · Limitations & responsible use

6 · References

Remote photoplethysmography

  1. Wang, W.; den Brinker, A. C.; Stuijk, S.; de Haan, G. (2017). Algorithmic Principles of Remote PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479–1491. doi:10.1109/TBME.2016.2609282
    Defines the POS (plane-orthogonal-to-skin) projection the app uses as its primary pulse-extraction algorithm from facial RGB traces.
  2. de Haan, G.; Jeanne, V. (2013). Robust Pulse Rate From Chrominance-Based rPPG. IEEE Transactions on Biomedical Engineering, 60(10), 2878–2886. doi:10.1109/TBME.2013.2266196
    Source of the CHROM chrominance method used as a fallback/cross-check and of the spectral SNR metric the app uses to score pulse-signal quality.
  3. Li, X.; Chen, J.; Zhao, G.; Pietikäinen, M. (2014). Remote Heart Rate Measurement From Face Videos Under Realistic Situations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014), 4264–4271. doi:10.1109/CVPR.2014.543
    Basis for the illumination-rectification step that subtracts a background-region signal from the face signal with an NLMS adaptive filter.
  4. Verkruysse, W.; Svaasand, L. O.; Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434–21445. doi:10.1364/OE.16.021434
    First demonstration that a consumer camera under ambient light captures a plethysmographic signal (strongest in the green channel), the premise of webcam rPPG.
  5. Poh, M.-Z.; McDuff, D. J.; Picard, R. W. (2010). Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762–10774. doi:10.1364/OE.18.010762
    Early automated webcam pulse-rate study with face tracking, cited as precedent for measuring heart rate from a basic webcam.
  6. Egorov, K.; Botman, S.; Blinov, P.; Zubkova, G.; et al. (2025). Gaze into the Heart: A Multi-View Video Dataset for rPPG and Health Biomarkers Estimation. Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), pp. 13053–13059. doi:10.1145/3746027.3758255
    MCD-rPPG (CC-BY-4.0) frontal-webcam recordings with synchronized PPG were used for end-to-end validation of the heart-rate channel.

Heart-rate variability

  1. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology (1996). Heart rate variability: standards of measurement, physiological interpretation and clinical use. Circulation, 93(5), 1043–1065. doi:10.1161/01.CIR.93.5.1043
    Standard definitions of time-domain HRV indices, including RMSSD, as computed from the rPPG inter-beat intervals.
  2. Shaffer, F.; Ginsberg, J. P. (2017). An Overview of Heart Rate Variability Metrics and Norms. Frontiers in Public Health, 5, 258. doi:10.3389/fpubh.2017.00258
    Justifies the app's use of short and ultra-short-term RMSSD windows and warns that such values are not interchangeable with 24 h norms.

Optical flow

  1. Lucas, B. D.; Kanade, T. (1981). An Iterative Image Registration Technique with an Application to Stereo Vision. Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI '81), Vol. 2, 674–679. www.ijcai.org/Proceedings/81-2/Papers/017.pdf
    The gradient-based Lucas–Kanade flow estimator the app uses to track chest/shoulder feature points for respiration.
  2. Bouguet, J.-Y. (2001). Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm. Technical report, Intel Corporation, Microprocessor Research Labs (OpenCV documentation). robots.stanford.edu/cs223b04/algo_tracking.pdf
    Describes the coarse-to-fine pyramidal Lucas–Kanade tracker the app implements so it can follow larger torso motion without losing sub-pixel accuracy.

Respiration

  1. Romano, C.; Schena, E.; Silvestri, S.; Massaroni, C. (2021). Non-Contact Respiratory Monitoring Using an RGB Camera for Real-World Applications. Sensors, 21(15), 5126. doi:10.3390/s21155126
    Validates optical-flow tracking of chest motion from a consumer RGB camera for breathing rate, the approach behind the app's respiration channel.

Ocular dynamics & face mesh

  1. Soukupová, T.; Čech, J. (2016). Real-Time Eye Blink Detection using Facial Landmarks. 21st Computer Vision Winter Workshop (CVWW 2016), Rimske Toplice, Slovenia. vision.fe.uni-lj.si/cvww2016/proceedings/papers/05.pdf
    Defines the Eye Aspect Ratio (EAR) computed from eye landmarks, which the app thresholds to detect blinks and compute blink rate.
  2. Leal, S.; Vrij, A. (2008). Blinking During and After Lying. Journal of Nonverbal Behavior, 32(4), 187–194. doi:10.1007/s10919-008-0051-0
    Supports interpreting blink suppression during an answer, followed by a compensatory rise afterwards, as a cognitive-load cue rather than a direct lie signal.
  3. Grishchenko, I.; Ablavatski, A.; Kartynnik, Y.; Raveendran, K.; et al. (2020). Attention Mesh: High-fidelity Face Mesh Prediction in Real-time. arXiv preprint arXiv:2006.10962. arxiv.org/abs/2006.10962
    Describes the refined eye, iris and lip landmarks (478-point mesh) the app uses for EAR and coarse gaze estimation.

Facial action coding

  1. Ekman, P.; Friesen, W. V. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA. www.paulekman.com/facial-action-coding-system/
    The action-unit taxonomy the app's facial channels are named after; the app approximates AUs from landmarks/blendshapes rather than performing certified FACS coding.
  2. Kartynnik, Y.; Ablavatski, A.; Grishchenko, I.; Grundmann, M. (2019). Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. arXiv preprint arXiv:1907.06724. arxiv.org/abs/1907.06724
    The MediaPipe Face Mesh model (468 3D landmarks from a single webcam) that supplies the face ROI for rPPG and the geometry for facial features.
  3. Grishchenko, I.; Yan, G.; Bazavan, E. G.; Zanfir, A.; et al. (2023). Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction. arXiv preprint arXiv:2309.05782. arxiv.org/abs/2309.05782
    Describes the MediaPipe blendshape coefficients the app maps to approximate facial action units.

Voice analysis

  1. de Cheveigné, A.; Kawahara, H. (2002). YIN, a fundamental frequency estimator for speech and music. Journal of the Acoustical Society of America, 111(4), 1917–1930. doi:10.1121/1.1458024
    The F0 (pitch) estimator the app runs on microphone frames; its period estimates also feed the jitter computation.
  2. Teixeira, J. P.; Oliveira, C.; Lopes, C. (2013). Vocal Acoustic Analysis – Jitter, Shimmer and HNR Parameters. Procedia Technology, 9, 1112–1122. doi:10.1016/j.protcy.2013.12.124
    Gives the formulas the app follows for local jitter, shimmer and harmonics-to-noise ratio.
  3. Boersma, P.; Weenink, D.; Shchupak, A. (2026). Praat: doing phonetics by computer [Computer program], Version 7.0.02. Software, University of Amsterdam (praat.org). praat.org
    Reference implementation whose jitter/shimmer/HNR conventions the app's voice features are designed to match and can be checked against.
  4. Lippold, O. C. (1970). Oscillation in the stretch reflex arc and the origin of the rhythmical, 8–12 c/s component of physiological tremor. Journal of Physiology, 206(2), 359–382. doi:10.1113/jphysiol.1970.sp009018
    Physiological basis for the 8–12 Hz band the app inspects when estimating micro-tremor modulation in the voice.
  5. Inbar, G. F.; Eden, G. (1983). Physiological evidence for central modulation of voice tremor. Biological Cybernetics, 47(1), 1–12. doi:10.1007/BF00340063
    Peer-reviewed evidence for a low-frequency (5–20 Hz) tremor that frequency-modulates the voice, the signal that voice-stress analyzers claim to read.

Statistics

  1. Mahalanobis, P. C. (1936). On the generalised distance in statistics. Proceedings of the National Institute of Sciences of India, 2, 49–55 (reprinted in Sankhyā A, 80(Suppl 1), 1–7, 2018). doi.org/10.1007/s13171-019-00164-5
    The covariance-scaled distance the app uses to fuse all channels into one deviation score relative to the subject's own baseline.
  2. Ledoit, O.; Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis, 88(2), 365–411. doi:10.1016/S0047-259X(03)00096-4
    Shrinkage covariance estimator that keeps the baseline covariance invertible and well-conditioned when there are few calibration samples relative to the number of features.

Interview protocols

  1. Verschuere, B.; Ben-Shakhar, G.; Meijer, E. (Eds.) (2011). Memory Detection: Theory and Application of the Concealed Information Test. Cambridge University Press. doi:10.1017/CBO9780511975196
    Theory and design of the Concealed Information Test (and its contrast with the Control Question Test) that the app's questioning protocol follows.
  2. Meijer, E. H.; klein Selle, N.; Elber, L.; Ben-Shakhar, G. (2014). Memory detection with the Concealed Information Test: a meta analysis of skin conductance, respiration, heart rate, and P300 data. Psychophysiology, 51(9), 879–904. doi:10.1111/psyp.12239
    Laboratory effect sizes for CIT respiration and heart-rate measures, used to set realistic expectations for the app's cardio-respiratory channels.

Validity & limitations

  1. Harnsberger, J. D.; Hollien, H.; Martin, C. A.; Hollien, K. A. (2009). Stress and deception in speech: evaluating layered voice analysis. Journal of Forensic Sciences, 54(3), 642–650. doi:10.1111/j.1556-4029.2009.01026.x
    Double-blind evaluation finding a commercial voice-stress analyzer at chance with 40–65% false positives, cited to warn that the vocal channel is not a lie detector.
  2. National Research Council, Committee to Review the Scientific Evidence on the Polygraph (2003). The Polygraph and Lie Detection. The National Academies Press, Washington, DC. doi:10.17226/10420
    Authoritative review finding a weak scientific basis for polygraph testing and limited accuracy (especially for screening), cited for the app's disclaimer that its output is not evidence of deception.
  3. Davoodi, M.; Aspis, N.; Drori, Y.; Weiser-Bitoun, I.; et al. (2024). LieRHRV system for remote lie detection using heart rate variability parameters. Scientific Reports, 14(1), 30749. doi:10.1038/s41598-024-80480-5
    Peer-reviewed rPPG-derived HRV deception study, cited as small-sample (n = 39) laboratory feasibility work rather than evidence of field validity.