Nguyen Minh Khoa
Back to selected work

HMER Research

Handwritten mathematical expression recognition

Research on converting handwritten mathematical expressions into structured sequences, including experiments in masked-language pretraining and multi-task learning.

From handwritten mathematics to structured LaTeX sequences.

Role
Research contributor
Period
2024 – 2026
Status
Research contributions

A handwritten expression is converted into a machine-readable LaTeX sequence.

A handwritten mathematical expression passes through visual encoding and structure-aware sequence modeling to become a machine-readable sequence.
CROHME sample*A handwritten expression is converted into a machine-readable LaTeX sequence.

* Sample from CROHME 2016 dataset. (Source)

Overview

Handwritten Mathematical Expression Recognition turns an image of handwritten mathematics into a structured sequence. Beyond reading individual symbols, the model must parse two-dimensional spatial layouts like fractions, superscripts, subscripts, and roots.

My role

  • Pretraining & multi-task learning

    Implemented Mask CoMER methods and contributed baseline reproduction, ablations, error analysis and paper writing; contributed multi-task learning experiments to MTL + CBARM.

  • Model training & evaluation

    Trained and evaluated PyTorch models, with experiment runs tracked in Weights & Biases.

Tech stack

Core ML

  • Python
  • PyTorch
  • PyTorch Lightning
  • CUDA

Training & tracking

  • Weights & Biases
  • TorchMetrics

Data & vision

  • CROHME
  • Torchvision + OpenCV
  • Pillow
  • Einops

Research pipeline

One recognition pipeline, with two complementary research interventions aimed at preserving context and mathematical structure.

  1. 01Handwritten expression

    A grayscale image contains symbols and two-dimensional layout.

  2. 02Visual encoding

    The encoder converts the image into spatial visual features.

  3. 03Structure-aware modeling

    The decoder uses visual context to generate a structured token sequence.

  4. 04Machine-readable output

    The sequence can be reconstructed as a mathematical expression.

Mask CoMER

A two-stage strategy: masked-language pretraining first, then the main HMER task with stochastic-depth regularization.

Simplified Mask CoMER diagram showing masked-token pretraining followed by HMER fine-tuning.
An overview of Mask CoMER methodology

MTL + CBARM

The main LaTeX-sequence task is trained alongside a structural auxiliary task, while CBARM refines cross-attention using channel and spatial cues.

Simplified multi-task learning and CBARM diagram with main and auxiliary targets feeding a shared model.
An overview of MTL + CBARM methodology

Research connection

ConferenceICDAR · 2025Published

Mask CoMER: Enhancing Handwritten Mathematical Expression Recognition with Masked Language Pretraining and Regularization

Nam Van Hai Phan, Khoa Minh Nguyen, Trung Thanh Nguyen, Trung Thanh Pham, Phuong-Nam Tran, Duc Ngoc Minh Dang

International Conference on Document Analysis and Recognition

Co-author

Masked-language pretraining and stochastic-depth regularization for HMER, achieving up to +5 ExpRate points over CoMER on CROHME.

Paper Code Project
Abstract

Excerpt from the abstract

We propose Mask CoMER, a two-stage training approach that combines a novel Masked Language Model Pretraining objective with stochastic depth regularization.
Read the full abstract at source
BibTeX 
@inproceedings{maskcomer2025,
  title = {Mask CoMER: Enhancing Handwritten Mathematical Expression Recognition with Masked Language Pretraining and Regularization},
  author = {Nam Van Hai Phan and Khoa Minh Nguyen and Trung Thanh Nguyen and Trung Thanh Pham and Phuong-Nam Tran and Duc Ngoc Minh Dang},
  booktitle = {International Conference on Document Analysis and Recognition},
  year = {2025},
  doi = {10.1007/978-3-032-04624-6_22}
}
JournalJIT · 2026Published

Enhancing handwritten mathematical expression recognition with convolutional block attention refinement module and multi-task learning

Thanh Trung Nguyen, Nhut Minh Nguyen, Nam Van Hai Phan, Khoa Minh Nguyen, Trung Thanh Pham, Phuong-Nam Tran, Cuong Tuan Nguyen, Duc Ngoc Minh Dang

Journal of Information and Telecommunication

Co-author

Multi-task learning framework with convolutional block attention refinement module (CBARM) for handwritten mathematical expression recognition, published online 19 August 2026.

Paper Code Project
Abstract

Excerpt from the abstract

Handwritten Mathematical Expression Recognition (HMER) is challenging due to complex structural relationships, implicit symbols, and variations in handwriting styles.
Read the full abstract at source
BibTeX 
@article{mtlcbarmhmer2026,
  title = {Enhancing handwritten mathematical expression recognition with convolutional block attention refinement module and multi-task learning},
  author = {Thanh Trung Nguyen and Nhut Minh Nguyen and Nam Van Hai Phan and Khoa Minh Nguyen and Trung Thanh Pham and Phuong-Nam Tran and Cuong Tuan Nguyen and Duc Ngoc Minh Dang},
  journal = {Journal of Information and Telecommunication},
  year = {2026},
  doi = {10.1080/24751839.2026.2716432},
  note = {Published online 19 August 2026; final issue metadata pending}
}

Outcome & current status

The HMER research record includes Mask CoMER, published at ICDAR 2025, and a Journal of Information and Telecommunication paper published online on 19 August 2026. The publication entries below give the authorship, venue and status for each paper.

Explore all work