Building a transformer-based OCR model to read mathematical expressions from images and convert them into grammatically correct LaTeX. Uses a hybrid two-stage learning approach (transfer learning followed by reinforcement learning). A key challenge is ensuring grammatically correct LaTeX output using an ad-hoc abstract syntax tree to represent loss for a context-sensitive, Turing-complete language like LaTeX, which lacks a standard Backus-Naur Form (BNF).
Applying vector quantization (VQ) to reinforcement learning. The project involves developing a framework for learning prototype vectors that a model can use in its learned feature space. Our initial experiments have yielded promising results, particularly with a new cosine similarity loss-based approach to learning the prototype vectors. We have a paper in the draft stage.
2021
Modular Procedural Generation for Voxel Maps
Adarsh Pyarelal, Aditya Banerjee, and Kobus Barnard
