Patrick H. Chen

Bio

I am an Assistant Professor in Computer Science at Binghamton University, leading the FUEL lab. I obtained my Ph.D. from UCLA, supervised by Cho-Jui Hsieh. I obtained my Bachelor's degree in Electrical Engineering and History from National Taiwan University. Before starting my Ph.D. journey, I worked with Shen-Shyang Ho as a Research Assistant at Nanyang Technological University. Before joining Binghamton University, I worked as a Research Scientist at ASAPP.

Research

My research aspiration is to add FUEL to machine learning!! I aim to work on Fundamental, Uncertainty, and Efficiency aspects of Learning! To make models reliably applicable, we studied the fundamental aspects of ML, such as factualness and robustness, and analyzed uncertainty issues in various applications. To make models more efficient for practical usage, I focused on 1) compressing machine learning models to make them deployable on devices with limited memory and 2) accelerating the training and inference time of machine learning models to meet latency requirements. If you are interested in joining the FUEL lab, please fill out this form. If you are interested in any collaboration/discussion, please check my schedule and feel free to drop me an invitation!

Selected Publications [Full List]

DRONE: Data-aware Low-rank Compression for Large NLP Models

Patrick H. Chen, Hsiang-fu Yu, Inderjit S. Dhillon, and Cho-jui, Hsieh

In Advances in Neural Information Processing Systems (NeurIPS), 2021

[Abstract] [PDF]

The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings efficiency challenges, including inference speed and model size when deploying models on mobile devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices are not low-rank and thus canonical matrix decompositions do not lead to efficient approximations. In this paper, we observe that the learned representation of each layer lies in a low-dimensional space. Based on this observation, we propose DRONE (data-aware low-rank compression), a provably optimal low-rank decomposition of weight matrices, which has a simple closed form solution that can be efficiently computed. DRONE can be applied to both fully-connected and self-attention layers appearing in the BERT model. In addition to compressing standard models, our method can also be used on distilled BERT models to further improve the compression rate. Experimental results show that DRONE is able to improve both model size and inference speed with limited loss in accuracy. Specifically, DRONE alone achieves 1.92x speedup on the MRPC task with only 1.5% loss in accuracy, and when DRONE is combined with distillation, it further achieves over 12.3x speedup on various natural language inference tasks.

GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking

Patrick H. Chen, Si Si, Yang Li, Ciprian Chelba and Cho-jui, Hsieh

In Advances in Neural Information Processing Systems (NeurIPS), 2018

[Abstract] [PDF]

Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing input tokens and a softmax layer for generating output tokens. For problems with a very large vocabulary size, the embedding and the softmax matrices can account for more than half of the model size. For instance, the bigLSTM model achieves great performance on the One-Billion-Word (OBW) dataset with around 800k vocabulary, and its word embedding and softmax matrices use more than 6GBytes space, and are responsible for over 90% of the model parameters. In this paper, we propose GroupReduce, a novel compression method for neural language models, based on vocabulary-partition (block) based low-rank matrix approximation and the inherent frequency distribution of tokens (the power-law distribution of words). The experimental results show our method can significantly outperform traditional compression methods such as low-rank approximation and pruning. On the OBW

Overcoming Catastrophic Forgetting by Bayesian Generative Regularization

Patrick H. Chen, Wei Wei, Cho-jui, Hsieh, and Bo Dai

In Conference of Machine Learning (ICML), 2021

[Abstract] [PDF]

In this paper, we propose a new method to over-come catastrophic forgetting by adding generative regularization to Bayesian inference frame-work. Bayesian method provides a general frame-work for continual learning. We could further construct a generative regularization term for all given classification models by leveraging energy-based models and Langevin dynamic sampling to enrich the features learned in each task. By combining discriminative and generative loss together, we empirically show that the proposed method outperforms state-of-the-art methods on a variety of tasks, avoiding catastrophic forgetting in continual learning. In particular, the proposed method outperforms baseline methods over 15%on the Fashion-MNIST dataset and 10%on the CUB dataset.