[Home]
[
News]
[ Events]
[ People]
[ Research]
[ Education]
[Visitor Info]
[UCSD Only]
[Admin]
Trustworthy learning of mechanical systems, and Stiefel optimization with applications to transformer and optimal transport
Molei Tao
Georgia Institute of Technology
Abstract:
The interaction of machine learning and dynamics can lead to both new methodology for dynamics, and deepened understanding and/or efficacious algorithms for machine learning. This talk will focus on such interactions centered around structured dynamics. Specifically, I will first discuss data-driven learning and prediction of mechanical dynamics, for which I will demonstrate one strong benefit of having physics hard-wired into deep learning models; more precisely, how to make symplectic predictions, and how that provably improves the accuracy of long-time predictions. Then I will report a construction of momentum-accelerated gradient descent algorithms on Riemannian manifolds, focusing on a particular case known as Stiefel manifold. The treatment will be based on, firstly, the design of continuous-time optimization dynamics on the manifold, and then a thoughtful time-discretization that preserves all geometric structures. Since Stiefel manifold corresponds to matrices that satisfy orthogonality constraint, two practical applications will also be described: (1) we markedly improved the performance of trained-from-scratch Vision Transformer by appropriately placing orthogonality into its self-attention mechanism, and (2) our optimizer also makes the useful notion of Projection Robust Wasserstein Distance for high-dim. optimal transport even more effective.
Tuesday, September 27, 2022
11:00AM AP&M 2402 and Zoom: https://ucsd.zoom.us/my/mleok
News]
[ Events]
[ People]
[ Research]
[ Education]
[Visitor Info]
[UCSD Only]
[Admin]
Directors:
Randolph E. Bank
Philip E. Gill
Michael Holst
Administrative Contact:
[ Click to Send Email ]
Randolph E. Bank
Philip E. Gill
Michael Holst
Administrative Contact:
[ Click to Send Email ]
Molei Tao
Georgia Institute of Technology
Abstract:
The interaction of machine learning and dynamics can lead to both new methodology for dynamics, and deepened understanding and/or efficacious algorithms for machine learning. This talk will focus on such interactions centered around structured dynamics. Specifically, I will first discuss data-driven learning and prediction of mechanical dynamics, for which I will demonstrate one strong benefit of having physics hard-wired into deep learning models; more precisely, how to make symplectic predictions, and how that provably improves the accuracy of long-time predictions. Then I will report a construction of momentum-accelerated gradient descent algorithms on Riemannian manifolds, focusing on a particular case known as Stiefel manifold. The treatment will be based on, firstly, the design of continuous-time optimization dynamics on the manifold, and then a thoughtful time-discretization that preserves all geometric structures. Since Stiefel manifold corresponds to matrices that satisfy orthogonality constraint, two practical applications will also be described: (1) we markedly improved the performance of trained-from-scratch Vision Transformer by appropriately placing orthogonality into its self-attention mechanism, and (2) our optimizer also makes the useful notion of Projection Robust Wasserstein Distance for high-dim. optimal transport even more effective.
Tuesday, September 27, 2022
11:00AM AP&M 2402 and Zoom: https://ucsd.zoom.us/my/mleok