Visualization of the loss landscape and optimization path of a neural network

A useful loss-landscape plot should preserve the geometry of the optimization path, not merely project a high-dimensional surface onto an arbitrary plane.

Neural-network loss landscapesA loss landscape is the scalar training objective viewed as a function of all model parameters. Its dimension therefore equals the number of trainable parameters. live in very high-dimensional parameter spaces, whereas visualization is limited to one-dimensional curves or two-dimensional surfaces. Several methods attempt to close this dimensionality gap. The central idea is to choose a low-dimensional linear subspaceChoose one or two directions in parameter space and evaluate the loss only on the line or plane they span. The resulting slice can be plotted directly. that maximally preserves the optimization trajectory’s shapeThe projection should retain as much as possible of the distances, turns, and relative arrangement of the parameter iterates, so patterns in the plot remain meaningful for the original trajectory..

Source The longer draft is available in the HackMD version. This web version is still being formatted and expanded.

@misc{tikengnotsawo2022losslandscapeoptimizationpath,
  author = {Tikeng Notsawo, Pascal Jr.},
  title = {{Visualization of the loss landscape and optimization path of a neural network}},
  year = {2022},
  month = may,
  url = {https://tikquuss.github.io/blog/loss-landscape-optimization-path/},
  note = {Blog post}
}