CCN meeting | Gido van de Ven (University of Groningen, The Netherlands), invited by Senne Braem

Lost, then found: transient forgetting in brains and machines

Incrementally learning new information from a non-stationary stream of data, referred to as 'continual learning', is a key feature of natural intelligence, but an open challenge for AI and deep learning. For example, artificial neural networks (ANNs) tend to catastrophically forget previous tasks or data distributions when trained on a new one. A popular way to deal with this kind of forgetting is to replay stored examples of past tasks when training on later tasks. Even with just a handful of stored examples, such replay can perform very well. Strikingly, however, we recently found that ANNs trained with replay still exhibit substantial forgetting when training on a new task begins, albeit that this forgetting is temporary and followed by a phase of performance recovery. Motivated by this counterintuitive yet consistent phenomenon in ANNs, in this presentation I will ask whether analogous forms of transient forgetting occur in humans.