Ilya Sutskever is the co-founder of OpenAI, is one of the most cited computer scientist in history with over 165,000 citations, and to me, is one of the most brilliant and insightful minds ever in the field of deep learning. There are very few people in this world who I would rather talk to and brainstorm with about deep learning, intelligence, and life than Ilya, on and off the mic. Support this podcast by signing up with these sponsors: – Cash App – use code “LexPodcast” and download: – Cash App (App Store): https://apple.co/2sPrUHe – Cash App (Google Play): https://bit.ly/2MlvP5w EPISODE LINKS: Ilya's Twitter: https://twitter.com/ilyasut Ilya's Website: https://www.cs.toronto.edu/~ilya/ This conversation is part of the Artificial Intelligence podcast. If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter, LinkedIn, Facebook, Medium, or YouTube where you can watch the video versions of these conversations. If you enjoy the podcast, please rate it 5 stars on Apple Podcasts, follow on Spotify, or support it on Patreon. Here's the outline of the episode. On some podcast players you should be able to click the timestamp to jump to that time. OUTLINE: 00:00 - Introduction 02:23 - AlexNet paper and the ImageNet moment 08:33 - Cost functions 13:39 - Recurrent neural networks 16:19 - Key ideas that led to success of deep learning 19:57 - What's harder to solve: language or vision? 29:35 - We're massively underestimating deep learning 36:04 - Deep double descent 41:20 - Backpropagation 42:42 - Can neural networks be made to reason? 50:35 - Long-term memory 56:37 - Language models 1:00:35 - GPT-2 1:07:14 - Active learning 1:08:52 - Staged release of AI systems 1:13:41 - How to build AGI? 1:25:00 - Question to AGI 1:32:07 - Meaning of life
In a conversation with Ilya Siskivar, co-founder and chief scientist of OpenAI, he discusses the history of money and cryptocurrency, as well as the future of artificial intelligence and its impact on society.
The evolution of neural networks has led to the discovery of powerful techniques such as the Hessian Free Optimizer and the convolutional neural network, making it possible to train large neural networks on a lot of supervised data, which can represent very complicated functions.
This podcast explores the idea that evolution does not have a cost function in the way we think of it in deep learning, raising questions about whether cost functions are holding us back in this field. It ponders the possibility of different architectures that can account for dynamic or multiple cost functions.
Recurrent neural networks (RNNs) have been surpassed by transformers, but they still have incredible potential. The recurrence in RNNs may be able to capture the timing phenomena important for neuronal firing in the brain.
The speaker acknowledges that achieving human-level understanding in both language and visual perception is a challenge that cannot be fully solved within just three months. However, they express uncertainty about whether language understanding is fundamentally different from visual scene understanding or simply a more difficult generalization of the same problem.
Increasing the size of a neural network slowly without early stopping leads to a rapid increase in performance. Although making people laugh is the reason people click on the internet, important work can still be done by small groups and individuals.
The discussion revolves around the idea of neural networks as the search for small circuits and general intelligence as the search for small programs. It explores whether the architecture needed to enable neural networks to reason will look similar to what we have today or require a different approach.
The self-awareness of neural nets could help them understand knowledge gaps and where to invest in increasing their skills, but the interpretation of their discovered knowledge isn't necessarily human-centric.
The conversation around language models in neural networks has shifted with the growth in data and compute, as empirically, larger language models show potential for understanding semantics without imposing a theory of language onto the learning mechanism. This change in trajectory has been evident in the recent work of OpenAI.
Active learning is a capability that naturally arises as problems that require it to pop up, especially when it comes to the optimization of selecting relevant data.
Self-play will be of the key ideas for building Artificial General Intelligence (AGI). However, as we develop more powerful AI systems, we need to consider the negative consequences of such technologies.
The transfer from simulation to the real world is possible and has been demonstrated by different groups. OpenAI demonstrated a robot hand trained entirely in simulation, which allowed for simulation to real transfer.
The progress of AI is determined by searching for cases where a system fails in a way humans would not, often leading to skepticism about the accuracy of deep learning when the mistakes made do not make sense.
The discussion explores the potential of AGI systems in the 21st century and who will have direct possession and control of the technology, and also delves into the nature of human behavior and reward functions.
The question of what the objective function of human existence is discussed, with the idea that our wants create the drives that ultimately determine this function. The possibility of training an AI objective perception system to recognize and internalize human judgments on different situations is also mentioned.
Lex Friedman had an interesting discussion with Ilya S. S. Guevara on ideas and innovation. He also talked about using machine learning to simulate the brain and improve education.