Episode
#94 – Ilya Sutskever: Deep Learning
1:37:55
Published: Fri May 08 2020
Description
Ilya Sutskever is the co-founder of OpenAI, is one of the most cited computer scientist in history with over 165,000 citations, and to me, is one of the most brilliant and insightful minds ever in the field of deep learning. There are very few people in this world who I would rather talk to and brainstorm with about deep learning, intelligence, and life than Ilya, on and off the mic. Support this podcast by signing up with these sponsors: – Cash App – use code “LexPodcast” and download: – Cash App (App Store): https://apple.co/2sPrUHe – Cash App (Google Play): https://bit.ly/2MlvP5w EPISODE LINKS: Ilya's Twitter: https://twitter.com/ilyasut Ilya's Website: https://www.cs.toronto.edu/~ilya/ This conversation is part of the Artificial Intelligence podcast. If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter, LinkedIn, Facebook, Medium, or YouTube where you can watch the video versions of these conversations. If you enjoy the podcast, please rate it 5 stars on Apple Podcasts, follow on Spotify, or support it on Patreon. Here's the outline of the episode. On some podcast players you should be able to click the timestamp to jump to that time. OUTLINE: 00:00 - Introduction 02:23 - AlexNet paper and the ImageNet moment 08:33 - Cost functions 13:39 - Recurrent neural networks 16:19 - Key ideas that led to success of deep learning 19:57 - What's harder to solve: language or vision? 29:35 - We're massively underestimating deep learning 36:04 - Deep double descent 41:20 - Backpropagation 42:42 - Can neural networks be made to reason? 50:35 - Long-term memory 56:37 - Language models 1:00:35 - GPT-2 1:07:14 - Active learning 1:08:52 - Staged release of AI systems 1:13:41 - How to build AGI? 1:25:00 - Question to AGI 1:32:07 - Meaning of life
Chapters
In a conversation with Ilya Siskivar, co-founder and chief scientist of OpenAI, he discusses the history of money and cryptocurrency, as well as the future of artificial intelligence and its impact on society.
00:00 - 01:52 (01:52)
Summary
In a conversation with Ilya Siskivar, co-founder and chief scientist of OpenAI, he discusses the history of money and cryptocurrency, as well as the future of artificial intelligence and its impact on society.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The evolution of neural networks has led to the discovery of powerful techniques such as the Hessian Free Optimizer and the convolutional neural network, making it possible to train large neural networks on a lot of supervised data, which can represent very complicated functions.
01:52 - 08:57 (07:05)
Summary
The evolution of neural networks has led to the discovery of powerful techniques such as the Hessian Free Optimizer and the convolutional neural network, making it possible to train large neural networks on a lot of supervised data, which can represent very complicated functions.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
This podcast explores the idea that evolution does not have a cost function in the way we think of it in deep learning, raising questions about whether cost functions are holding us back in this field.
09:00 - 13:24 (04:24)
Summary
This podcast explores the idea that evolution does not have a cost function in the way we think of it in deep learning, raising questions about whether cost functions are holding us back in this field. It ponders the possibility of different architectures that can account for dynamic or multiple cost functions.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
Recurrent neural networks (RNNs) have been surpassed by transformers, but they still have incredible potential.
13:24 - 22:25 (09:01)
Summary
Recurrent neural networks (RNNs) have been surpassed by transformers, but they still have incredible potential. The recurrence in RNNs may be able to capture the timing phenomena important for neuronal firing in the brain.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The speaker acknowledges that achieving human-level understanding in both language and visual perception is a challenge that cannot be fully solved within just three months.
22:25 - 29:27 (07:01)
Summary
The speaker acknowledges that achieving human-level understanding in both language and visual perception is a challenge that cannot be fully solved within just three months. However, they express uncertainty about whether language understanding is fundamentally different from visual scene understanding or simply a more difficult generalization of the same problem.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
Increasing the size of a neural network slowly without early stopping leads to a rapid increase in performance.
29:27 - 40:17 (10:49)
Summary
Increasing the size of a neural network slowly without early stopping leads to a rapid increase in performance. Although making people laugh is the reason people click on the internet, important work can still be done by small groups and individuals.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The discussion revolves around the idea of neural networks as the search for small circuits and general intelligence as the search for small programs.
40:17 - 50:16 (09:59)
Summary
The discussion revolves around the idea of neural networks as the search for small circuits and general intelligence as the search for small programs. It explores whether the architecture needed to enable neural networks to reason will look similar to what we have today or require a different approach.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The self-awareness of neural nets could help them understand knowledge gaps and where to invest in increasing their skills, but the interpretation of their discovered knowledge isn't necessarily human-centric.
50:16 - 55:11 (04:55)
Summary
The self-awareness of neural nets could help them understand knowledge gaps and where to invest in increasing their skills, but the interpretation of their discovered knowledge isn't necessarily human-centric.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The conversation around language models in neural networks has shifted with the growth in data and compute, as empirically, larger language models show potential for understanding semantics without imposing a theory of language onto the learning mechanism.
55:11 - 1:04:16 (09:04)
Summary
The conversation around language models in neural networks has shifted with the growth in data and compute, as empirically, larger language models show potential for understanding semantics without imposing a theory of language onto the learning mechanism. This change in trajectory has been evident in the recent work of OpenAI.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
Active learning is a capability that naturally arises as problems that require it to pop up, especially when it comes to the optimization of selecting relevant data.
1:04:16 - 1:09:38 (05:22)
Summary
Active learning is a capability that naturally arises as problems that require it to pop up, especially when it comes to the optimization of selecting relevant data.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
Self-play will be of the key ideas for building Artificial General Intelligence (AGI).
1:09:39 - 1:16:31 (06:52)
Summary
Self-play will be of the key ideas for building Artificial General Intelligence (AGI). However, as we develop more powerful AI systems, we need to consider the negative consequences of such technologies.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The transfer from simulation to the real world is possible and has been demonstrated by different groups.
1:16:31 - 1:20:36 (04:04)
Summary
The transfer from simulation to the real world is possible and has been demonstrated by different groups. OpenAI demonstrated a robot hand trained entirely in simulation, which allowed for simulation to real transfer.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The progress of AI is determined by searching for cases where a system fails in a way humans would not, often leading to skepticism about the accuracy of deep learning when the mistakes made do not make sense.
1:20:36 - 1:25:07 (04:31)
Summary
The progress of AI is determined by searching for cases where a system fails in a way humans would not, often leading to skepticism about the accuracy of deep learning when the mistakes made do not make sense.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The discussion explores the potential of AGI systems in the 21st century and who will have direct possession and control of the technology, and also delves into the nature of human behavior and reward functions.
1:25:07 - 1:31:47 (06:39)
Summary
The discussion explores the potential of AGI systems in the 21st century and who will have direct possession and control of the technology, and also delves into the nature of human behavior and reward functions.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
The question of what the objective function of human existence is discussed, with the idea that our wants create the drives that ultimately determine this function.
1:31:48 - 1:36:34 (04:46)
Summary
The question of what the objective function of human existence is discussed, with the idea that our wants create the drives that ultimately determine this function. The possibility of training an AI objective perception system to recognize and internalize human judgments on different situations is also mentioned.
Episode#94 – Ilya Sutskever: Deep Learning
PodcastLex Fridman Podcast
Lex Friedman had an interesting discussion with Ilya S. S. Guevara on ideas and innovation.
1:36:35 - 1:37:35 (01:00)
Summary
Lex Friedman had an interesting discussion with Ilya S. S. Guevara on ideas and innovation. He also talked about using machine learning to simulate the brain and improve education.