Dear Fellow Scholars, this is 2 minutes paper Károly Zsolnai-Fehér
This is an incredible paper from OpenAI
in which the goal is to teach AI to read a piece of text
and perform common nature language processing operation.
For instance, answering questions, completing text,
例如 回答问题 文本补充
reading comprehension, summarization, and more.
And not only that,
but additionally, the AI has to be able to perform these tasks
with as little supervision as possible.
This means that we seek to unleash the algorithm that they call GPT-2
to read the internet and learn the intricacies of our language by itself.
To perform this, of course, we need a lot of training data
当然 为了达到目的 我们需要大量的训练数据
and here, the AI reads 40 gigabytes of internet text,
which is 40 gigs of non-binary plaintext data,
which is a stupendously large amount of text.
It is always hard to put these big numbers in context,
so as an example, to train similar text completion algorithms,
AI people typically reach out to a text file
containing every significant work of Shakespeare himself
and this file is approximately 5 megabytes.
so the 40 gigabytes basically means an amount of text
that is 8000 times the size of Shakespeare’s works.
That’s a lot of text
And now, let’s have a look at how it fares with the text completion part.
This part was written by a human,quoting：
“In a shocking finding,
scientist discovered a herd of unicorns living in a remote
previously unexplored valley, in the Andes Mountains.
even more surprising to researchers was the fact that the unicorns spoke perfect English. ”
And the AI continued the text the following way, quoting a short snippet of it:
“The scientist named the population, after their distinctive horn, Ovid’s Unicorn.
These four-horned, silver-white unicorns were previously unknown to science.”
Wow! Now note that this is clearly not perfect.
if there is even such a thing as a perfect continuation, and it took 10 tries,
which means that the algorithm was run 10 times
and the best result was cherrypicked and recorded here.
And despite all of these,
this is a truly incredible result,
especially given that the algorithm learns on its own.
After giving it a piece of text,
it can also answer questions in a quite competent manner.
Worry not, later in this video,
I will show you more of these examples and likely talk over them
so if you are curious,
feel free to pause the video
while you read the prompts and their completions.
The validation part of the paper reveals
that this method is able to achieve state-of-the-art results
on several language modeling tasks,
and you can see here
that we still shouldn’t expect it to match a human in terms of reading comprehension,
which is the question answering test.
More on that in a moment.
So, there are plenty of natural language processing algorithms out there
that can perform some of these tasks,
in fact, some articles already stated that there is not much new here,
it’s just the same problem,
but stated in a more general manner, and with more compute.
A ha! It is not the first time that this happens.
Remember our video by the name “The Bitter lesson”?
I’ve put a link to it in the video description,
but in case you missed it,
let me quote how Richard Sutton addressed his situation:
“The bitter lesson is based on the historical observations
that 1 ) AI researchers have often tried to build knowledge into their agents,
2 ) this always helps in the short term
and is personally satisfying to the researcher,
but 3 ) in the long run it plateaus
and even inhibits further progress
and 4 ) breakthrough progress eventually arrives by an opposing approach
based on scaling computation by search and learning.
The eventual success is tinged with bitterness,
and often incompletely digested,
because it success over a favored, human-centric approach.“
So what is the big lesson here?
Why is GPT-2 so interesting?
Well, big lesson number one is that
this is one of the clearer cases of what the quote was talking about,
where we can do a whole lot given a lot of data and compute power,
and we don’t need to insert too much additional knowledge into our algorithms.
And lesson number two,
as a result, this algorithm becomes quite general
So it can perform more tasks than most other techniques.
This is an amazing value proposition.
I will also add
that not every learning technique scales well when we add more compute,
in fact,you can see here yourself
that even GPT-2 plateaus on the summarization task.
Making sure that these learning algorithms scale well
is a great contribution in and of itself
and should not be taken for granted.
There has been a fair bit of discussion on
whether OpenAI should publish the entirety of this model.
They opted to release a smaller part of the source code
and noted that they are aware
that the full model could be used for nefarious purposes.
Why did they do this?
What is the matter with everyone having an AI
with a subhuman-level reading comprehension?
Well, so far, we have only talked about quality.
But another key part is quantity.
And boy, are these learning methods superhuman in terms of quantity
just imagine that they can write articles
with a chosen topic and sentiment all day long.
and much quicker than human beings
Also note that the blueprint of the algorithm is described in the paper,
and a top-tier research group is expected to be able to reproduce it.
So does one release the full source code and models or not?
This is a quite difficult question:
we need to keep publishing both papers and source code to advance science,
but we also have to find new ways to do it
in an ethical manner.
This needs more discussion
and would definitely be worthy of a conference-style meeting, or more.
There is so much to talk about,
and so far,we have really only scratched the surface,
So make sure to have a look in the video description,
I left a link to the paper
and some more super interesting reading materials for you.
Make sure to check them out.
Also just a quick comment on
why this video came so late after the paper has appeared.
Since there were a lot of feelings and intense discussion on
whether the algorithm should be published or not,
I was looking to wait until the dust settles
and there is enough information out there
to create a sufficiently informed video for you.
This of course means that we are late to the party
and missed out on the whole lot of views and revenue.
But that’s okay.
In fact, that’s what we’ll keep doing going forward
to make sure you get the highest quality information that I can provide.
If you have enjoyed this episode
and would like to help us,
please consider supporting us on Patreon.
Remember our motto,
a dollar a month is almost nothing,
but it keeps the papers coming.
And there are hundreds of papers on my reading list.
As always, we are available through Patreon.com/TwoMinutePapers,
and the link is also available in the video description.
Thanks for watching and for your generous support,
and see you next time
Dear Fellow Scholars, this is 2 minutes paper Károly Zsolnai-Fehér