Milestone-Proposal talk:Theories on Neural Networks

Advocates’ Checklist

1. Is the proposal for an achievement rather than for a person? If the citation includes a person's name, have the proposers provided the required justification for inclusion of the person's name?
2. Was the proposed achievement a significant advance rather than an incremental improvement to an existing technology?
3. Were there prior or contemporary achievements of a similar nature? If so, have they been properly considered in the background information and in the citation?
4. Has the achievement truly led to a functioning, useful, or marketable technology?
5. Is the proposal adequately supported by significant references (minimum of five) such as patents, contemporary newspaper articles, journal articles, or citations to pages in scholarly books? At least one of the references should be from a peer-reviewed scholarly book or journal article. The full text of the material, not just the references, shall be present. If the supporting texts are copyright-encumbered and cannot be posted on the ETHW for intellectual property reasons, the proposers shall email a copy to the History Center so that it can be forwarded to the Advocate. If the Advocate does not consider the supporting references sufficient, the Advocate may ask the proposer(s) for additional ones.
6. Are the scholarly references sufficiently recent?
7. Does the proposed citation explain why the achievement was successful and impactful?
8. Does the proposed citation include important technical aspects of the achievement?
9. Is the proposed citation readable and understandable by the general public?
10. Will the citation be read correctly in the future by only using past tense? Does the citation wording avoid statements that read accurately only at the time that the proposal is written?
11. Does the proposed plaque site fulfill the requirements?
12. Is the proposal quality comparable to that of IEEE publications?
13. Are any scientific and technical units correct (e.g., km, mm, hertz, etc.)? Are acronyms correct and properly upper-cased or lower-cased? Are the letters in any acronym explained in the title or the citation?
14. Are date formats correct as specified in Section 6 of Milestones Program Guidelines? Helpful Hints on Citations, plaque locations
15. Do the year(s) appearing in the citation fall within the range of the year(s) included at the end of the title?
16. Note that it is the Advocate's responsibility to confirm that the independent reviewers have no conflict of interest (e.g., that they do not work for a company or a team involved in the achievement being proposed, that they have not published with the proposer(s), and have not worked on a project related to the funding of the achievement). An example of a way to check for this would be to search reviewers' publications on IEEE Xplore.

Reviewers’ Checklist

1. Is suggested wording of the Plaque Citation accurate?
2. Is evidence presented in the proposal of sufficient substance and accuracy to support the Plaque Citation?
3. Does proposed milestone represent a significant technical achievement?
4. Were there similar or competing achievements? If so, have the proposers adequately described these and their relationship to the achievement being proposed?
5. Have proposers shown a clear benefit to humanity?

In answering these questions, the History Committee asks that you apply a similar level of rigor to that used to peer-review an article, or evaluate a research proposal. Some elaboration is desirable. Of course the Committee would welcome any additional observations that you may have regarding this proposal.

Submission and Approval Log

Submitted date: 26 November 2024
Advocate approval date:
History Committee approval date:
Board of Directors approval date:

Citation as originally submitted

Theories on Neural Networks

From 1988 to 1996 at the Adaptive Systems Research Department at Bell Labs, Yann LeCun lead the development of a host of computational technologies that 25 years later formed the foundation of the Artificial Intelligence revolution of the early 21st century. These included the convolutional neural nets, and associated back propagation, regularization and pruning methods which were successfully applied to the world’s first handwritten letter and number recognition system.

Expert Review #1 -- Jbart64 (talk) 13:57, 18 February 2025 (UTC)

Dr. Joao Mota is an Assistant Professor in Signal and Image Processing at Heriot-Watt University. He holds a dual PhD degree from Carnegie Mellon University, US, and the Technical University of Lisbon, Portugal. From 2013 to 2016, he was a postdoctoral researcher at University College London, UK. His research focuses on the theoretical and practical aspects of high-dimensional data processing, machine learning, multimodal signal processing, inverse problems, optimization theory and algorithms, data science, and distributed information processing and control. He received the IEEE Signal Processing Society Young Author Best Paper Award for the paper "Distributed Basis Pursuit", published in IEEE Transactions on Signal Processing.

I received the following expert review from Dr. Mota on Friday, February 14, 2025 11:16 am:

Please find below my comments on the proposal. Please let me know if I should expand on anything.

1. Is suggested wording of the Plaque Citation accurate? The plaque citation is mostly accurate. Just a few suggestions:

- I would replace "...theory and creation of convolutional..." with "...theory and application of convolutional...". This is because CNNs were invented by Fukushima (https://doi.org/10.1109/TSSC.1969.300225), but LeCun revolutionized the application of CNNs to classification/recognition problems.

- I would also write either "convolutional neural networks" or "Convolutional Neural Networks".

2. Is evidence presented in the proposal of sufficient substance and accuracy to support the Plaque Citation? Yes. The text doesn't look complete though: it ends with "convolutio". Also, I would probably replace "deep neural network" by "neural network", as the network in the 1989 paper had just 3 hidden layers.

3. Does proposed milestone represent a significant technical achievement? Yes, definitely.

4. Were there similar or competing achievements? If so, have the proposers adequately described these and their relationship to the achievement being proposed? Yes, at the time the main competing framework was symbolic methods, as described in the paragraph "Obstacles". Before the (modern) revolution of deep learning starting in 2012, the other competing framework was support vector machines (SVMs), including kernel methods. But the performance of modern deep neural networks quickly surpassed the one of SVMs.

5. Have proposers shown a clear benefit to humanity? Yes. They connected the LeCun's contributions in the late 1980's to today's developments in AI, citing examples in creative writing, image analysis, and computer code generation. There are additional, perhaps more impactful applications, for example perception systems used in robotics and autonomous vehicles and automated image generation (both of which rely heavily on convolutional layers).

Mathini Sellathurai Milestone Advocate

Expert Review #2 -- Jbart64 (talk) 14:03, 18 February 2025 (UTC)

Sergios Theodoridis is currently Professor of Signal Processing and Machine Learning in the Department of Informatics and Telecommunications of the National and Kapodistrian University of Athens, Greece. His research interests lie in the areas of Adaptive Algorithms, Distributed and Sparsity-Aware Learning, Machine Learning and Pattern Recognition, Signal Processing for Audio Processing and Retrieval. He is the author of the book "Machine Learning: A Bayesian and Optimization Perspective", Academic Press, 2015, the co-author of the best-selling book "Pattern Recognition", Academic Press, 4th ed. 2009, the co-author of the book "Introduction to Pattern Recognition: A MATLAB Approach", Academic Press, 2010, and the co-editor of the book "Efficient Algorithms for Signal Processing and System Identification", Prentice Hall 1993. He has also co-authored three books in Greek, two of them for the Greek Open University. He is the recipient of the 2014 IEEE Signal Processing Society Education Award and the 2014 EURASIP Meritorious Service Award.

He is the co-author of seven papers that have received Best Paper Awards including the 2014 IEEE Signal Processing Magazine best paper award and the 2009 IEEE Computational Intelligence Society Transactions on Neural Networks Outstanding Paper Award. He serves as Editor-in-Chief for the IEEE Transactions on Signal Processing (2015-2017). He is Editor-in-Chief for the Signal Processing Book Series, Academic Press and co-Editor in Chief (with Rama Chellapa) for the E-Reference Signal Processing, Elsevier. He has served as a Distinguished Lecturer for the IEEE Signal Processing and IEEE Circuits and Systems Societies. He was Otto Monstead Guest Professor, Technical University of Denmark, 2012, and holder of the Excellence Chair, Dept. of Signal Processing and Communications, University Carlos III, Madrid, Spain, 2011.

He was the general chairman of EUSIPCO-98, the Technical Program co-chair for ISCAS-2006 and ISCAS-2013, co-chairman and co-founder of CIP-2008, co-chairman of CIP-2010 and Technical Program co-chair of ISCCSP-2014. He has served as President of the European Association for Signal Processing (EURASIP), as a member of the Board of Governors for the IEEE CAS Society, as a member of the Board of Governors (Member-at-Large) of the IEEE SP Society and as a Chair of the Signal Processing Theory and Methods (SPTM) technical committee of IEEE SPS. He has served as a member of the Greek National Council for Research and Technology and he was Chairman of the SP advisory committee for the Edinburgh Research Partnership (ERP). He has served as vice chairman of the Greek Pedagogical Institute and he was for four years member of the Board of Directors of COSMOTE (the Greek mobile phone operating company). He is Fellow of IET, a Corresponding Fellow of the Royal Society of Edinburgh (RSE), a Fellow of EURASIP and a Fellow of IEEE.

I received the following expert review from Dr. Theodoridis Thursday, February 6, 2025 9:18 am:

All reads well. However, I would add Yan's pioneering work on self-supervised learning. I attach related paper. Self-supervised learning is the key concept these days behind LLMs. my suggestion is:

=

From 1988 to 1996 at the Adaptive Systems Research Department at Bell Labs, Yann LeCun lead the development of a host of computational technologies that 25 years later formed the foundation of the Artificial Intelligence revolution of the early 21st century. These included the convolutional neural nets and the associated back propagation algorithm, regularization and pruning methods, and his pioneering work on self-supervised learning, which were successfully applied to the world’s first handwritten letter and number recognition system.

==

Mathini Sellathurai Milestone Advocate

Response to expert suggestions from the proposer -- Jbart64 (talk) 15:32, 18 February 2025 (UTC)

As of 02/17/2025, the proposer believes all items have been addressed from the expert reviewers, stating "There was an earlier recommendation on this topic where I made a change in the historical section to include the MOVPE role in epitaxy. I believe the earlier change satisfies this comment from the reviewer." Dave Bart

Advocate Approval -- Jbart64 (talk) 15:34, 18 February 2025 (UTC)

The proposal underwent a thorough review, both externally and internally, by the milestone subcommittee and its members. It is now in very good shape. Additionally, this milestone is critical and timely, given the significant impact currently being created by neural networks and deep learning. I believe it has the necessary and sufficient evidence. Mathini Sellathurai Milestone Advocate

Support of Milestone Proposal -- Jbart64 (talk) 15:37, 18 February 2025 (UTC)

I have worked closely with the proposer plus Brian Berg on the preparation of this milestone proposal and coordinated with Mathini Sellathurai,the Milestone Advocate. It has undergone significant review and modification during the proposal process. I fully support it and believe it satisfies the requirements for milestone approval. Dave Bart, Milestone Committee

citation 2nd and 3rd sentence confusing -- Amy Bix (talk) 20:16, 19 February 2025 (UTC)

I"m finding the both the second and third sentence of the proposed citation awkward and confusing - "Yann LeCun led work that included the theory and creation of Convolutional Neural Networks (CNNs), along with its backpropagation, regularization, pruning methods, and pioneering work on self-supervised learning to successfully apply to handwriting recognition and number recognition systems. Named LeNet, these enabled performing computer vision using deep neural network (DNN) architecture." My questions are:

a. In sentence 2, what exactly does "its" refer to? this is unclear.

b. In sentence 3, what exactly does "these" refer to? this is unclear.

c. It sounds as if LeNet was the name for the first practical CNN (or the first sets of these) - but the 3rd sentence makes it sound as if all CNNs are now known as LeNet. Can this be made more clear and accurate?

d. The phrase "these enabled performing computer vision" is awkward and unclear. Is "enabled performing" meant to act as one verb, or does the word "performing" here mean something else. In other words, could one make this shorter and simpler and just say "these enabled computer vision"? [whatever "these" refers to]

I recommend rewriting both sentence 2 and 3 for better clarification.

Re: citation 2nd and 3rd sentence confusing -- Bberg (talk) 23:47, 23 February 2025 (UTC)

I have tweaked the citation to remedy various issues and improve readability with these 65 words:

In 1989, research on computational technologies at Bell Laboratories helped establish the foundations of Artificial Intelligence. Key efforts managed by Yann LeCun led to the theory and creation of Convolutional Neural Networks, which included methods of backpropagation and pruning, regularization, and self-supervised learning. Named LeNet, this was a Deep Neural Network architecture which was applied to areas including computer vision, and handwriting and pattern recognition.

Brian Berg

Re: Re: citation 2nd and 3rd sentence confusing -- Tsizer (talk) 11:03, 24 February 2025 (UTC)

Largely accept the proposed revision. A slight rewording for clarity in Yann's role and to fix grammatical errors.

Tod Sizer

Re: Re: Re: citation 2nd and 3rd sentence confusing -- Jbart64 (talk) 14:10, 24 February 2025 (UTC)

:: I agree with the edits and updates and approve this version of the proposal. Dave Bart

Re: Re: citation 2nd and 3rd sentence confusing -- Amy Bix (talk) 04:41, 25 February 2025 (UTC)

Big improvement, thanks! Could I recommend some minor tightening of the last sentence: "Named LeNet, this Deep Neural Network architecture was applied to areas including computer vision, and handwriting and pattern recognition." Further, "was applied to areas" is a fairly underwhelming phrase. Could I recommend, ""Named LeNet, this Deep Neural Network architecture advanced developments in computer vision, handwriting and pattern recognition, and related technologies."

Remarks by Antonio Savini -- Savini (talk) 09:52, 25 February 2025 (UTC)

I am impressed by this achievement at the beginning of the current AI revolution A tiny question about the date in the title: 1989. In fact, from the original citation we understood that the research activity on convolutional neural networks at Bell Labs. spanned over almost a decade ( 1988-1986). And the LeNet architecture dates back to the second part of the 1990s. I would suggest to verify the date in the title.

Re: Remarks by Antonio Savini -- Mathini (talk) 23:23, 26 February 2025 (UTC)

LeNet was named later, but its invention dates precisely to 1989, according to publication and patent records.

-- Dmichelson (talk) 10:25, 26 February 2025 (UTC)

Title of the proposed milestone:

Convolutional Neural Networks, 1989

Plaque citation

In 1989, key efforts at Bell Laboratories led by Yann LeCun established the theory and creation of Convolutional Neural Networks, which included methods of backpropagation, pruning, regularization, and self-supervised learning. Named LeNet, this was a Deep Neural Network architecture that was applied to areas including computer vision, handwriting, and pattern recognition.

[51 words]

As I mentioned during the Milestones subcommittee call on Mon, 24 Feb, I don't believe that

"research on computational technologies at Bell Laboratories helped establish the foundations of Artificial Intelligence"

is helpful for two reasons:

- the award-winning work upon which this is based was performed at the University of Toronto in collaboration with Geoffrey Hinton

- it's not helpful from a public history perspective to conflate machine learning and pattern recognition (or even large language models, for that matter) with the more cognitive forms of computing

Use of name and conflating neural networks with AI -- John Vardalas (talk) 20:17, 26 February 2025 (UTC)

I want to see more evidence showing that Yann LeCunn alone should be singled out for recognition in this citation.

Were there any patents, in LeCunn’s name and assigned to Bell Labs regarding the innovations listed in the citation?

I also feel that, for the interests of public history, the proposal should be more careful about the one-to-one identification of AI with neural networks. A lot of different approaches have come under the AI tent.

I reserve judgement on this proposal until I have read the proposers's responses.

Re: Use of name and conflating neural networks with AI -- Mathini (talk) 23:31, 26 February 2025 (UTC)

"There are 7-8 patents on this topic assigned to Bell Labs, with LeCun named as an author on all of them. Given the extent of the work, the patents are co-authored by others from Bell Labs.

Helping with the Confusion with AI, Neural Network and computational technologies -- Mathini (talk) 23:17, 26 February 2025 (UTC)

A Convolutional Neural Network (CNN) – specifically, LeNet – is a type of artificial intelligence (AI). Handwriting recognition, being an AI application, is not merely a computational technology or problem. This is a widely accepted fact. Therefore, it is reasonable to claim that this work helped establish the foundations of AI. LeNet is among the first AI applications to address a real-world problem. Hence, I concur with the proposed citation. Reviewer2 supports this and reviewer2 is also an AI expert/pioneer.

Third External Review -- Mathini (talk) 19:20, 27 February 2025 (UTC)

Third Review is received on 27/3/2025, as below from Prof Mark Schmidt via email. Prof Mark Schmidt is a computer scientist and a machine learning expert and an Associate Professor, University of British Columbia (2019-Present). His career History includes the following: Canada CIFAR AI Chair, Alberta Machine Intelligence Institute (2019-Present); Canada Research Chair in Large-Scale Machine Learning (2016-Present); Assistant Professor, University of British Columbia (2014-2019); Postdoc, Simon Fraser University (2013-2014); Postdoc, Ecole Normale Superieure (2011-2013); Postdoc, University of British Columbia (2010); Ph.D., University of British Columbia (2005-2010); M.Sc., University of Alberta (2003-2005); B.Sc., University of Alberta (2000-2003).

1. Is suggested wording of the Plaque Citation accurate?

The statement "helped establish the foundations of Artificial Intelligence" is a bit odd. To me it sounds like general AI has been achieved. If you want to stay close to this statement it could be something like "helped establish fundamental models in Artificial Intelligence" or you could use something about how it revolutionized the field of computer vision and/or was a key development that led to the modern deep learning revolution.

2. Is evidence presented in the proposal of sufficient substance and accuracy to support the Plaque Citation? 3. Does proposed milestone represent a significant technical achievement?

These questions are similar so I will comment on them together.

In my opinion, it is hard to under estimate the impact of this work. I had already learned about it in textbooks when I was a grad student 20 years ago, and despite the delay before CNNs reached mainstream popularity Yann continued to advocate that we need to learn filters. The obscenely-influential AlexNet work from 2012 to me is the start of modern computer vision (with over 140k citations), and it is quite similar and clearly inspired by the LeNet architectures. I had already added CNNs as a major topic of study in my introductory machine learning course back in 2015. CNNs have evolved to be key components of state of the art systems for basically all computer vision tasks. In the last few years vision transformers have in some cases replaced CNNs, but it seems that this may be due to fashion rather than them actually working better than CNNs (most researchers I have talked to about this believe they perform equally well).

4. Were there similar or competing achievements? If so, have the proposers adequately described these and their relationship to the achievement being proposed?

I am not familiar enough with the history here, but at least according to the CNN Wikipedia article there are some relevant related works: - The Neocognitron was an earlier CNN, although the training procedure was different. - Time delay neural networks are apparently one-dimensional CNNs, and were trained by backpropagation.

So it seems the key differentiating factor is the combination of 2+ dimensional CNNs with backpropagation for training, as well as being architecturally similar to influential more-modern networks.

5. Have proposers shown a clear benefit to humanity?

My understanding is that variants of LeNet have been used to read some fraction of zip codes in the US, although I am not familiar with the precise details. Beyond this historical application, CNNs have now been used for basically all computer vision tasks. This ranges from things like face detection on phones to deployed medical imaging systems. Visually interpreting the world is so important to the human experience that having systems that are quite good at this opens the door to endless applications that could have a clear benefit.

Re: Third External Review -- Mathini (talk) 19:23, 27 February 2025 (UTC)

I see a mediation between the committee and the proposer from this reviewer in rewording the first sentence by replacing 'helped establish the foundations of Artificial Intelligence' with 'helped establish fundamental models in Artificial Intelligence.

Revised Citation -- Tsizer (talk) 16:32, 28 February 2025 (UTC)

Have provided a revised citation following the compromise language suggested by the Advocate and clarification of the last sentence as suggested by Amy Bix.

With respect to the fundamental work of Hinton at UToronto, I note that there is no implication that this work is competitive to the work recognized by the Nobel Committee but in fact complimentary. This is not surprising as Yann was a postdoc under Hinton prior to coming to Murray Hill. The AI work performed in Murray Hill was certainly influenced by the pioneering work at UToronto and a valuable and unique extension as recognized by the Turing Award in 2019 which recognized this fact when awarded jointly to Hinton, LeCun, and Yoshua Bengio(who was a Bell Labs researcher from 1992-1993 working with LeCun during that time and collaborating after he moved to UMontreal)

Tod Sizer

Re: Revised Citation -- Dmichelson (talk) 13:33, 2 March 2025 (UTC)

Title of the proposed milestone:

Convolutional Neural Networks, 1989

Plaque citation

In 1989, efforts at Bell Laboratories led by Yann LeCun were a key development that led to the deep learning revolution. The work established the theory and creation of Convolutional Neural Networks, which included methods of backpropagation, pruning, regularization, and self-supervised learning. Named LeNet, this was a Deep Neural Network architecture that was applied to areas including computer vision, handwriting, and pattern recognition.

[63 words]