Difference between revisions of "Milestone-Proposal:MPEG integrated circuits"

 
(23 intermediate revisions by the same user not shown)
Line 12: Line 12:
 
|a3=1986
 
|a3=1986
 
|a1=Multimedia Integrated Circuits for MPEG, 1986
 
|a1=Multimedia Integrated Circuits for MPEG, 1986
|plaque citation=THOMSON (now STMicroelectronics) pioneered the family of multimedia integrated circuits, accelerating Moving Picture Experts Group (MPEG) standards. Discrete Cosine Transform designed with ENST (now Telecom ParisTech) was the first. Innovative memory reduction techniques, highly optimized dedicated hardware accelerators, super-integration of heterogeneous processor units achieved significant performances and cost reduction with minimized energy consumption. These integrated circuits were the key enablers of digital multimedia services to entertain end-users in everyday life.
+
|plaque citation=THOMSON (now STMicroelectronics) pioneered the family of multimedia integrated circuits, accelerating Moving Picture Experts Group (MPEG) standards. Discrete Cosine Transform designed with ENST (now Telecom ParisTech) was the first. Innovative memory reduction techniques, low bandwidth motion estimation, compensation, super-integration of heterogeneous processor units achieved significant performances and cost reduction with minimized energy consumption. These integrated circuits were the key enablers of digital multimedia services to entertain end-users in everyday life.
 
|a2b=France Section
 
|a2b=France Section
 
|IEEE units paying={{IEEE Organizational Unit Paying
 
|IEEE units paying={{IEEE Organizational Unit Paying
Line 37: Line 37:
 
}}
 
}}
 
|a2a=(45.203333, 5.695833)
 
|a2a=(45.203333, 5.695833)
|a7=The Grenoble site is of premier importance for STMicroelectronics. It was owned by THOMSON SEMICONDUCTEURS, then merged with SGS in 1987 to form SGS-Thomson (now STMicroelectronics). Nowadays is the biggest R&D site of the company, hosting many product divisions from the three Product Groups of the company, including silicon and software design, test, and advanced packaging developments. In this site MPEG chips were conceived and designed
+
|a7=The Grenoble site is of premier importance for STMicroelectronics. It was owned by THOMSON SEMICONDUCTEURS, then merged with SGS in 1987 to form SGS-Thomson. Nowadays is the biggest R&D site of the company, hosting many product divisions from the three Product Groups of the company, including silicon and software design, test, and advanced packaging developments. In this site MPEG chips were conceived and designed
 
|a8=Yes, even if part of them were restructured they still exists @ STMicroelectronics, 12, rue Jules Horowitz F-38000 Grenoble, France
 
|a8=Yes, even if part of them were restructured they still exists @ STMicroelectronics, 12, rue Jules Horowitz F-38000 Grenoble, France
 
|mounting details=Plaques will be installed in a public place at the main entrance as shown in figure 1. It will be in front of company security personnel who are monitoring entrance 24 hours in a day, every day in a year. Very close as indicated there is the public road.  Plaque will be placed at 12, rue Jules Horowitz F-38000 Grenoble, France.  GPS coordinates are: 45°12’12” N ; 05°41’45” E (45.203333, 5.695833). Therefore, interested visitors will not need to be escorted by the company security personnel to watch the plaque or to take a photo with it.  The milestone plaque will be installed right in front of the main entrance access. It is where employees, visitors and customers must pass through to get into the Grenoble offices on daily basis.  It is monitored continuously 24h/7d by security human resources and surveillance camera. It is a public place, right outside the site restricted perimeter.
 
|mounting details=Plaques will be installed in a public place at the main entrance as shown in figure 1. It will be in front of company security personnel who are monitoring entrance 24 hours in a day, every day in a year. Very close as indicated there is the public road.  Plaque will be placed at 12, rue Jules Horowitz F-38000 Grenoble, France.  GPS coordinates are: 45°12’12” N ; 05°41’45” E (45.203333, 5.695833). Therefore, interested visitors will not need to be escorted by the company security personnel to watch the plaque or to take a photo with it.  The milestone plaque will be installed right in front of the main entrance access. It is where employees, visitors and customers must pass through to get into the Grenoble offices on daily basis.  It is monitored continuously 24h/7d by security human resources and surveillance camera. It is a public place, right outside the site restricted perimeter.
Line 46: Line 46:
 
[[image:Figure2b.jpg|thumb|Figure 2 How to get the plaque from Lyon airport]]
 
[[image:Figure2b.jpg|thumb|Figure 2 How to get the plaque from Lyon airport]]
  
Since mid 80s the technology objective of STMicroelectronics was to conceive, ahead of MPEG standards, and introduce innovative hardware solutions to be produced in high volumes, able to achieve several order of magnitude speed acceleration, compared to unfeasible, hypothetical, very costly full software implementation of MPEG decoding and encoding functionalities, by introducing for the first time very advanced techniques such as RAM memory compression, reductions and associated embedding on chip, dedicated and customized hardware implementations in order to save precious silicon area per technology node being used (starting from 2um and the next ones), super integrate new functionalities, as the needs evolved in the next 30 years, in a less than a watt power envelope.  
+
Since mid 80s the technology objective of STMicroelectronics (ST) was to conceive, ahead of MPEG standards, and introduce innovative hardware solutions to be produced in high volumes, able to achieve several order of magnitude speed acceleration, compared to unfeasible, hypothetical, very costly full software implementation of MPEG decoding and encoding functionalities, by introducing for the first time very advanced techniques such as RAM memory compression, reductions and associated embedding on chip, dedicated and customized hardware implementations in order to save precious silicon area per technology node being used (starting from 2um and the next ones), super integrate new functionalities, as the needs evolved in the next 30 years, in a less than a watt power envelope.  
The DCT (discrete cosine transform forward and backward) integrated circuit, the first in the family, was designed to be very reliably and produced in mass volumes such that was subsequently used in the Cassini-Huygens probe for transmitting compressed images of Saturn for 20 years. This is the most important proof that the integrated circuits was very resilient to cosmic rays and therefore also for on earth multimedia entertainment applications.
+
The DCT (discrete cosine transform forward and backward) integrated circuit, the first in the family, was designed to be very reliable and produced in mass volumes such that was subsequently used in the Cassini-Huygens probe for transmitting compressed images of Saturn for 20 years. This is an important proof that the integrated circuits was very resilient to cosmic rays and therefore also for (on earth) multimedia entertainment applications.
  
Specific objectives, that motivated STMicroelectronics (short name ST) to pioneer Mpeg family of System on a Chips and pursue associated CMOS silicon technology developments to implement it and starting since 1986 and onward, were:
+
Specific objectives, that motivated ST to pioneer Mpeg family of System on a Chips and to pursue associated CMOS silicon technology developments to implement it, starting since 1986 and onward, were:
  
1) To create reliable and low complex digital CMOS building (integrated circuits) blocks at mass production maturity to dramatically accelerate real time MPEG video decoding and encoding functionalities at increasing image resolutions and frame rates.
+
1) To create reliable and low complex digital CMOS building (integrated circuits) blocks at mass production maturity to dramatically accelerate MPEG video decoding and encoding functionalities to achieve real time at increasing image resolutions and frame rates.
  
 
2) To super-integrate such building blocks in more complex integrated circuits and SoCs to accelerate MPEG 1, 2, 4, H264, HEVC and subsequent standardized MPEG specifications, at the same time they were under development by ISO/IEC SC29 WG11 (known as MPEG committee). In other words before the standards were officially frozen and be formally published.
 
2) To super-integrate such building blocks in more complex integrated circuits and SoCs to accelerate MPEG 1, 2, 4, H264, HEVC and subsequent standardized MPEG specifications, at the same time they were under development by ISO/IEC SC29 WG11 (known as MPEG committee). In other words before the standards were officially frozen and be formally published.
  
3) To achieve lowest on chip memory footprint and bit-depth of computing data paths to dramatically save silicon area given the existing (from 2µm bulk to 25nm FDSOI) ST CMOS technology constraints for mass production and reliable manufacturability with marginal chip defectivity.
+
3) To achieve lowest on chip memory footprint and bit-depth of computing data paths to dramatically save silicon area given the existing (from 2µm bulk to 25nm FDSOI) in house developed CMOS technology constraints for mass production and reliable manufacturability with marginal chip defectivity.
  
4) To achieve minimal power consumption, to and below 1W across different silicon technologies generations and starting from 2um silicon gate manufacturing processes. Successive lower dimension lithography, were needed considering an increasing need to super integrate functionalities, heterogeneous hardware accelerators, microprocessors, graphic engines, crypto engines, a rich set of peripherals and interfaces etc. ST developed in house all subsequent CMOS technologies such as 0.5um, 0.35um up to the most sophisticated 25nm FDSOI, fully depleted silicon on insulator) needed to sustain the pace of super integrating the functionalities.  
+
4) To achieve minimal power consumption, to and below 1W across different silicon technology generations, starting from 2um silicon gate manufacturing processes. Successive lower dimension lithography, were needed considering an increasing need to super integrate functionalities, heterogeneous hardware accelerators, microprocessors, graphic engines, crypto engines, a rich set of peripherals and interfaces etc. ST developed in house all subsequent CMOS technologies such as 0.5um, 0.35um up to the most sophisticated 25nm FDSOI, fully depleted silicon on insulator) needed to sustain the pace of super integrating the multimedia functionalities.  
  
5) To remove any need of costly and bulky heat-sinks as well as complex and motorized cooling mechanisms to offer an easier viable integration into thinner and very cheap video equipment manufactured systems with very affordable costs.
+
5) To remove any need of costly and bulky heat-sinks as well as complex and motorized cooling mechanisms to offer an easier viable integration into thinner and very cheap video equipment manufactured systems with very low and affordable costs.
  
6) To dramatically reduce as low as possible video communication bandwidth between the chip and the external DRAM memories to minimize memory requirements of those chips and associated costly power consumption.
+
6) To dramatically reduce as low as possible video communication bandwidth between the chip and the external DRAM memories, to minimize memory requirements of those and associated power consumption below 1 W.
  
7) To control those chips with very simple and low computing power 8bits micro controllers and incrementally (as requirements has grown) with more sophisticated 32 bits: this to support flexibility and complex operating systems that required further software computing needs to offer un precedented user experience, multitasking and programmability. Including the use of advanced graphics to ease content accessibility and fruition.
+
7) To control those chips with very simple and low computing power 8bits micro controllers and incrementally (as requirements has grown) with more sophisticated 32 bits ones; this to add flexibility and to support complex operating systems that required further software computing needs to offer un precedented user experience, multitasking and programmability. Including the use of advanced graphics to ease content accessibility and fruition.
  
8) To enable viable, widely adopted, broad range of diversified digital multimedia applications and services, implying affordable software developments and deployments to a huge range of end users at worldwide level with different regional languages, habits and needs.
+
8) To enable viable, widely adopted, broad range of diversified digital multimedia applications and services, implying affordable software developments, end to end full stack and deployments to a huge range of end users at worldwide level with different regional languages, habits and needs.
  
In the following, there is the summary of achievements by STi (no named by ST) family of integrated circuits subject of the present milestone proposal. To meet the above-mentioned challenging objectives and long-term technology vision, ST, since 1986, has developed following key innovative technologies and chips. A few but very relevant set of them will be described, because the family is too huge to report completely in this document because was spanning in 30 years of intense and diversified developments.
+
In the following, there is the summary of achievements by the family of MPEG integrated circuits subject of the present milestone proposal. To meet the above-mentioned challenging objectives and long-term technology vision, since 1986, ST has developed following key innovative technologies and chips. A few but very relevant set of them will be described, since the chip family is too numerous to be reported exhaustively in this document and as it was spanning in 30 years of intense and diversified developments.
  
1) Seminal work on ST CMOS technologies, started at ENST [1.24] [1.25] [7.0] with Thomson Semiconductors (now STMicroelectronics), to produce the architecture of an economically viable chip implementing various DCY block sizes 16x16, 16x8, 8x8, 8x4, 4x4 very ahead of the requirements MPEG will specify in the subsequent years . The main features of the chip under development with THOMSON SEMICONDUCTEURS was using 2 µm CMOS lithography first and next 1.25 µm CMOS technology . Design was embodying 70,400 Transistors, in 25 mm2, with Internal clock 13.5 MHz, forward and backward  DCT, 8 bits pixels, 16 bits internal accuracy. The chip was sampled by THOMSON SEMICONDUCTEUR in October 1987.
+
1) Seminal work on started at ENST [1.24] [1.25] [7.0] with Thomson Semiconductors (now ST), to produce the architecture of an economically viable chip implementing various DCT block sizes 16x16, 16x8, 8x8, 8x4, 4x4 much ahead of the requirements MPEG specified in the subsequent years . The chip under development with Thomson Semiconductors was using 2 µm CMOS lithography first and then 1.25 µm CMOS technology . Design was embodying 70,400 Transistors, in 25 mm2, with Internal clock 13.5 MHz, forward and backward  DCT, 8 bits pixels, 16 bits internal accuracy. The chip was sampled by Thomson Semiconductors in October 1987.
  
2) Then STV3200 [1.1] [1.2] was a dedicated integrated circuit to accelerate the discrete cosine transform (DCT). The two-dimensional forward DCT (FDCT) and inverse DCT (IDCT) was implemented for various image block sizes and the pixel rate was up to 15.0 MHz The circuit architecture was fully bidirectional with a 9-bit magnitude pixel data bus and a 12-bit magnitude coefficient data bus programmed as input or output depending on the selection of Forward DCT or Inverse DCT function. DCT was so complex that to be implemented on a CPU required about 1 billion additions per second (1 giga Hertz operations per second) to process moving pictures in real time. CPUs and CMOS silicon process were unable to achieve such a very high frequency, equivalently implying the computing power of fifty Motorola 68020 [2.1], one of the faster micro-processors in the 1980s. The integrated circuit went officially in production at the end of 1987 and was a world premiere product. Since 1988, this component was successfully sold by SGS-Thomson. It implemented key patents [4.1][4.2] for rearranging, permuting or selecting data according to predetermined rules, for changing the order of data flow, e.g. matrix transposition, LIFO (last in first out) buffers; internally compensating and handling overflow or underflow  that was key for DCT and IDCT accurate and reliable implementations.
+
2) Then STV3200 [1.1] [1.2] was a dedicated integrated circuit to accelerate the discrete cosine transform (DCT). The two-dimensional forward DCT (FDCT) and inverse DCT (IDCT) were implemented for various image block sizes and the pixel rate was up to 15.0 MHz The circuit architecture was fully bidirectional with a 9-bit magnitude pixel data bus and a 12-bit magnitude coefficient data bus programmed as input or output depending on the selection of Forward DCT or Inverse DCT function. DCT was so complex that to be implemented on a CPU required about 1 billion additions per second (1 giga Hertz operations per second) to process moving pictures in real time. CPUs and CMOS silicon process were unable to achieve such a very high frequency, equivalently implying the computing power of fifty Motorola 68020 [2.1], one of the faster micro-processors in the 1980s. The integrated circuit went officially in production at the end of 1987 and was a world premiere product. Since 1988, this component was successfully sold by SGS-Thomson after the merge. It implemented key patents [4.1][4.2] for rearranging, permuting or selecting data according to predetermined rules, for changing the order of data flow, e.g. matrix transposition, LIFO (last in first out) buffers; internally compensating and handling overflow or underflow  that was key for DCT and IDCT accurate and reliable implementations.
  
 
Innovative memory footprint reduction and distribution techniques were used since the hardware design did not use the traditional architecture (such as a unique multiplier that would read/write in a large memory), but a bit-serial pipelined architecture, where dynamic memory was distributed across the chip, to feed the 16 bit/serial dedicated multipliers that the chip had.
 
Innovative memory footprint reduction and distribution techniques were used since the hardware design did not use the traditional architecture (such as a unique multiplier that would read/write in a large memory), but a bit-serial pipelined architecture, where dynamic memory was distributed across the chip, to feed the 16 bit/serial dedicated multipliers that the chip had.
  
The STV3200 chip was considered so reliable the was used in the DISR (descent imager) [1.26] [2.9] [7.0] of Cassini-Huygens probe, therefore subject of cosmic rays interactions, that was launched in 1997 and has transmitted wonderful images of Saturn planet until 2017. All those historic images that are available in this website [2.10] have been processed through the STV3200 for 10 years through the solar system!
+
The STV3200 chip was considered very reliable at the point it was used in the DISR (descent imager) [1.26] [2.9] [7.0] of Cassini-Huygens probe, therefore resilient to cosmic rays; it was launched in 1997 and has transmitted wonderful images of Saturn planet until 2017. All those historic images that are available in this website [2.10] have been processed through the STV3200 for 20 years through the solar system!
  
 
Design of the STV3200 happened during the December 1986 to August 1987 period (product sampling around October 1987) as mentioned at the end of [1.25]. As said commercialization of the STV3200 started in 1988. It must be noted that on November 1988 there was the publication of the H261 standard (1st standard using DCT for real time video coding) while on May 1988 there was the 1st meeting of the MPEG standard (that was published in 1993) using block based DCT. This chip was considered as proof of implementation by MPEG ad-hoc group
 
Design of the STV3200 happened during the December 1986 to August 1987 period (product sampling around October 1987) as mentioned at the end of [1.25]. As said commercialization of the STV3200 started in 1988. It must be noted that on November 1988 there was the publication of the H261 standard (1st standard using DCT for real time video coding) while on May 1988 there was the 1st meeting of the MPEG standard (that was published in 1993) using block based DCT. This chip was considered as proof of implementation by MPEG ad-hoc group
Line 87: Line 87:
 
4) Super integration of those fundamental functions with others fundamental building blocks (e.g. scalar quantization and de-quantization, entropy coding and decoding as later needed by MPEG standard, motion compensation, unified dynamic memory management), was the next essential step required to manufacture reliable cheap system solutions for mass production and broad adoption while before they were consisting of few non integrated chips. In this respect [1.20] presents the deployment of STV3200 and STi3220 in a complete video codec hardware architecture for high pixel rates addressing standard (SD) and high definition (HD) picture formats.
 
4) Super integration of those fundamental functions with others fundamental building blocks (e.g. scalar quantization and de-quantization, entropy coding and decoding as later needed by MPEG standard, motion compensation, unified dynamic memory management), was the next essential step required to manufacture reliable cheap system solutions for mass production and broad adoption while before they were consisting of few non integrated chips. In this respect [1.20] presents the deployment of STV3200 and STi3220 in a complete video codec hardware architecture for high pixel rates addressing standard (SD) and high definition (HD) picture formats.
  
5) STi3240 [1.18] was the 1st MPEG1/H261 chip with STV3208 (DCT processor) [1.19] supporting video decoding schemes up to 10 Mbit/s of the ISO/MPEG future (at that time under active definition) video standard and CCITT/H261 recommendation at video rate (352x288 pixels at 30fps). It required between 512 K Bytes up to 4Mbytes of external costly power hungry DRAM with a 8/16 bits microprocessor interface. External DRAM was a scarce and costly resource, dramatically impacting the overall power consumption by order of magnitude and bill of material. Therefore, the use of on the fly/real time memory reduction/compression techniques to support high chip calculations throughput and minimized data bandwidth between the chip and the memory were conceived and applied by ST for the first time and implemented. The fundamental idea behind the introduction of these techniques was to let MPEG compression noise to hide the one implied by the on the fly lossy compression generated by the memory reduction. countless tests at different bit rates (1.5, 4, 8, 16 mbit/s) confirmed the goodness of the idea as well as the deployment into multimedia systems sold for many years.
+
5) STi3240 [1.18] was the 1st MPEG1/H261 chip with STV3208 (DCT processor) [1.19] supporting video decoding schemes up to 10 Mbit/s of the ISO/MPEG future (at that time under active definition) video standard and CCITT/H261 recommendation at video rate (352x288 pixels at 30fps). It required between 512 K Bytes up to 4Mbytes of external costly power hungry DRAM with a 8/16 bits microprocessor interface. External DRAM was a scarce and costly resource, dramatically impacting the overall power consumption by order of magnitude and bill of material. Therefore, the use of on the fly, real time memory reduction based on compression techniques to support high chip calculations throughput and minimized data bandwidth between the chip and the memory were conceived and applied within the company for the first time and implemented. The fundamental idea behind the introduction of these techniques was to let MPEG compression noise to hide the one implied by the on the fly lossy compression generated by the memory reduction. Numerous tests at different bit rates (1.5, 4, 8, 16 mbit/s) done in cooperation with National Council for Researches confirmed the applicability of the original idea as well as the low complex deployment into multimedia systems sold for many years.
 
   
 
   
 
[[image:Figure4 decoder figure.png|thumb|Figure 4 decoder system, STV3208 was DCT co processor]]
 
[[image:Figure4 decoder figure.png|thumb|Figure 4 decoder system, STV3208 was DCT co processor]]
  
6) Then the system on chip super integration era started with active developments. Earliest members of the STi digital multimedia chip family for mpeg moving picture processing were: STi3400, STi3500, STi3520A. Other early examples were STi7000, STi5000 (and the very many others that followed for 30 years of developments and market adoption). They were the first able to offer a dramatic reduction in silicon and package size, power consumption, memory usage with higher level of transistors integration, functionalities, and lowest cost of equipment’s. Such equipment’s were made previously by many discrete signal processors (DSP), making the system architecture very redundant, and which were placed into too many more boards. They were far from being viable from a mass manufacture point of view and cost affordable for MPEG-1 and MPEG-2 standards deployment at end user side.  STi chips were very instrumental to accelerate the introduction to the end users of CD-ROM, DVDs, Set Top Box and digital TVs home systems that became ubiquitously adopted throughout the digital entertainment world and in everyone home (and mobile phones) across 3 decades all over the world.
+
6) Then the system on chip super integration era started with active developments. Earliest members of the digital multimedia chip family for mpeg moving picture processing were: STi3400, STi3500, STi3520A. Other early examples were STi7000, STi5000 (and the very many others that followed for 30 years of developments and market adoption). They were the first able to offer a dramatic reduction in silicon and package size, power consumption, memory usage with higher level of transistors integration, functionalities, and lowest cost of equipment’s. Such equipment’s were made previously by many discrete signal processors (DSP), making the system architecture very redundant, and which were placed into too many more boards. They were far from being viable from a mass manufacture point of view and cost affordable for MPEG-1 and MPEG-2 standards deployment at end user side.  Those chips were very instrumental to accelerate the introduction to the end users of CD-ROM, DVDs, Set Top Box and digital TVs home systems that became ubiquitously adopted throughout the digital entertainment world and in everyone home (and mobile phones) across 3 decades all over the world.
  
 
7) Inside those chips’, innovative and advanced reduction techniques of external memory bandwidth were conceived and implemented first. They were key to lower power consumptions to and under 1W. In that respect several key methods are described in the following subsections:
 
7) Inside those chips’, innovative and advanced reduction techniques of external memory bandwidth were conceived and implemented first. They were key to lower power consumptions to and under 1W. In that respect several key methods are described in the following subsections:
Line 99: Line 99:
 
b) compressed picture decoding on the fly [4.4] was a break-through technology to decompress the mpeg bitstream and motion compensate it at the same time while displaying MPEG B-pictures. This to avoid inefficient and further storage of them in the external frame buffer, thus resulting in a dramatic reduction of the external memory to only 16 Mbits for SD definition [1.20] within the memory bandwidth budget
 
b) compressed picture decoding on the fly [4.4] was a break-through technology to decompress the mpeg bitstream and motion compensate it at the same time while displaying MPEG B-pictures. This to avoid inefficient and further storage of them in the external frame buffer, thus resulting in a dramatic reduction of the external memory to only 16 Mbits for SD definition [1.20] within the memory bandwidth budget
  
c) frame buffer lossy compression [1.11] [4.5] was another key technology used to co-decompress on the fly I , P and B MPEG pictures by halving the associated external DRAM memory and the memory bandwidth without annoying picture artefacts. As before introduced, in STi solutions the memory reduction quantization noise was cleverly masked within the MPEG coding noise, resulting in not human perceivable and visible loss of quality. This allowed to reach 8 Mbits DRAM footprint in SD definition enabling super integration of 1 bit transistor per cell DRAM on the same chip, in a more competitive way than integrating 6 transistor per cell SRAM. therefore STi was the 1st to not require any additional external memory for the MPEG SD video decompression task. Moreover another kind of memory reduction was applied to HDTV decoders with even further benefits. [1.23] was novel method to reduce the external memory needed by STi MPEG-2 HDTV decoder architecture. The total amount of memory was reduced from 128 to 32 Mbits preserving very good picture quality. Furthermore, it required low hardware complexity increases of less than 5%, in 0.35 um ST silicon technology, the total decoder silicon area with respect to the standard decoder implementation.
+
c) frame buffer lossy compression [1.11] [4.5] was another key technology used to co-decompress on the fly I , P and B MPEG pictures by halving the associated external DRAM memory and the memory bandwidth without annoying picture artefacts. As before introduced, in the chip family the quantization noise implied by the memory reduction was cleverly masked under the MPEG coding noise, resulting in not human perceivable and visible loss of quality. This allowed to reach 8 Mbits DRAM footprint in SD definition enabling super integration of 1 bit transistor per cell DRAM on the same chip, in a more competitive way than integrating 6 transistor per cell SRAM. therefore those chips were the 1st to not require any additional external memory for the MPEG SD video decompression task. Moreover another kind of memory reduction was applied to HDTV decoders with even greater memory benefits. [1.23] was novel method to reduce the external memory needed by MPEG-2 HDTV decoder architecture. The total amount of memory was reduced from 128 to 32 Mbits preserving very good picture quality. Furthermore, it required low hardware complexity increases of less than 5%, in 0.35 um CMOS silicon technology, the total decoder silicon area with respect to the standard decoder implementation.
  
8) The STi family implementation of the aforementioned ST comprehensive vision was very instrumental to create the digital Multimedia domain. That was made possible with those and many other chips we are not mentioning for space reason but that spanned across 30 years for ST engineering efforts.  
+
8) The chip family implementation of the aforementioned vision was very instrumental to create the digital Multimedia domain. That was made possible with those and many other chips we are not mentioning for space reason but that spanned across 30 years for company engineering efforts.  
  
It's important to underline that the pioneering development work began since 1986 [1.24] [1.26] among different and key French partners as proof of the passion, motivation, technical vision permeating the France R&D context:
+
It's important to detail that the pioneering development work began since 1986 [1.24] [1.26] among different and key French partners as proof of the passion, motivation, technical vision permeating the France R&D context:
  
1) Thomson Semiconductor (merged later into SGS-THOMSON, now STMicroelectronics) launched innovative circuit architecture studies for integrated for the Discrete Cosine Transform since 1986
+
1) Thomson Semiconductor launched innovative circuit architecture studies for integrated for the Discrete Cosine Transform since 1986
  
 
2) ENST Paris (now Telecon ParisTech) was very active since 1986 in integrated circuit architectures for DCT [1.24] [1.25] single chip for video rates design.
 
2) ENST Paris (now Telecon ParisTech) was very active since 1986 in integrated circuit architectures for DCT [1.24] [1.25] single chip for video rates design.
Line 113: Line 113:
 
Under the finance support of the DAII (that time French State Agency) these three laboratories collaborated to form a formidable team responsible for realizing the first DCT component (STV3200) and the Motion Estimation chip (STi3220).  The DCT and Motion estimation chips existed thanks to the strong collaboration between SGS-Thomson and the public research lab of ENST.
 
Under the finance support of the DAII (that time French State Agency) these three laboratories collaborated to form a formidable team responsible for realizing the first DCT component (STV3200) and the Motion Estimation chip (STi3220).  The DCT and Motion estimation chips existed thanks to the strong collaboration between SGS-Thomson and the public research lab of ENST.
  
Thus, on 1988, SGS-THOMSON finalized these seminal developments into production (with background developments since 1986 [1.24] [1.26] in Thomson Semiconductor) therefore creating the wave of key integrated circuits and SoC for MPEG deployment. By consequence it mastered at industrial level initially the 2 key components which made possible and economically viable the realization of more complex, reliable, ready for mass production, yet low cost and low power MPEG-1 and MPEG-2 encoders and decoder systems for the (mid 80s) for the not yet existing digital multimedia television markets.  
+
Thus, on 1988, SGS-Thomson finalized these seminal developments into production (with background developments since 1986 [1.24] [1.26] in Thomson Semiconductor) therefore starting the wave of key integrated circuits and SoC for MPEG deployment. By consequence it mastered at industrial level initially the 2 key components which made possible and economically viable the realization of more complex, reliable, ready for mass production, yet low cost and low power MPEG-1 and MPEG-2 encoders and decoder systems for the (mid 80s) for the not yet existing digital multimedia television markets.  
  
9) Apple Computer was looking for a technology partner in the field of image compression. Since aware of above-mentioned developments, contacted SGS-THOMSON in 1988. The 68020 microprocessors was used from March 1987 to March 1992 and only used twice, once in the Macintosh II as a high-end processor, and then again with the Macintosh LC, as a low-end processor. However, it was too slow and inadequate to support real time video decoding functions as for MPEG decoding at interactive frames. By recognizing this key technical limit, the collaboration with SGS-Thomson was started, the first in its kind, to study and develop highly integrated set of components to provide video functions for Apple's microcomputers. That was a very advanced concept for that time.  
+
9) In 1988, SGS-Thomson, aware of his advanced technical development for future MPEG SoC, contacted Apple Computer to collaborate with a reference system company in the field of image compression. The 68020 microprocessors was used from March 1987 to March 1992 and only used twice, once in the Macintosh II as a high-end processor, and then again with the Macintosh LC, as a low-end processor. However, it was too slow and inadequate to support real time video encoding functions as for MPEG decoding at interactive frames. By acknowledging this technical limit, the collaboration between the 2 companies started, the first in its kind, to study and develop highly integrated set of components to provide video functions for Apple's microcomputers. That was a very advanced concept for that time.  
Therefore SGS-THOMSON entered into a partnership with Apple ATG (Avanced Technology Group) headed by Al Alcorn Apple Fellow. Thus, based on SGS-THOMSON understanding of 68020 limits and the contribution of innovative chip solutions from SGS-THOMSON background work, the first architecture of an integrated system for image compression was defined to support the MPEG1 standard: the STi3240 [1.18], STi3400 [1.6]. STi3400 was a real time video, super integrated with DCT, decompression integrated processor supporting MPEG-1 and H.261 standards. The digital output was for PAL 50Hz and NTSC 60 Hz interlaced displays.
+
The partnership was with Apple ATG (Advanced Technology Group) headed by Al Alcorn Apple Fellow [2.11].
 +
The following is an extract from the Alcorn, Al (Allan) oral history Catalog Number 102658257 Computer History Museum, Mountain View, California.
 +
From [2.13] pag 49 Al answered to Lowood question : What was the state of MPEG at that time?
 +
"Alcorn: Academic. You had people from AT&T, Bell Labs, which was kind of industry academic, you had
 +
the French phone company involved. It was really academic but they were associated with some
 +
corporations. The French, SGS Thomson, we became partners with....".
 +
And in pag 50
 +
Alcorn: We built at Apple, that team under Mark Cutter, actually built the first real-time MPEG decoder
 +
ever in hardware. The state of technology at that time was you certainly couldn’t do this in software. It had
 +
to be dedicated hardware to decode or encode MPEG. So they actually used very complex DSP boards
 +
in a Mac -- we had to use a Mac because we’re at Apple -- but it took like three high-powered cards with a
 +
high-speed bus across the top. It was certainly not product development. It could never be a product that
 +
way. So that was the state. It really was something that could not be done under CPU software.
 +
Fascinating story of how this evolved. So two things we did: One was we got SGS Thomson interested in
 +
making this chip because they already had a couple of DCT [Discrete Cosine Transform] cores, parts of
 +
the company, and it was kind of a castaway little team of very bright engineers that didn’t fit the traditional
 +
mold of Thomson. They were using high level languages, VHDL [VHSIC hardware description language],
 +
to do chips, which was new. It was coming off of polygons on physical stuff. We actually worked with them
 +
cooperatively with no agreement, nothing, and co-worked with this to help them make a real MPEG
 +
decoder chip. They were driven by the idea that that hey, I’m Apple, we’re going to buy a million of these
 +
things. And ___________________________ Vergon (actual name is Guy Lauvergeon) was the guy at Thomson SGS to do this. I worked with him and they finally built the chip, got it to work. How many am I going to buy? I’ll buy two. Uh-oh, no, you’re kidding. No, I can’t buy any of these. I’m research. “Al, I’m going to get clobbered, I’m going to get
 +
fired. I did all this stuff and you promised. Oh, come on.” He was in trouble, but then there was this little
 +
project. Hughes Satellite had this idea of beaming digital signals to the home for movies and stuff. Guess
 +
who had the only chip that would do this decoded in the set? Now, he (Guy Lauvergeon) was a hero, but there were two months when he was in trouble, but he became a hero. So we co-opted them. What was the other one?
 +
Well, I’ll think of it." .
 +
Allan Alcorn testimony is considered very relevant to support this milestone nomination. His endorsement letter is included in the proposal as reference [7.8] 
  
10) Next leveraging its expertise, SGS-THOMSON started a background work with RCA, which ended in 1992 with the development of an MPEG2 video decoder, STi3500 [1.7] and subsequent STi3520 [1.8] enabling DirecTV broadcast. Those were other essential components at the heart of the revolutionary multimedia digital television services introduced firstly from the USA Digital Satellite DirectTV firm. SGS-THOMSON succeeded in delivering the first MPEG2 decoder in the world, much ahead of the Californian start-up C-CUBE. As such the STi3500 was the enabler of the first Digital TV broadcast ever, that was the key to the worldwide leadership of SGS-THOMSON in digital multimedia market. It must be noticed that C-Cube was funded on 1988 when SGS-THOMSON had already above-mentioned chips in production and when Thomson semiconductor started on 1986.  
+
Thus, based on the key understanding of 68020 limits and the contribution of innovative chip solutions from our company background work, the first architecture of an integrated system for image compression was defined to support the MPEG1 standard: the STi3240 [1.18], STi3400 [1.6]. STi3400 was a real time video, super integrated with DCT, decompression integrated processor supporting MPEG-1 and H.261 standards. The digital output was for PAL 50Hz and NTSC 60 Hz interlaced displays.
  
11) STi3500 was the real time video decompression integrated circuit supporting MPEG-1 and MPEG-2 standards at video rates up to 720x480x60Hz or 720x576x50Hz requiring minimal support from an external 8bit microcontroller (ST8) used only to initialize the chip, reducing the complexity of software and associated code footprint. It required only 1W of power consumption. The STi3500 was a success since achieved a turnover greater than $ 100M in less than 3 years as a sign of significant market adoption. In that chip STV3200 (which was super integrated) represented only 10% of the total silicon area. STi3520 on top of STi3500, used only 16Mbits memory and integrated audio decoder compliant with MPEG layers I and II supporting sampling rates of 32, 44.1 and 48 KHz.  STI3520A [1.20] was a single chip for MPEG MP@ML video decoding in only 16 Mbit RAM integrated with STi4500 MPEG L1 and L2 Audio decoder and integrating on screen display generator. The chip was using 0.5µm CMOS SGS-THOMSON technology
+
10) Next leveraging its expertise, the our company started a background work with RCA, which ended in 1992 with the development of an MPEG2 video decoder, STi3500 [1.7] and subsequent STi3520 [1.8] enabling DirecTV broadcast. Those were other essential components at the heart of the revolutionary multimedia digital television services introduced firstly from the USA Digital Satellite DirectTV firm. Our company succeeded in delivering the first MPEG2 decoder in the world, much ahead of the Californian start-up C-CUBE. As such the STi3500 was the enabler of the first Digital TV broadcast ever, that was the key to the worldwide leadership of SGS-THOMSON in digital multimedia market. It must be noticed that C-Cube was funded on 1988 when our company had already above-mentioned chips in production and since when Thomson semiconductor started on 1986.  
  
12) STi7000 [1.12] [2.2] was an integrated system for High-Definition Television (HDTV), which combined an MPEG-2 decoder with a display and format converter onto one chip in 1998. As already introduced in point 5) the most advanced trick was the frame buffer memory reduction through tiny on chip on the fly compression engine based on scalar adaptive quantization allowing full HDTV pictures to be decoded and displayed with only 64 Mbits of external memory instead of 128 Mbits. Designed for use in HDTV and other digital TV receivers, set top boxes and PCs, the STi7000 was developed in collaboration with Thomson Multimedia, a strategic partner with whom SGS-THOMSON shared a joint design center in Grenoble, France. The chip incorporated all the 18 video formats defined by the ATSC (Advanced Television Systems Committee) and Grand Alliance specifications. STi7000 supported video rates of up to 1920 x 1088 x 30Hz interlaced or 1280 x 720 x 60Hz progressive. Built in 0.35-micron SGS-Thomson HCMOS6 silicon technology, the STi7000 also included interfaces for a host microcontroller, local SDRAM, standard or high-definition video output and D1 digitized video input. Next versions STi7100 [1.13] STi7200 [1.14] also integrated graphics engine and power powerful micro controllers. Next generation STi7108 had dual ST40-300 CPU host processors linked to a 256K L2 cache giving up to 2000 DMIPS performance and a total of 4000 DMIPS. A 3D graphics engine [1.15] enabled advanced Internet content and high-performance gaming. It was the first set-top box IC [1.16] in the market to combine 3D OpenGL-ES 2.0, OpenVG vector graphics, Ethernet, USB and e-SATA interfaces to connect Internet devices, DVR storage or external Flash or hard-disk (HDD) drives.  
+
11) STi3500 [1.30] was the real time video decompression integrated circuit supporting MPEG-1 and MPEG-2 standards at video rates up to 720x480x60Hz or 720x576x50Hz requiring minimal support from an external 8bit microcontroller (ST8) used only to initialize the chip, reducing the complexity of software and associated code footprint. It required only 1W of power consumption.
 +
This chip was a first of its kind [7.9] and successfully developed. This was a major breakthrough for Thomson Multimedia - RCA business for video decoding and was critical in launching DirecTV business on time
 +
The STi3500 was a success since achieved a turnover greater than $ 100M in less than 3 years as a sign of significant market adoption. In that chip STV3200 (which was super integrated) represented only 10% of the total silicon area. STi3520 on top of STi3500, used only 16Mbits memory and integrated audio decoder compliant with MPEG layers I and II supporting sampling rates of 32, 44.1 and 48 KHz. STI3520A [1.20] was a single chip for MPEG MP@ML video decoding in only 16 Mbit RAM integrated with STi4500 MPEG L1 and L2 Audio decoder and integrating on screen display generator. The chip was using 0.5µm CMOS company technology
  
13) The STi5500 was the first member of back-end decoders for set-top boxes and DVD players. Compliant with MPEG-2 to decode its transport stream and convert it into sounds and images that sent to loudspeakers, monitors, TV screens and similar human interfaces. The STi5500 was the first integrated circuit in the world to incorporate not only all the circuitry required to handle all the back-end functions but also a 32-bit microprocessor. The built-in microprocessor had sufficient processing power to handle system-level functions as well as the managing the MPEG decoding function, so it eliminated the need for a separate system controller in most applications. Its super integration was the crucial factor and achieved immediate customers adoption. Since then, ST has developed even more powerful members including the STi5505, which is the first silicon chip in the world to integrate all the back-end functions of a DVD player.  
+
12) STi7000 [1.12] [2.2] was an integrated system for High-Definition Television (HDTV), which combined an MPEG-2 decoder with a display and format converter onto one chip in 1998. As already introduced in point 5) the most advanced trick was the frame buffer memory reduction through tiny on chip on the fly compression engine based on scalar adaptive quantization allowing full HDTV pictures to be decoded and displayed with only 32 or 64 Mbits of external memory instead of 128 Mbits. Designed for use in HDTV and other digital TV receivers, set top boxes and PCs, the STi7000 was developed in collaboration with Thomson Multimedia, a strategic partner with whom our company shared a joint design center in Grenoble, France. The chip incorporated all the 18 video formats defined by the ATSC (Advanced Television Systems Committee) and Grand Alliance specifications. STi7000 supported video rates of up to 1920 x 1088 x 30Hz interlaced or 1280 x 720 x 60Hz progressive. Built in 0.35-micron HCMOS6 silicon technology, the STi7000 also included interfaces for a host microcontroller, local SDRAM, standard or high-definition video output and D1 digitized video input. Next versions STi7100 [1.13] STi7200 [1.14] also integrated graphics engine and power powerful micro controllers. Next generation STi7108 had dual ST40-300 CPU host processors linked to a 256K L2 cache giving up to 2000 DMIPS performance and a total of 4000 DMIPS. A 3D graphics engine [1.15] enabled advanced Internet content and high-performance gaming. It was the first set-top box IC [1.16] in the market to combine 3D OpenGL-ES 2.0, OpenVG vector graphics, Ethernet, USB and e-SATA interfaces to connect Internet devices, DVR storage or external Flash or hard-disk (HDD) drives.
 +
 
 +
13) The STi5500 was the first member of back-end decoders for set-top boxes and DVD players. Compliant with MPEG-2 to decode its transport stream and convert it into sounds and images that sent to loudspeakers, monitors, TV screens and similar human interfaces. The STi5500 was the first integrated circuit in the world to incorporate not only all the circuitry required to handle all the back-end functions but also a 32-bit microprocessor. The built-in microprocessor had sufficient processing power to handle system-level functions as well as the managing the MPEG decoding function, so it eliminated the need for a separate system controller in most applications. Its super integration was the crucial factor and achieved immediate customers adoption. Since then, ST has developed other more powerful members including the STi5505, which is the first silicon chip in the world to integrate all the back-end functions of a DVD player.  
 
As demonstration of the innovation it brought in the field of digital multimedia entertainment , STi5500 won [5.1] from a field of nearly 300 nominations, the prestigious European IT PrizesPrize awarded by the European Commission in conjunction with Euro-CASE, the European Council of Applied Science and Engineering, for the development of products that are expected to play fundamental roles in helping European industry to increase its share of world markets.
 
As demonstration of the innovation it brought in the field of digital multimedia entertainment , STi5500 won [5.1] from a field of nearly 300 nominations, the prestigious European IT PrizesPrize awarded by the European Commission in conjunction with Euro-CASE, the European Council of Applied Science and Engineering, for the development of products that are expected to play fundamental roles in helping European industry to increase its share of world markets.
  
Summarizing SGS-THOMSON STi gained its leadership position in the field of digital consumer multimedia decoder-type products for digital satellite TV and DVD players. Early members such as STV3200, STi3220, STi3240, STi3400, STi3500, STi3520A, STi7000, STi7100, STi7108, STi5500 SoC through the 80s, 90s and 2000 years continued for the development of digital multimedia applications at heart of the worldwide digital multimedia convergence of the TV services for everyday users, the world of telecommunications and the world of the PC.  
+
Summarizing the MPEG family of integrated circuits gained its leadership position in the field of digital consumer multimedia decoder-type products for digital satellite TV and DVD players. Early members such as STV3200, STi3220, STi3240, STi3400, STi3500, STi3520A, STi7000, STi7100, STi7108, STi5500 SoC through the 80s, 90s and 2000 years continued for the development of digital multimedia applications at heart of the worldwide digital multimedia convergence of the TV services for everyday users, the world of telecommunications and the world of the PC.  
  
 
Many other generations were manufactured across the years as shown in the next figure 5 even supporting complex operative systems such as Linux.
 
Many other generations were manufactured across the years as shown in the next figure 5 even supporting complex operative systems such as Linux.
Line 137: Line 166:
 
[[image:Figure6 chip integration.jpg|thumb|Figure 6 Super integration of DCT with other multimedia functions]]
 
[[image:Figure6 chip integration.jpg|thumb|Figure 6 Super integration of DCT with other multimedia functions]]
  
STMicroelectronics has sold over 500,000,000 (cumulated) of those integrated systems worldwide. This figure undoubtedly demonstrates the pervasivity of STi family through the digital multimedia entertainment world. Figure 7 shows cumulative shipments of ST products per year basis to further support the statement.
+
ST has sold over 500,000,000 (cumulated) of those integrated systems worldwide. This figure undoubtedly demonstrates the pervasivity of the MPEG chip family through the digital multimedia entertainment world. Figure 7 shows cumulative shipments of company products per year basis to further support the statement.
 
   
 
   
 
[[image:Figure7 cumulative shipments.png|thumb|Figure 7]]
 
[[image:Figure7 cumulative shipments.png|thumb|Figure 7]]
  
14) As proof of the high relevance of the proposed milestone a number of letter of supports were provided by key and well know experts in the field
+
14) As proof of the high relevance of the proposed milestone a number of letter of supports were provided by key, respectful and well know experts in the field
  
a) A letter of endorsement [7.0] to ST milestone is provided by Nicolas Demassieux, today Senior Vice President of Research, Orange [2.3], and in 80s assistant professor at ENST (now Telecom ParisTech) working on very optimized VLSI implementations for signal processing, included the DCT processor.
+
a) [7.0] by Nicolas Demassieux, today Senior Vice President of Research, Orange [2.3], and in 80s assistant professor at ENST (now Telecom ParisTech) working on very optimized VLSI implementations for signal processing, included the DCT processor.
  
b) A letter of endorsement [7.1] to ST milestone is provided by Leonardo Chiariglione [2.3], the undisputed father of MPEG ISO/IEC SC29 WG11 and the tireless driving force of MPEG standards for digitized video. Leonardo is unanimously considered at world wide level a genius since created a new way of enjoying music, with Leonardo Chiariglione’ s MP3 standard.
+
b) [7.1] by Leonardo Chiariglione [2.3], the undisputed father of MPEG ISO/IEC SC29 WG11 and the tireless driving force of MPEG standards for digitized video. Leonardo is unanimously considered at world wide level a genius since created a new way of enjoying music, with Leonardo Chiariglione’ s MP3 standard.
 
As mentioned in the public article [6.1] with Leonardo Chiariglione, SGS-­Thomson has been a big MPEG driver since it was the world's leading producer of MPEG-related ICs and supplied close to 70 percent of the world's MPEG-2 decoder chips. That was the result of the audiovisual vision that was shared by SGS-Thomson, Thomson Multimedia SA, Paris, and France's government. Also, Chiariglione noted, Thomson's U.S. subsidiary RCA was in the desperate need of MPEG chips for a huge order of 1st generation satellite TV decoders--a million set-top boxes for the Hughes-RCA DirecTV newborn system--and SGS-Thomson Microelectronics got the contract, as proof of its leading integrated circuits technology started since 1986.
 
As mentioned in the public article [6.1] with Leonardo Chiariglione, SGS-­Thomson has been a big MPEG driver since it was the world's leading producer of MPEG-related ICs and supplied close to 70 percent of the world's MPEG-2 decoder chips. That was the result of the audiovisual vision that was shared by SGS-Thomson, Thomson Multimedia SA, Paris, and France's government. Also, Chiariglione noted, Thomson's U.S. subsidiary RCA was in the desperate need of MPEG chips for a huge order of 1st generation satellite TV decoders--a million set-top boxes for the Hughes-RCA DirecTV newborn system--and SGS-Thomson Microelectronics got the contract, as proof of its leading integrated circuits technology started since 1986.
 +
He is the author of the book "Even the stars die: The history of MPEG and how it made digital media happen" about the story of how the Moving Picture Experts Group (MPEG) was conceived and established. It's also available [2.12]. In summary, the book tells the story of how MPEG engineered the transformation of analogue media, how it fostered the development of digital media, how it was terminated and how its spirit continues in the field of data coding by Artificial Intelligence through MPAI.
 +
 +
c) [7.2] by Professor emeritus Fabio Rocca. He on 1969 was the pioneer on motion estimation and compensation technologies fundamental technologies being adopted by MPEG standards 20 years later.
 +
 +
d) [7.3] by Hisafumi Yamada, past Sony USA TV CTO that prove the high innovation STi HDTV chips provided to ATV USA television.
 +
 +
e) [7.4] by Sylvain Kritter, currently Product Director VIZYON with proven expertise on HW design. During mid 80s he was a IC design manager working on very optimized hardware implementations of discrete cosine transform functions as witnessed by his IEEE papers and patents mentioned in the letter . He was using Thomson
 +
1.25 µm CMOS technology at that time
  
c) A letter of endorsement [7.2] to ST milestone is provided by prof Fabio Rocca. Prof Fabio Rocca on 1969 was the pioneer on motion estimation and compensation technologies fundamental technologies being adopted by MPEG standards 20 years later.
+
f) [7.5] by Professor Emeritus Virginio Cantoni, University of Pavia. Also Fellow of the IAPR since 1994 and Life Fellow of the IEEE (he was Fellow since 1997). In 2006 the President of the Italian Republic conferred to Professor Virginio Cantoni the title of ‘Commander of the Order of Merit of the Italian Republic’. Professor Virginio Cantoni is an expert of VLSI image processors exploiting reduced memory architectures
  
d) A letter of endorsement [7.3] to ST milestone is provided by Hisafumi Yamada, past Sony USA TV CTO that prove the high innovation STi HDTV chips provided to ATV USA television.
+
g) [7.6] by Professor Emeritus Mariagiovanna Sami, Politecnico di Miano. Past Full Professor, Digital Systems. IEEE Life Member. Minerva prize for woman scientists, the Seymour Cray prize for contributions in the area of parallel processing, the Herbert A. Simon Gold Medal, assigned by the Society for Design and Process Science.  Cavaliere della Repubblica Italiana (knight of the Italian republic). Member of the Italian National Science Academy (“Dei Quaranta”). Expert of high-performance, fault-tolerant array architectures for digital signal and image processing.
  
e) A letter of endorsement [7.4] to ST milestone is provided by Sylvain Kritter, Product Director VIZYON with proven expertize on HW design.
+
h) [7.7] by Professor Andrea Basso who has been active in ISO/IEC SC19 WG11 working group also known as MPEG for almost 25 years. Since 1990 while in EPFL in the group of Murat Kunt and then in AT&T Bell labs Holmdel NJ and AT&T Research Laboratories, Middletown, NJ, USA in the group of Barry Haskell, Professor Basso directly contributed the development of several MPEG standards starting from MPEG1 and MPEG2 till the most recent HEVC standard.
  
f) A letter of endorsement [7.5] to ST milestone is provided by Professor Emeritus Virginio Cantoni, University of Pavia. Also Fellow of the IAPR since 1994 and Life Fellow of the IEEE (he was Fellow since 1997). In 2006 the President of the Italian Republic conferred to Professor Virginio Cantoni the title of ‘Commander of the Order of Merit of the Italian Republic’. Professor Virginio Cantoni is an exper of VLSI image processors exploiting reduced memory architectures
+
i) [7.8] by Allan Alcorn Apple Fellow in the late 80s in the Advanced Technology Group at Apple Computer, Cupertino USA
  
g) letter [7.6] of endorsement by Professor Emeritus Mariagiovanna Sami, Politecnico di Miano. Past Full Professor, Digital Systems. IEEE Life Member. Minerva prize for woman scientists, the Seymour Cray prize for contributions in the area of parallel processing, the Herbert A. Simon Gold Medal, assigned by the Society for Design and Process Science.  Cavaliere della Repubblica Italiana (knight of the Italian republic). Member of the Italian National Science Academy (“Dei Quaranta”). Expert of high-performance, fault-tolerant array architectures for digital signal and image processing.
+
l) [7.9] Endorsement by Jeff Cooper, former  Senior R&D Engineer at Thomson Multimedia - RCA in the 90s responsible for DirectTV MPEG2 set top box design based on STi3500 that was 1st MPEG2 decoder critical for launching DirecTV service.  
  
g) letter of endorsement by Professor Andrea Basso who has been active in ISO/IEC SC19 WG11 working group also known as MPEG for almost 25 years. Since 1990 while in EPFL in the group of Murat Kunt and then in AT&T Bell labs Holmdel NJ and AT&T Research Laboratories, Middletown, NJ, USA in the group of Barry Haskell, Professor Basso directly contributed the development of several MPEG standards starting from MPEG1 and MPEG2 till the most recent HEVC standard.
+
m) [7.10] Endorsement by Shigenobu Minami, former  R&D team leader at Toshiba for MPEG4 standardization and LSI planning in the 90s.
  
15) In the following years STi products on MPEG where the key enabling factor of a huge digital consumer ecosystem composed by many operators, equipment manufacturers, software developers and conditional access companies as shown in the next figure 6
+
15) In the following years the MPEG products were the key enabling factor of a rich value chain digital consumer ecosystem composed by many operators, equipment manufacturers, software developers and conditional access companies as shown in the next figure 6
 
   
 
   
 
[[image:Figure8 ecosystem.png|thumb|Figure 8 Worlds wide level ecosystem for digital multimedia services created thanks to STi family]]
 
[[image:Figure8 ecosystem.png|thumb|Figure 8 Worlds wide level ecosystem for digital multimedia services created thanks to STi family]]
Line 170: Line 207:
 
18) as written into a public article STMicroelectronics NV History [2.6] quoting “ ST is the world's leading manufacturer of analog ICs (integrated circuits) and MPEG-2 decoder chips, used to provide video decompression for DVD players and digital television set-top boxes.” and “Another area in which SGS Thomson became an early player was its development of MPEG decompression chips. The company's dedication to MPEG technology was risky, as a number of other digital video decompression schemes were competing for what promised to be a huge market in the future. SGS Thomson gained an early lead in the MPEG market, however, releasing its first Motion Estimation Processor in 1990. By 1993, the company had debuted its "multimedia" chip, capable of decompressing digital video files for display on a television set. This chip helped the company take a major position in the new set-top box market, starting with supplying the chip for the Hughes digital satellite television set-top box.” It helps to also state that ST was dominating the MPEG domain winning over the risks due to its early investments.
 
18) as written into a public article STMicroelectronics NV History [2.6] quoting “ ST is the world's leading manufacturer of analog ICs (integrated circuits) and MPEG-2 decoder chips, used to provide video decompression for DVD players and digital television set-top boxes.” and “Another area in which SGS Thomson became an early player was its development of MPEG decompression chips. The company's dedication to MPEG technology was risky, as a number of other digital video decompression schemes were competing for what promised to be a huge market in the future. SGS Thomson gained an early lead in the MPEG market, however, releasing its first Motion Estimation Processor in 1990. By 1993, the company had debuted its "multimedia" chip, capable of decompressing digital video files for display on a television set. This chip helped the company take a major position in the new set-top box market, starting with supplying the chip for the Hughes digital satellite television set-top box.” It helps to also state that ST was dominating the MPEG domain winning over the risks due to its early investments.
  
19) SGS-THOMSON was also not only pioneer of MPEG2 in satellite digital multimedia business with DirecTV but also in Cable business with Scientific Atlanta (which was acquired by Cisco later). Thanks to the collaboration the STi chips named 5600 and 5610 were the first chips to integrate graphics engine with multimedia processors [1.21], [1.22].
+
19) ST was also not only pioneer of MPEG2 in satellite digital multimedia business with DirecTV but also in Cable business with Scientific Atlanta (which was acquired by Cisco later). Thanks to the collaboration the MPEG chips named 5600 and 5610 were the first chips to integrate graphics engine with multimedia processors [1.21], [1.22].
|a6=One key initial obstacle was represented by the 2.0 and 1.2 um CMSO silicon technology that prevented to immediately super integrate in a single chip many other hardwired and micro controller bocks. Therefore, hardware designers had to carefully decide what to accelerate with respect to a software implementation, for example on 68020 micro controllers used in Apple computers or on ST8 micro controller. therefore as result of an accurate design, STV3200 needed only 115,000 transistors because of a deep hardware optimized design involving multiple technical dimensions such as.  
+
|a6=One key initial obstacle was represented by the 2.0 and 1.25 um CMOS silicon technology that prevented to super integrate in a single chip many complex hardwired and micro controllers blocks. Therefore, hardware designers had to carefully decide what to accelerate with respect to what to assign to a software implementation, for example on 68020 micro controllers used in Apple computers or on ST8 micro controller. Therefore as result of an accurate design, STV3200 needed only 115,000 transistors because of the hardware optimized design involving multiple technical dimensions such as.  
  
 
1) data precision of internal calculations, as minimized between 8, 12 to 16 bits  
 
1) data precision of internal calculations, as minimized between 8, 12 to 16 bits  
Line 179: Line 216:
 
3) internal memory distribution, transposition [4.1][4.2] of read and write memory data access to avoid using costly additional SRAM blocks.  
 
3) internal memory distribution, transposition [4.1][4.2] of read and write memory data access to avoid using costly additional SRAM blocks.  
  
These dimensions were put in the chip despite the need to support various block formats such as 16x16, 8x8, 4x4, 16x8, 8x4, 8x16, 4x8 that will be specified many years later by MPEG-1,2, 4, H.264 and HEVC video standards. DCT coefficients were minimized to use only 12bits reducing furthermore internal memory footprint [4.6]. Internal data have been processed at 16 bits to minimize loss of precision instead of using 32 bits integer arithmetic. floating points was not considered as too transistor and silicon area demanding. Integrated memory was only 4Kbits and be transposable in the usage [4.1] [4.2] in real time by the internal controller avoiding unnecessary duplications and pipelining. Also, DCT mathematical separability to 1D was exploited to save silicon resources [4.7], [4.8], [4.9].
+
These dimensions were put in the chip despite the need to support various block formats such as 16x16, 8x8, 4x4, 16x8, 8x4, 8x16, 4x8 that will be specified many years later by MPEG-1, 2, 4, H.264 and HEVC video standards. DCT coefficients were minimized to use only 12bits reducing furthermore internal memory footprint [4.6]. Internal data have been processed at 16 bits to minimize loss of precision instead of using 32 bits integer arithmetic. floating points was not considered as too transistor and silicon area demanding. Integrated memory was only 4Kbits and be transposable in the usage [4.1] [4.2] in real time by the internal controller avoiding unnecessary duplications and pipelining. Also, DCT mathematical separability to 1D was exploited to save silicon resources [4.7], [4.8], [4.9].
  
 
Similar optimizations were applied to STi3220 motion estimator based on block matching between block of pixels; by limiting the search window to 256 positions the silicon complexity of the block matcher was reduced, shifting the random-access memory bottleneck into proper pixel burst access to avoid the costly penalties of opening and closing memory pages (to access frame buffers) at pixel level with associated deadly loss of efficiency of the processor. Further innovative algorithms and hw design to achieve independence from the block based search windows estimation, therefore reducing computational complexity and be able to accurately estimate real life motion [1.27] [1.28]. Cache based motion estimation was introduced due to the recursive algorithms we invented and was designed to avoid block based data fetching with highly on chip data reusability therefore reducing memory bandwidth. [4.12] [4.13]
 
Similar optimizations were applied to STi3220 motion estimator based on block matching between block of pixels; by limiting the search window to 256 positions the silicon complexity of the block matcher was reduced, shifting the random-access memory bottleneck into proper pixel burst access to avoid the costly penalties of opening and closing memory pages (to access frame buffers) at pixel level with associated deadly loss of efficiency of the processor. Further innovative algorithms and hw design to achieve independence from the block based search windows estimation, therefore reducing computational complexity and be able to accurately estimate real life motion [1.27] [1.28]. Cache based motion estimation was introduced due to the recursive algorithms we invented and was designed to avoid block based data fetching with highly on chip data reusability therefore reducing memory bandwidth. [4.12] [4.13]
Line 187: Line 224:
 
1) By hardwiring DCT and motion estimation/compensation processing functions, the most silicon area demanding function, hundreds of powerful micro controllers (such as Motorola 68020 or equivalent) were not anymore needed, while only an external 8bit controller using only 8Kbytes of ROM was capable to initiate decoding and searching operations once for any picture that was under decompression without being on the decoding critical path nor performing  pixel level operations. That removed the need for any time critical handshake between the external micro controller and the integrated circuits, avoiding hardware pipeline stalls that could result in picture freeze or annoying block artefacts affecting picture quality.
 
1) By hardwiring DCT and motion estimation/compensation processing functions, the most silicon area demanding function, hundreds of powerful micro controllers (such as Motorola 68020 or equivalent) were not anymore needed, while only an external 8bit controller using only 8Kbytes of ROM was capable to initiate decoding and searching operations once for any picture that was under decompression without being on the decoding critical path nor performing  pixel level operations. That removed the need for any time critical handshake between the external micro controller and the integrated circuits, avoiding hardware pipeline stalls that could result in picture freeze or annoying block artefacts affecting picture quality.
  
Another challenge, as the silicon technology advanced for STi3240, STi3400, STi3500, STi3520, STi7000, STi5500 and successive derivative chips, was the memory bottleneck that became even more urgent to address due to the unified addressing space used by different accelerators. DRAM was costly, power hungry and severely limited in storage space and bandwidth which in turn prevented the super integration since required to store I, P, B pictures and MPEG compressed bitstream. Decoding PAL and NTSC resolutions into only 16 Mbits of memory, to keep low the costs, implied several tricks to be implemented such as MPEG-2 B pictures decoding on the fly [4.4] and frame buffer on the fly compression [4.3] [4.5] [1.11] applied to both standard and high-definition decoders across the STi family. Novel memory reductions technique fixed the obstacle allowing MPEG SD decoding in only 8Mbits and HTDV in 32Mbits one of the most impressive achievement at that time. The memory bottleneck was solved. In figure 9 the picture of the first consumer high definition MPEG video chip. The Motion compensation unit with memory reduction is the block on the bottom right corner, with a raw of memory at the bottom and two almost parallel rows on the top. These are mostly due to memory related to data management for decompression  
+
Another challenge, as the silicon technology advanced for STi3240, STi3400, STi3500, STi3520, STi7000, STi5500 and successive derivative chips, was the memory bottleneck that became even more urgent to address due to the unified memory addressing space used by different accelerators. DRAM was costly, power hungry and severely limited in storage space and bandwidth which in turn prevented the super integration since required to store full resolution  I, P, B pictures and MPEG compressed bitstream. Decoding PAL and NTSC resolutions into only 16 Mbits of memory, to keep low the system costs, implied several tricks to be implemented such as MPEG-2 B pictures decoding on the fly [4.4] and frame buffer on the fly compression [4.3] [4.5] [1.11] applied to both standard and high-definition decoders across the chip family. Novel memory reductions technique fixed the obstacle allowing MPEG SD decoding in only 8Mbits and HTDV in 32Mbits one of the most impressive achievement at that time. The memory bottleneck was solved. In figure 9 the picture of the first consumer high definition MPEG video chip. The Motion compensation unit with memory reduction is the block on the bottom right corner, with a raw of memory at the bottom and two almost parallel rows on the top. These are mostly due to memory related to data management for decompression  
  
 
[[image:Figure12hdmpeg.GIF|thumb|Figure 9 STi7000 layout with memory reduction integrated]]
 
[[image:Figure12hdmpeg.GIF|thumb|Figure 9 STi7000 layout with memory reduction integrated]]
  
The energy consumption obstacle needed to be addressed to avoid power hungry systems. SoC optimizations allowed to keep the power consumption challenge between 0.5 and 1 W because higher power consumption would have reduced dramatically the reliability of the chips which were not using any heat sink other than a cheap plastic package.  This allowed to increase speed and functionality up to full High-Definition pictures within a single chip, also addressing the engine orchestrating obstacle due the on-chip CPU, video, audio and graphics concurrent executions and later on, super integrating also the micro controller (8 to 32bits), which resulted into dramatically minimized costs and greatly improved performances, as compared with competition multi-chip implementations.
+
The energy consumption obstacle needed to be addressed to avoid power hungry systems. SoC optimizations allowed to keep the power consumption challenge between 0.5 and 1 W because higher power consumption would have reduced dramatically the reliability of the chips and the costs. Chips were not using any heat sink other than a cheap plastic package.  This allowed to increase speed and functionality up to full High-Definition pictures within a single chip, also addressing the engine orchestration challenge due the on-chip CPU, video, audio and graphics concurrent executions and later on, super integrating also the micro controller (8 to 32bits), which resulted into dramatically minimized bill of material and greatly improved performances, as compared with competition multi-chip implementations.
  
The STi family initially consisted of: STV3200 (DCT), STV3220 (Motion Estimator), STi3240, STi3400 (MPEG 1 video decoder), STi3500 (MPEG2 Video decoder), STi3520 (MPEG 2 audio and video decoder) , STi5600, STi5610 (MPEG 2 audio/video decoder with integrated powerful 2D graphics) and STi7000 (HD MPEG 2 video decoder and HD Display processor), STi5000 (integrating micro controllers) and many many others which we are not listing here for sake of brevity since spanning 30 years of developments.
+
The MPEG family initially consisted of: STV3200 (DCT), STV3220 (Motion Estimator), STi3240, STi3400 (MPEG 1 video decoder), STi3500 (MPEG2 Video decoder), STi3520 (MPEG 2 audio and video decoder) , STi5600, STi5610 (MPEG 2 audio/video decoder with integrated powerful 2D graphics) and STi7000 (HD MPEG 2 video decoder and HD Display processor), STi5000 (integrating micro controllers) and many many others which we are not listing here for sake of brevity since spanning 30 years of developments.
  
 
Using low pin count and plastic packages without heat sinks mitigated the manufacturing cost obstacle and increased reliability of those chips that also necessitated from a simple 8-bit micro controller to a more complex 32bits CPU as companion first and then integrated in the SoC.  
 
Using low pin count and plastic packages without heat sinks mitigated the manufacturing cost obstacle and increased reliability of those chips that also necessitated from a simple 8-bit micro controller to a more complex 32bits CPU as companion first and then integrated in the SoC.  
  
In term of addressing the acceleration obstacle, by using such a deep hardware optimized design approach vs a full software one, the achieved performances were incomparable to state of art micro controllers such as 68020 which was unthinkable to be used due the excessive two digits number of them (to implement the decoding function) since 68020 was not optimized to process 60 or 120 million of pixels per second such the one required by high-definition MPEG 2 decoding. This need was further exacerbated later by the need to use on screen graphics to enhance used experience.
+
In term of addressing the acceleration obstacle, by using such a deep hardware optimized design approach vs a full software one, the achieved performances were incomparable to state of art micro controllers such as 68020 which was not an option to be used due the excessive two digits number of them (to implement the decoding function) since 68020 was not optimized to process 60 or 120 million of pixels per second such the one required by high-definition MPEG 2 decoding. This need was further exacerbated later by the need to use on screen graphics to enhance used experience.
  
 
Another obstacle to address was user experience which required high quality images produced with real time graphics processing. This was addressed by super integrating incrementally powerful image blitters, on screen display processor, 3D and 2D vector graphics engines to support user interfaces, internet browsers, gaming and 3D TV. ST was the 1st in bringing 3D graphics to the digital consumer market by implementing OpenGL-ES and OpenVG standards from the Khronos group.
 
Another obstacle to address was user experience which required high quality images produced with real time graphics processing. This was addressed by super integrating incrementally powerful image blitters, on screen display processor, 3D and 2D vector graphics engines to support user interfaces, internet browsers, gaming and 3D TV. ST was the 1st in bringing 3D graphics to the digital consumer market by implementing OpenGL-ES and OpenVG standards from the Khronos group.
Line 205: Line 242:
 
Organizational obstacles
 
Organizational obstacles
  
At the beginning in 1986 the digital multimedia applications and services were not existing because of the analog TV transmissions were dominating in everyone entertainment life, while the PC one was marginal for that domain. By consequence, the SGS-THOMSON management was reluctant to invest initially in a full generation of STi chips with an unpredictable return on investment a-head. The internal investments were purely R&D without any revenues, just losses in term of money discouraging further investments. The seminal cooperation between Rennes Electronics Laboratory, ENST Paris and Thomson Semiconductor under DAII sponsorship since 1986 first, then the development opportunities with Apple computer and RCA/DirecTV removed those barriers and were instrumental for SGS-Thomson to put 30 years of efforts into a roadmap of chips, manufacture them and to boost the chip production motivating subsequent investments and multimedia applications and service flourishment.
+
At the beginning in 1986 the digital multimedia applications and services were not existing because of the analog TV transmissions were dominating in everyone entertainment life, while the PC one was marginal for that domain. By consequence, the company management was reluctant to start investing in a full roadmap of MPEG chips with an unpredictable return on investments. The internal investments were purely R&D without any revenues, just losses in term of money discouraging further actions. The seminal cooperation between Rennes Electronics Laboratory, ENST Paris and Thomson Semiconductor under DAII sponsorship since 1986 first, then the development opportunities with Apple computer and then RCA/DirecTV removed those barriers and were instrumental for the company to be motivated to engage 30 years of efforts and into a roadmap of chips, manufacture them and to boost the chip production motivating subsequent investments and multimedia applications and service flourishment.
  
The architecture of the DCT chip was invented at ENST, [4.10] [4.11] [7.0], and the DCT industrial chip that was designed within the ENST laboratory (Alain Artieri, a key ST architect, inventor and design engineer of the STi family spent a year in ENST team) using a specific CAD tool that was also developed within ENST, that made possible to generate this Hardware-Optimized solution. At that time, the head of the IC research group at ENST was Professor Francis Jutand, and Nicolas Demassieux was assistant professor. Nicolas Demassieux was personally participating to the MPEG1 (and later MPEG2) standardization efforts, and this work was fueled by EC-funded projects (such as the VADIS project). Both of Jutand and Demassieux, with Alain Artieri came to SGS-Thomson, to convince the company managers to adopt DCT architecture and CAD tool. At that time, the internal ST design team was proposing 12 chips solutions, using gate-arrays. It took nearly a year to convince the manager of ST design team that Jutand and Demassieux had a credible solution, and a contract was signed between ENST and ST to a) licensing the intellectual property of two patents to ST, and 2) set-up a joint-design team that would design the DCT chip. Alain Artieri came for a year in the ENST team, and we designed together (with help of a couple of younger students) the layout of chip, while all the verifications (DRC, logical simulation....) were carried out in ST at Grenoble. Both Alain Artieri and Nicolas Demassieux designed the chip as equals in the collaboration.
+
The architecture of the DCT chip was invented at ENST, [4.10] [4.11] [7.0], and the DCT industrial chip that was designed within the ENST laboratory (Alain Artieri, a key ST architect, inventor and design engineer of the MPEG family spent a year in ENST team) using a specific CAD tool that was also developed within ENST, that made possible to generate this Hardware-Optimized solution. At that time, the head of the IC research group at ENST was Professor Francis Jutand, and Nicolas Demassieux was assistant professor. Nicolas Demassieux was personally participating to the MPEG1 (and later MPEG2) standardization efforts, and this work was fueled by EC-funded projects (such as the VADIS project). Both Prof Jutand and Demassieux, with Alain Artieri came to SGS-Thomson, to convince the company managers to adopt DCT architecture and CAD tool. At that time, the internal ST design team was proposing 12 chips solutions, using gate-arrays. It took nearly a year to convince the manager of ST design team that Prof Jutand and Demassieux had a credible solution, and a contract was signed between ENST and ST to a) licensing the intellectual property of two patents to ST, and 2) set-up a joint-design team that would design the DCT chip. Alain Artieri came for a year in the ENST team, and designed together (with help of a couple of younger students) the layout of chip, while all the verifications (DRC, logical simulation....) were carried out in ST at Grenoble. Both Alain Artieri and Nicolas Demassieux designed the chip as equals in the collaboration.
  
The ST team in Grenoble should be hailed for this adventure, and for being open-minded to change their "usual way of working" and accept this innovative idea of a joint academic/industry design team, to deliver this world's first STV3200 chip. This was a unique case (to our best knowledge) of an industrial chip being designed by a joint academic/industry team, through the collaboration between ENST and ST. Initial research work at ENST in 1986, described in [1.24]. [4.10] [4.11] patents granted to ENST in 1986 resulting from this research. Licensing and contractual agreement between ST and ENST during 1986 with final meeting to close the deal on Nov 12, 1986.
+
The ST team in Grenoble should be hailed for this adventure, and for being open-minded to change their "usual way of working" and accept this innovative idea of a joint academic/industry design team, to deliver this world's first STV3200 chip. This was an unique case (to our best knowledge) of an industrial successful chip being designed by a joint academic/industry team, through the collaboration between ENST and ST. Initial research work at ENST in 1986, described in [1.24]. [4.10] [4.11] patents granted to ENST in 1986 resulting from this research. Licensing and contractual agreement between ST and ENST during 1986 with final meeting to close the deal on Nov 12, 1986.
  
Key top management directive (from Aldo Romano, top executive with almost 47 years of tireless entrepreneurial work in ST) was to seek the market interest on those chips and to start working with an initial customer: it was Apple Computer with whom SGS-Thomson specified the first MPEG1 ST3240 chip using available STV3200 implementing DCT. Secondly it was RCA, acquired by Thomson Consumer Electronics, was massively investing on Digital Satellite broadcast but its end-to-end system was lacking affordable decoding technologies. SGS-Thomson management decision was to take the risk on the decoder development without any commitment from RCA to buy those chips, with potential loss of development costs if the product would go in production too late. Thanks to the deep hard work of SGS-Thomson a key ST engineer such as Alain Artieri and his colleagues, it was released to RCA, one and half year ahead of C-Cube therefore being the first decoder being adopted into DirectTV digital multimedia broadcast. The business was granted by ST !  The business started to be self sustainable and went on for 30 years !
+
Key top management directive (from Aldo Romano, CEO and President of the product division with almost 47 years of tireless entrepreneurial work in ST) was to seek the market interest on those chips and to start working with an initial customer: it was Apple Computer with whom SGS-Thomson specified the first MPEG1 ST3240 chip using available STV3200 implementing DCT. Secondly it was RCA, acquired by Thomson Consumer Electronics, was massively investing on Digital Satellite broadcast but its end-to-end system was lacking affordable decoding technologies. Company management decision was to take the risk on the decoder development without any commitment from RCA to buy those chips, with potential loss of development costs if the product would go in production too late. Thanks to the deep hard work of key ST engineers such as Alain Artieri and his colleagues, it was released to RCA, one and half year ahead of C-Cube therefore being the first decoder being adopted into DirectTV digital multimedia broadcast. The business was granted by ST !  The business started to be self sustainable and went on for 30 years !
|a5=As reported into [1.9] many solutions for image and compression ICs were existing in 1992 however SGS-Thomson started since 1986. The ST innovations, which have been outlined in this IEEE Milestone proposal and which have created and provided significantly advancements in the field of digital multimedia, can be concisely summarized as follows:
+
|a5=[1.29] is the report of the meeting with DCT manufacturers invited to report chip maturity to CCITT SGXV on 24-25 May 1988. Annex I reported a summary of documents and presentations on 24 May 1988.  S. Kritter  was describing the Thomson chip and performances. It is reported that current chip design available June 88. it was the only one ready for production. New version with improved performance available mid 89. J Guichard (CNET) was examinating IDCT error when using RM5 at 64kbit/s and the Thomson chip in the decoder concluding that there was no visible mismatch error when using the Thompson chip at the decoder (Miss America and Clare sequences). Guichard was reporting also the maturity of the SGS-Thomson chip.  Inmos declared 1st samples on march 1989. Toshiba had a dsp (80MIPS) not a dedicated chip implementation for real time video. Fujitsu described an architecture and did not comment on chips. Telettra had working chip, but this did not meet current CCITT performance requirement therefore a new chip with improved performance was needed on 1989. ANT was not working at rates greater than 2Mbit/s and was for still pictures not for video rates. No specific conclusions from ATT.
 +
As reported into [1.9] many solutions for image and compression ICs were existing in 1992 however we claimed our developments started since 1986. The innovations have been outlined in this IEEE Milestone proposal, have created and provided significantly advancements in the field of digital multimedia, can be concisely summarized as follows:
  
1) The silicon development started on 1986 between Rennes Electronics Laboratory, ENST Paris and Thomson Semiconductor under DAII sponsorship a-head of anybody else
+
1) The silicon development started on 1986 between ENST Paris, Thomson Semiconductor and Rennes Electronics Laboratory, under DAII sponsorship
  
 
2) The 1st chip for mass production for forward and inverse DCT was STV3200 in 1988.
 
2) The 1st chip for mass production for forward and inverse DCT was STV3200 in 1988.
Line 220: Line 258:
 
3) The 1st chip for mass production for motion estimator was STV3220 in 1989. search window independence was further achieved with [1.27] [1.28] which mitigated the memory  bottleneck by introducing cache based estimation
 
3) The 1st chip for mass production for motion estimator was STV3220 in 1989. search window independence was further achieved with [1.27] [1.28] which mitigated the memory  bottleneck by introducing cache based estimation
  
4) Their hardwired implementations replaced software solutions which needed hundreds of Motorola 68020 which were unable to achieve production and deployment into end user Set top boxes and digital TV.
+
4) Their hardwired implementations replaced software solutions which would need hundreds of Motorola 68020 which were unable to achieve mass production and deployment into end user Set top boxes and digital TV.
  
 
5) They incrementally implemented innovative techniques to reduce memory bandwidth such as block based efficient burst memory read and write, picture decoding on the fly, frame buffer on the fly compression to minimize external RAM needs and enable super integration of 1 bit transistor memory RAM cells in an on per requirement fashion.
 
5) They incrementally implemented innovative techniques to reduce memory bandwidth such as block based efficient burst memory read and write, picture decoding on the fly, frame buffer on the fly compression to minimize external RAM needs and enable super integration of 1 bit transistor memory RAM cells in an on per requirement fashion.
  
6) Gave the designer the freedom to use more advanced CMOS silicon lithography such as 0.5, 0.35um and successive in order to achieve super integration of more and more functionalities, to meet affordable mass production. the last one developed by ST was 25nm FDSOI offering low power high density integration capabilities
+
6) Gave the designer the freedom to use more advanced CMOS silicon lithography such as 0.5, 0.35um and subsequent in order to achieve super integration of more and more functionalities, to meet affordable mass production being the last one 25nm FDSOI offering low power high density integration capabilities
  
7) Integrate more heterogeneous functions such as micro controller (8 bits, 32bits), audio decoding and 3D OpenGL-ES graphics
+
7) Integrate more heterogeneous functions such as micro controller (8 bits, 32bits), audio decoding and 3D OpenGL-ES and OpenVG graphics
  
 
8) System on Chip integration including faster processor cores (high instruction per cycles and frequency), caches (to handle code density and latency), memory controllers (for minimal latency communication with DDR), assisted by tool chains (with compilers adopted by a wide developer community)  
 
8) System on Chip integration including faster processor cores (high instruction per cycles and frequency), caches (to handle code density and latency), memory controllers (for minimal latency communication with DDR), assisted by tool chains (with compilers adopted by a wide developer community)  
  
9) Addressing HDTV/UDTV decoding, Video encoding and content Transcoding in real time
+
9) Addressing HDTV/UDTV decoding, Video encoding and Transcoding in real time
  
 
10) Super integrating MPEG2 transport processing capabilities using STM32 bit microprocessor.
 
10) Super integrating MPEG2 transport processing capabilities using STM32 bit microprocessor.
Line 238: Line 276:
 
[[image:Figure10 power consumption.png|thumb|Figure 11 Power dissipation minimized by STi family under 1W ]]
 
[[image:Figure10 power consumption.png|thumb|Figure 11 Power dissipation minimized by STi family under 1W ]]
  
12) A continued investment on R&D and production started on 1986 till 2016 for 30 years of break-through products that shaped the digital multimedia domain (figure 12).   
+
12) A continued investment on R&D and production started on 1986 till 2016 for 30 years of break-through unwavered product commitment that shaped the digital multimedia domain (figure 12).   
  
 
13) STi3500 was the enabler of the first Digital TV broadcast ever, that was the key to the worldwide leadership of SGS-THOMSON in digital multimedia market.
 
13) STi3500 was the enabler of the first Digital TV broadcast ever, that was the key to the worldwide leadership of SGS-THOMSON in digital multimedia market.
Line 304: Line 342:
 
IEEE Transactions on image processing 11 (4), 387-392
 
IEEE Transactions on image processing 11 (4), 387-392
  
 +
[1.29] Report of the meeting of the DCT chip manufacturers in Martlesham UK 24-25 May 1988, CCITT SGXV Document #349, Working Party XV/1 21 June 1988, Specialists Group on Coding for Visual Telephony
 +
 +
[1.30] An Integrated MPEG-1 and MPEG-2 Decoder,  IEEE International Conference on Consumer Electronics,  21-23 June 1994
 +
https://ieeexplore.ieee.org/document/582280
  
 
2. ONLINE INFORMATION AND CITATIONS
 
2. ONLINE INFORMATION AND CITATIONS
Line 326: Line 368:
  
 
[2.10] http://ciclops.org/ir_index/208/In-Orbit?js=1  
 
[2.10] http://ciclops.org/ir_index/208/In-Orbit?js=1  
 +
 +
[2.11] https://en.wikipedia.org/wiki/Allan_Alcorn
 +
 +
[2.12] https://www.amazon.com/dp/B096G6TSF9/
 +
 +
[2.13] Computer History Museum,  Oral History of Allan Alcorn Recorded: April 26, 2008 and May 23, 2008
 +
Mountain View, California
 +
https://archive.computerhistory.org/resources/access/text/2012/09/102658257-05-01-acc.pdf
  
  
Line 390: Line 440:
  
 
[7.7] Endorsement by Professor Andrea Basso who directly contributed to MPEG standards since 90s while in EPFL and in AT&T Bell labs and AT&T Research Laboratories, Middletown, NJ, USA
 
[7.7] Endorsement by Professor Andrea Basso who directly contributed to MPEG standards since 90s while in EPFL and in AT&T Bell labs and AT&T Research Laboratories, Middletown, NJ, USA
 +
 +
[7.8] Endorsement by Allan Alcorn, Apple Fellow  in the late 80s in the Advanced Technology Group at Apple Computer, Cupertino USA
 +
 +
[7.9] Endorsement by Jeff Cooper, former  Senior R&D Engineer at Thomson Multimedia - RCA in the 90s responsible for DirectTV MPEG2 set top box design based on STi3500.
 +
 +
[7.10] Endorsement by Shigenobu Minami, former  R&D team leader at Toshiba for MPEG4 standardization and LSI planning in the 90s.
 
|submitted=Yes
 
|submitted=Yes
 
}}
 
}}

Latest revision as of 16:06, 28 June 2021


To see comments, or add a comment to this discussion, click here.

Docket #:2021-10

This proposal has been submitted for review.


To the proposer’s knowledge, is this achievement subject to litigation? No

Is the achievement you are proposing more than 25 years old? Yes

Is the achievement you are proposing within IEEE’s designated fields as defined by IEEE Bylaw I-104.11, namely: Engineering, Computer Sciences and Information Technology, Physical Sciences, Biological and Medical Sciences, Mathematics, Technical Communications, Education, Management, and Law and Policy. Yes

Did the achievement provide a meaningful benefit for humanity? Yes

Was it of at least regional importance? Yes

Has an IEEE Organizational Unit agreed to pay for the milestone plaque(s)? Yes

Has an IEEE Organizational Unit agreed to arrange the dedication ceremony? Yes

Has the IEEE Section in which the milestone is located agreed to take responsibility for the plaque after it is dedicated? Yes

Has the owner of the site agreed to have it designated as an IEEE Milestone? Yes


Year or range of years in which the achievement occurred:

1986

Title of the proposed milestone:

Multimedia Integrated Circuits for MPEG, 1986

Plaque citation summarizing the achievement and its significance:

THOMSON (now STMicroelectronics) pioneered the family of multimedia integrated circuits, accelerating Moving Picture Experts Group (MPEG) standards. Discrete Cosine Transform designed with ENST (now Telecom ParisTech) was the first. Innovative memory reduction techniques, low bandwidth motion estimation, compensation, super-integration of heterogeneous processor units achieved significant performances and cost reduction with minimized energy consumption. These integrated circuits were the key enablers of digital multimedia services to entertain end-users in everyday life.

In what IEEE section(s) does it reside?

France Section

IEEE Organizational Unit(s) which have agreed to sponsor the Milestone:

IEEE Organizational Unit(s) paying for milestone plaque(s):

Unit: France
Senior Officer Name: Claire LAJOIE-MAZENC

IEEE Organizational Unit(s) arranging the dedication ceremony:

Unit: France with STMicroelectronics
Senior Officer Name: Claire LAJOIE-MAZENC

IEEE section(s) monitoring the plaque(s):

IEEE Section: France
IEEE Section Chair name: Claire LAJOIE-MAZENC

Milestone proposer(s):

Proposer name: Danilo Pau
Proposer email: Proposer's email masked to public

Proposer name: Jean-Michel Moutin
Proposer email: Proposer's email masked to public

Please note: your email address and contact information will be masked on the website for privacy reasons. Only IEEE History Center Staff will be able to view the email address.

Street address(es) and GPS coordinates of the intended milestone plaque site(s):

(45.203333, 5.695833)

Describe briefly the intended site(s) of the milestone plaque(s). The intended site(s) must have a direct connection with the achievement (e.g. where developed, invented, tested, demonstrated, installed, or operated, etc.). A museum where a device or example of the technology is displayed, or the university where the inventor studied, are not, in themselves, sufficient connection for a milestone plaque.

Please give the address(es) of the plaque site(s) (GPS coordinates if you have them). Also please give the details of the mounting, i.e. on the outside of the building, in the ground floor entrance hall, on a plinth on the grounds, etc. If visitors to the plaque site will need to go through security, or make an appointment, please give the contact information visitors will need. The Grenoble site is of premier importance for STMicroelectronics. It was owned by THOMSON SEMICONDUCTEURS, then merged with SGS in 1987 to form SGS-Thomson. Nowadays is the biggest R&D site of the company, hosting many product divisions from the three Product Groups of the company, including silicon and software design, test, and advanced packaging developments. In this site MPEG chips were conceived and designed

Are the original buildings extant?

Yes, even if part of them were restructured they still exists @ STMicroelectronics, 12, rue Jules Horowitz F-38000 Grenoble, France

Details of the plaque mounting:

Plaques will be installed in a public place at the main entrance as shown in figure 1. It will be in front of company security personnel who are monitoring entrance 24 hours in a day, every day in a year. Very close as indicated there is the public road. Plaque will be placed at 12, rue Jules Horowitz F-38000 Grenoble, France. GPS coordinates are: 45°12’12” N ; 05°41’45” E (45.203333, 5.695833). Therefore, interested visitors will not need to be escorted by the company security personnel to watch the plaque or to take a photo with it. The milestone plaque will be installed right in front of the main entrance access. It is where employees, visitors and customers must pass through to get into the Grenoble offices on daily basis. It is monitored continuously 24h/7d by security human resources and surveillance camera. It is a public place, right outside the site restricted perimeter.

How is the site protected/secured, and in what ways is it accessible to the public?

It is protected by company security human resources and with surveillance camera, 24h/7d. It is publicly and easily accessible from 12, rue Jules Horowitz F-38000 Grenoble, France. The site can be reached also from Lion Airport, close to Grenoble as shown in figure 2.

Who is the present owner of the site(s)?

STMicroelectronics (Grenoble 2) SAS

What is the historical significance of the work (its technological, scientific, or social importance)?

Figure 1 Candidate place for plaque installation
Figure 2 How to get the plaque from Lyon airport

Since mid 80s the technology objective of STMicroelectronics (ST) was to conceive, ahead of MPEG standards, and introduce innovative hardware solutions to be produced in high volumes, able to achieve several order of magnitude speed acceleration, compared to unfeasible, hypothetical, very costly full software implementation of MPEG decoding and encoding functionalities, by introducing for the first time very advanced techniques such as RAM memory compression, reductions and associated embedding on chip, dedicated and customized hardware implementations in order to save precious silicon area per technology node being used (starting from 2um and the next ones), super integrate new functionalities, as the needs evolved in the next 30 years, in a less than a watt power envelope. The DCT (discrete cosine transform forward and backward) integrated circuit, the first in the family, was designed to be very reliable and produced in mass volumes such that was subsequently used in the Cassini-Huygens probe for transmitting compressed images of Saturn for 20 years. This is an important proof that the integrated circuits was very resilient to cosmic rays and therefore also for (on earth) multimedia entertainment applications.

Specific objectives, that motivated ST to pioneer Mpeg family of System on a Chips and to pursue associated CMOS silicon technology developments to implement it, starting since 1986 and onward, were:

1) To create reliable and low complex digital CMOS building (integrated circuits) blocks at mass production maturity to dramatically accelerate MPEG video decoding and encoding functionalities to achieve real time at increasing image resolutions and frame rates.

2) To super-integrate such building blocks in more complex integrated circuits and SoCs to accelerate MPEG 1, 2, 4, H264, HEVC and subsequent standardized MPEG specifications, at the same time they were under development by ISO/IEC SC29 WG11 (known as MPEG committee). In other words before the standards were officially frozen and be formally published.

3) To achieve lowest on chip memory footprint and bit-depth of computing data paths to dramatically save silicon area given the existing (from 2µm bulk to 25nm FDSOI) in house developed CMOS technology constraints for mass production and reliable manufacturability with marginal chip defectivity.

4) To achieve minimal power consumption, to and below 1W across different silicon technology generations, starting from 2um silicon gate manufacturing processes. Successive lower dimension lithography, were needed considering an increasing need to super integrate functionalities, heterogeneous hardware accelerators, microprocessors, graphic engines, crypto engines, a rich set of peripherals and interfaces etc. ST developed in house all subsequent CMOS technologies such as 0.5um, 0.35um up to the most sophisticated 25nm FDSOI, fully depleted silicon on insulator) needed to sustain the pace of super integrating the multimedia functionalities.

5) To remove any need of costly and bulky heat-sinks as well as complex and motorized cooling mechanisms to offer an easier viable integration into thinner and very cheap video equipment manufactured systems with very low and affordable costs.

6) To dramatically reduce as low as possible video communication bandwidth between the chip and the external DRAM memories, to minimize memory requirements of those and associated power consumption below 1 W.

7) To control those chips with very simple and low computing power 8bits micro controllers and incrementally (as requirements has grown) with more sophisticated 32 bits ones; this to add flexibility and to support complex operating systems that required further software computing needs to offer un precedented user experience, multitasking and programmability. Including the use of advanced graphics to ease content accessibility and fruition.

8) To enable viable, widely adopted, broad range of diversified digital multimedia applications and services, implying affordable software developments, end to end full stack and deployments to a huge range of end users at worldwide level with different regional languages, habits and needs.

In the following, there is the summary of achievements by the family of MPEG integrated circuits subject of the present milestone proposal. To meet the above-mentioned challenging objectives and long-term technology vision, since 1986, ST has developed following key innovative technologies and chips. A few but very relevant set of them will be described, since the chip family is too numerous to be reported exhaustively in this document and as it was spanning in 30 years of intense and diversified developments.

1) Seminal work on started at ENST [1.24] [1.25] [7.0] with Thomson Semiconductors (now ST), to produce the architecture of an economically viable chip implementing various DCT block sizes 16x16, 16x8, 8x8, 8x4, 4x4 much ahead of the requirements MPEG specified in the subsequent years . The chip under development with Thomson Semiconductors was using 2 µm CMOS lithography first and then 1.25 µm CMOS technology . Design was embodying 70,400 Transistors, in 25 mm2, with Internal clock 13.5 MHz, forward and backward DCT, 8 bits pixels, 16 bits internal accuracy. The chip was sampled by Thomson Semiconductors in October 1987.

2) Then STV3200 [1.1] [1.2] was a dedicated integrated circuit to accelerate the discrete cosine transform (DCT). The two-dimensional forward DCT (FDCT) and inverse DCT (IDCT) were implemented for various image block sizes and the pixel rate was up to 15.0 MHz The circuit architecture was fully bidirectional with a 9-bit magnitude pixel data bus and a 12-bit magnitude coefficient data bus programmed as input or output depending on the selection of Forward DCT or Inverse DCT function. DCT was so complex that to be implemented on a CPU required about 1 billion additions per second (1 giga Hertz operations per second) to process moving pictures in real time. CPUs and CMOS silicon process were unable to achieve such a very high frequency, equivalently implying the computing power of fifty Motorola 68020 [2.1], one of the faster micro-processors in the 1980s. The integrated circuit went officially in production at the end of 1987 and was a world premiere product. Since 1988, this component was successfully sold by SGS-Thomson after the merge. It implemented key patents [4.1][4.2] for rearranging, permuting or selecting data according to predetermined rules, for changing the order of data flow, e.g. matrix transposition, LIFO (last in first out) buffers; internally compensating and handling overflow or underflow that was key for DCT and IDCT accurate and reliable implementations.

Innovative memory footprint reduction and distribution techniques were used since the hardware design did not use the traditional architecture (such as a unique multiplier that would read/write in a large memory), but a bit-serial pipelined architecture, where dynamic memory was distributed across the chip, to feed the 16 bit/serial dedicated multipliers that the chip had.

The STV3200 chip was considered very reliable at the point it was used in the DISR (descent imager) [1.26] [2.9] [7.0] of Cassini-Huygens probe, therefore resilient to cosmic rays; it was launched in 1997 and has transmitted wonderful images of Saturn planet until 2017. All those historic images that are available in this website [2.10] have been processed through the STV3200 for 20 years through the solar system!

Design of the STV3200 happened during the December 1986 to August 1987 period (product sampling around October 1987) as mentioned at the end of [1.25]. As said commercialization of the STV3200 started in 1988. It must be noted that on November 1988 there was the publication of the H261 standard (1st standard using DCT for real time video coding) while on May 1988 there was the 1st meeting of the MPEG standard (that was published in 1993) using block based DCT. This chip was considered as proof of implementation by MPEG ad-hoc group

Figure 3a DCT chip package
Figure 3b DCT chip layout

3) STi3220 [1.3] [1.4] [1.5] was the second member of the family, a real time motion estimation integrated chip dedicated to motion estimation at video rates. The chip was optimized to compute the displacement vector of variable size 8x4n or 16x4n pixel blocks in a search window, and between temporally distant pictures, defined by a maximum horizontal and vertical displacement of +7/-8 pixels corresponding to 256 different vectors. The chip computed 256 distortions for each block search according to the MAE (mean absolute error) criteria. The motion estimation required about 4 billion of additions per second (4 giga Hertz operations per second). If it would be hypothetically implemented on micro processor it would require the equivalent computing power of several hundred of Motorola 68020. Therefore, it was unfeasible in software. Its development started in 1988, went in production and has been sold by SGS-Thomson since 1989. This component was another world premiere product in conjunction with STV3200. Furthermore, the Motion estimation processor was a key building block to accelerate more complex MPEG-2 Video Encoders [1.17]. Further innovative algorithms and hw design were conceived to remove any dependence from the block based search windows estimation, therefore reducing computational complexity as well as being able to accurately estimate real life motion [1.27] [1.28]. Cache based motion estimation was possible due to the recursive algorithms we invented and was designed to avoid block based data fetching with highly on chip data reusability therefore reducing memory bandwidth. [4.12] [4.13]

4) Super integration of those fundamental functions with others fundamental building blocks (e.g. scalar quantization and de-quantization, entropy coding and decoding as later needed by MPEG standard, motion compensation, unified dynamic memory management), was the next essential step required to manufacture reliable cheap system solutions for mass production and broad adoption while before they were consisting of few non integrated chips. In this respect [1.20] presents the deployment of STV3200 and STi3220 in a complete video codec hardware architecture for high pixel rates addressing standard (SD) and high definition (HD) picture formats.

5) STi3240 [1.18] was the 1st MPEG1/H261 chip with STV3208 (DCT processor) [1.19] supporting video decoding schemes up to 10 Mbit/s of the ISO/MPEG future (at that time under active definition) video standard and CCITT/H261 recommendation at video rate (352x288 pixels at 30fps). It required between 512 K Bytes up to 4Mbytes of external costly power hungry DRAM with a 8/16 bits microprocessor interface. External DRAM was a scarce and costly resource, dramatically impacting the overall power consumption by order of magnitude and bill of material. Therefore, the use of on the fly, real time memory reduction based on compression techniques to support high chip calculations throughput and minimized data bandwidth between the chip and the memory were conceived and applied within the company for the first time and implemented. The fundamental idea behind the introduction of these techniques was to let MPEG compression noise to hide the one implied by the on the fly lossy compression generated by the memory reduction. Numerous tests at different bit rates (1.5, 4, 8, 16 mbit/s) done in cooperation with National Council for Researches confirmed the applicability of the original idea as well as the low complex deployment into multimedia systems sold for many years.

Figure 4 decoder system, STV3208 was DCT co processor

6) Then the system on chip super integration era started with active developments. Earliest members of the digital multimedia chip family for mpeg moving picture processing were: STi3400, STi3500, STi3520A. Other early examples were STi7000, STi5000 (and the very many others that followed for 30 years of developments and market adoption). They were the first able to offer a dramatic reduction in silicon and package size, power consumption, memory usage with higher level of transistors integration, functionalities, and lowest cost of equipment’s. Such equipment’s were made previously by many discrete signal processors (DSP), making the system architecture very redundant, and which were placed into too many more boards. They were far from being viable from a mass manufacture point of view and cost affordable for MPEG-1 and MPEG-2 standards deployment at end user side. Those chips were very instrumental to accelerate the introduction to the end users of CD-ROM, DVDs, Set Top Box and digital TVs home systems that became ubiquitously adopted throughout the digital entertainment world and in everyone home (and mobile phones) across 3 decades all over the world.

7) Inside those chips’, innovative and advanced reduction techniques of external memory bandwidth were conceived and implemented first. They were key to lower power consumptions to and under 1W. In that respect several key methods are described in the following subsections:

a) block based pixel memory addressing and communication [4.3] allowed to minimize the cost of external memory page openings and closures, distributing the costly overhead per block size (composed by several pixels) instead of per single or per few pixels. In turn this allowed to avoid hardware pipeline stalls resulting to achieve very close to peak performances and resulting in a full motion interactive frame rates, clear picture decoding, free from block artefacts

b) compressed picture decoding on the fly [4.4] was a break-through technology to decompress the mpeg bitstream and motion compensate it at the same time while displaying MPEG B-pictures. This to avoid inefficient and further storage of them in the external frame buffer, thus resulting in a dramatic reduction of the external memory to only 16 Mbits for SD definition [1.20] within the memory bandwidth budget

c) frame buffer lossy compression [1.11] [4.5] was another key technology used to co-decompress on the fly I , P and B MPEG pictures by halving the associated external DRAM memory and the memory bandwidth without annoying picture artefacts. As before introduced, in the chip family the quantization noise implied by the memory reduction was cleverly masked under the MPEG coding noise, resulting in not human perceivable and visible loss of quality. This allowed to reach 8 Mbits DRAM footprint in SD definition enabling super integration of 1 bit transistor per cell DRAM on the same chip, in a more competitive way than integrating 6 transistor per cell SRAM. therefore those chips were the 1st to not require any additional external memory for the MPEG SD video decompression task. Moreover another kind of memory reduction was applied to HDTV decoders with even greater memory benefits. [1.23] was novel method to reduce the external memory needed by MPEG-2 HDTV decoder architecture. The total amount of memory was reduced from 128 to 32 Mbits preserving very good picture quality. Furthermore, it required low hardware complexity increases of less than 5%, in 0.35 um CMOS silicon technology, the total decoder silicon area with respect to the standard decoder implementation.

8) The chip family implementation of the aforementioned vision was very instrumental to create the digital Multimedia domain. That was made possible with those and many other chips we are not mentioning for space reason but that spanned across 30 years for company engineering efforts.

It's important to detail that the pioneering development work began since 1986 [1.24] [1.26] among different and key French partners as proof of the passion, motivation, technical vision permeating the France R&D context:

1) Thomson Semiconductor launched innovative circuit architecture studies for integrated for the Discrete Cosine Transform since 1986

2) ENST Paris (now Telecon ParisTech) was very active since 1986 in integrated circuit architectures for DCT [1.24] [1.25] single chip for video rates design.

3) Rennes Electronics Laboratory, expert in the field of image compression, built image transmission equipment between studios and digital recording.

Under the finance support of the DAII (that time French State Agency) these three laboratories collaborated to form a formidable team responsible for realizing the first DCT component (STV3200) and the Motion Estimation chip (STi3220). The DCT and Motion estimation chips existed thanks to the strong collaboration between SGS-Thomson and the public research lab of ENST.

Thus, on 1988, SGS-Thomson finalized these seminal developments into production (with background developments since 1986 [1.24] [1.26] in Thomson Semiconductor) therefore starting the wave of key integrated circuits and SoC for MPEG deployment. By consequence it mastered at industrial level initially the 2 key components which made possible and economically viable the realization of more complex, reliable, ready for mass production, yet low cost and low power MPEG-1 and MPEG-2 encoders and decoder systems for the (mid 80s) for the not yet existing digital multimedia television markets.

9) In 1988, SGS-Thomson, aware of his advanced technical development for future MPEG SoC, contacted Apple Computer to collaborate with a reference system company in the field of image compression. The 68020 microprocessors was used from March 1987 to March 1992 and only used twice, once in the Macintosh II as a high-end processor, and then again with the Macintosh LC, as a low-end processor. However, it was too slow and inadequate to support real time video encoding functions as for MPEG decoding at interactive frames. By acknowledging this technical limit, the collaboration between the 2 companies started, the first in its kind, to study and develop highly integrated set of components to provide video functions for Apple's microcomputers. That was a very advanced concept for that time. The partnership was with Apple ATG (Advanced Technology Group) headed by Al Alcorn Apple Fellow [2.11]. The following is an extract from the Alcorn, Al (Allan) oral history Catalog Number 102658257 Computer History Museum, Mountain View, California. From [2.13] pag 49 Al answered to Lowood question : What was the state of MPEG at that time? "Alcorn: Academic. You had people from AT&T, Bell Labs, which was kind of industry academic, you had the French phone company involved. It was really academic but they were associated with some corporations. The French, SGS Thomson, we became partners with....". And in pag 50 Alcorn: We built at Apple, that team under Mark Cutter, actually built the first real-time MPEG decoder ever in hardware. The state of technology at that time was you certainly couldn’t do this in software. It had to be dedicated hardware to decode or encode MPEG. So they actually used very complex DSP boards in a Mac -- we had to use a Mac because we’re at Apple -- but it took like three high-powered cards with a high-speed bus across the top. It was certainly not product development. It could never be a product that way. So that was the state. It really was something that could not be done under CPU software. Fascinating story of how this evolved. So two things we did: One was we got SGS Thomson interested in making this chip because they already had a couple of DCT [Discrete Cosine Transform] cores, parts of the company, and it was kind of a castaway little team of very bright engineers that didn’t fit the traditional mold of Thomson. They were using high level languages, VHDL [VHSIC hardware description language], to do chips, which was new. It was coming off of polygons on physical stuff. We actually worked with them cooperatively with no agreement, nothing, and co-worked with this to help them make a real MPEG decoder chip. They were driven by the idea that that hey, I’m Apple, we’re going to buy a million of these things. And ___________________________ Vergon (actual name is Guy Lauvergeon) was the guy at Thomson SGS to do this. I worked with him and they finally built the chip, got it to work. How many am I going to buy? I’ll buy two. Uh-oh, no, you’re kidding. No, I can’t buy any of these. I’m research. “Al, I’m going to get clobbered, I’m going to get fired. I did all this stuff and you promised. Oh, come on.” He was in trouble, but then there was this little project. Hughes Satellite had this idea of beaming digital signals to the home for movies and stuff. Guess who had the only chip that would do this decoded in the set? Now, he (Guy Lauvergeon) was a hero, but there were two months when he was in trouble, but he became a hero. So we co-opted them. What was the other one? Well, I’ll think of it." . Allan Alcorn testimony is considered very relevant to support this milestone nomination. His endorsement letter is included in the proposal as reference [7.8]

Thus, based on the key understanding of 68020 limits and the contribution of innovative chip solutions from our company background work, the first architecture of an integrated system for image compression was defined to support the MPEG1 standard: the STi3240 [1.18], STi3400 [1.6]. STi3400 was a real time video, super integrated with DCT, decompression integrated processor supporting MPEG-1 and H.261 standards. The digital output was for PAL 50Hz and NTSC 60 Hz interlaced displays.

10) Next leveraging its expertise, the our company started a background work with RCA, which ended in 1992 with the development of an MPEG2 video decoder, STi3500 [1.7] and subsequent STi3520 [1.8] enabling DirecTV broadcast. Those were other essential components at the heart of the revolutionary multimedia digital television services introduced firstly from the USA Digital Satellite DirectTV firm. Our company succeeded in delivering the first MPEG2 decoder in the world, much ahead of the Californian start-up C-CUBE. As such the STi3500 was the enabler of the first Digital TV broadcast ever, that was the key to the worldwide leadership of SGS-THOMSON in digital multimedia market. It must be noticed that C-Cube was funded on 1988 when our company had already above-mentioned chips in production and since when Thomson semiconductor started on 1986.

11) STi3500 [1.30] was the real time video decompression integrated circuit supporting MPEG-1 and MPEG-2 standards at video rates up to 720x480x60Hz or 720x576x50Hz requiring minimal support from an external 8bit microcontroller (ST8) used only to initialize the chip, reducing the complexity of software and associated code footprint. It required only 1W of power consumption. This chip was a first of its kind [7.9] and successfully developed. This was a major breakthrough for Thomson Multimedia - RCA business for video decoding and was critical in launching DirecTV business on time The STi3500 was a success since achieved a turnover greater than $ 100M in less than 3 years as a sign of significant market adoption. In that chip STV3200 (which was super integrated) represented only 10% of the total silicon area. STi3520 on top of STi3500, used only 16Mbits memory and integrated audio decoder compliant with MPEG layers I and II supporting sampling rates of 32, 44.1 and 48 KHz. STI3520A [1.20] was a single chip for MPEG MP@ML video decoding in only 16 Mbit RAM integrated with STi4500 MPEG L1 and L2 Audio decoder and integrating on screen display generator. The chip was using 0.5µm CMOS company technology

12) STi7000 [1.12] [2.2] was an integrated system for High-Definition Television (HDTV), which combined an MPEG-2 decoder with a display and format converter onto one chip in 1998. As already introduced in point 5) the most advanced trick was the frame buffer memory reduction through tiny on chip on the fly compression engine based on scalar adaptive quantization allowing full HDTV pictures to be decoded and displayed with only 32 or 64 Mbits of external memory instead of 128 Mbits. Designed for use in HDTV and other digital TV receivers, set top boxes and PCs, the STi7000 was developed in collaboration with Thomson Multimedia, a strategic partner with whom our company shared a joint design center in Grenoble, France. The chip incorporated all the 18 video formats defined by the ATSC (Advanced Television Systems Committee) and Grand Alliance specifications. STi7000 supported video rates of up to 1920 x 1088 x 30Hz interlaced or 1280 x 720 x 60Hz progressive. Built in 0.35-micron HCMOS6 silicon technology, the STi7000 also included interfaces for a host microcontroller, local SDRAM, standard or high-definition video output and D1 digitized video input. Next versions STi7100 [1.13] STi7200 [1.14] also integrated graphics engine and power powerful micro controllers. Next generation STi7108 had dual ST40-300 CPU host processors linked to a 256K L2 cache giving up to 2000 DMIPS performance and a total of 4000 DMIPS. A 3D graphics engine [1.15] enabled advanced Internet content and high-performance gaming. It was the first set-top box IC [1.16] in the market to combine 3D OpenGL-ES 2.0, OpenVG vector graphics, Ethernet, USB and e-SATA interfaces to connect Internet devices, DVR storage or external Flash or hard-disk (HDD) drives.

13) The STi5500 was the first member of back-end decoders for set-top boxes and DVD players. Compliant with MPEG-2 to decode its transport stream and convert it into sounds and images that sent to loudspeakers, monitors, TV screens and similar human interfaces. The STi5500 was the first integrated circuit in the world to incorporate not only all the circuitry required to handle all the back-end functions but also a 32-bit microprocessor. The built-in microprocessor had sufficient processing power to handle system-level functions as well as the managing the MPEG decoding function, so it eliminated the need for a separate system controller in most applications. Its super integration was the crucial factor and achieved immediate customers adoption. Since then, ST has developed other more powerful members including the STi5505, which is the first silicon chip in the world to integrate all the back-end functions of a DVD player. As demonstration of the innovation it brought in the field of digital multimedia entertainment , STi5500 won [5.1] from a field of nearly 300 nominations, the prestigious European IT PrizesPrize awarded by the European Commission in conjunction with Euro-CASE, the European Council of Applied Science and Engineering, for the development of products that are expected to play fundamental roles in helping European industry to increase its share of world markets.

Summarizing the MPEG family of integrated circuits gained its leadership position in the field of digital consumer multimedia decoder-type products for digital satellite TV and DVD players. Early members such as STV3200, STi3220, STi3240, STi3400, STi3500, STi3520A, STi7000, STi7100, STi7108, STi5500 SoC through the 80s, 90s and 2000 years continued for the development of digital multimedia applications at heart of the worldwide digital multimedia convergence of the TV services for everyday users, the world of telecommunications and the world of the PC.

Many other generations were manufactured across the years as shown in the next figure 5 even supporting complex operative systems such as Linux.

Figure 5

Interestingly it has to be noted that many decades later, the pioneering work that conducted to STV3200 represented less than 0.1% of the surface of the SoC (figure 6).

Figure 6 Super integration of DCT with other multimedia functions

ST has sold over 500,000,000 (cumulated) of those integrated systems worldwide. This figure undoubtedly demonstrates the pervasivity of the MPEG chip family through the digital multimedia entertainment world. Figure 7 shows cumulative shipments of company products per year basis to further support the statement.

Figure 7

14) As proof of the high relevance of the proposed milestone a number of letter of supports were provided by key, respectful and well know experts in the field

a) [7.0] by Nicolas Demassieux, today Senior Vice President of Research, Orange [2.3], and in 80s assistant professor at ENST (now Telecom ParisTech) working on very optimized VLSI implementations for signal processing, included the DCT processor.

b) [7.1] by Leonardo Chiariglione [2.3], the undisputed father of MPEG ISO/IEC SC29 WG11 and the tireless driving force of MPEG standards for digitized video. Leonardo is unanimously considered at world wide level a genius since created a new way of enjoying music, with Leonardo Chiariglione’ s MP3 standard. As mentioned in the public article [6.1] with Leonardo Chiariglione, SGS-­Thomson has been a big MPEG driver since it was the world's leading producer of MPEG-related ICs and supplied close to 70 percent of the world's MPEG-2 decoder chips. That was the result of the audiovisual vision that was shared by SGS-Thomson, Thomson Multimedia SA, Paris, and France's government. Also, Chiariglione noted, Thomson's U.S. subsidiary RCA was in the desperate need of MPEG chips for a huge order of 1st generation satellite TV decoders--a million set-top boxes for the Hughes-RCA DirecTV newborn system--and SGS-Thomson Microelectronics got the contract, as proof of its leading integrated circuits technology started since 1986. He is the author of the book "Even the stars die: The history of MPEG and how it made digital media happen" about the story of how the Moving Picture Experts Group (MPEG) was conceived and established. It's also available [2.12]. In summary, the book tells the story of how MPEG engineered the transformation of analogue media, how it fostered the development of digital media, how it was terminated and how its spirit continues in the field of data coding by Artificial Intelligence through MPAI.

c) [7.2] by Professor emeritus Fabio Rocca. He on 1969 was the pioneer on motion estimation and compensation technologies fundamental technologies being adopted by MPEG standards 20 years later.

d) [7.3] by Hisafumi Yamada, past Sony USA TV CTO that prove the high innovation STi HDTV chips provided to ATV USA television.

e) [7.4] by Sylvain Kritter, currently Product Director VIZYON with proven expertise on HW design. During mid 80s he was a IC design manager working on very optimized hardware implementations of discrete cosine transform functions as witnessed by his IEEE papers and patents mentioned in the letter . He was using Thomson 1.25 µm CMOS technology at that time

f) [7.5] by Professor Emeritus Virginio Cantoni, University of Pavia. Also Fellow of the IAPR since 1994 and Life Fellow of the IEEE (he was Fellow since 1997). In 2006 the President of the Italian Republic conferred to Professor Virginio Cantoni the title of ‘Commander of the Order of Merit of the Italian Republic’. Professor Virginio Cantoni is an expert of VLSI image processors exploiting reduced memory architectures

g) [7.6] by Professor Emeritus Mariagiovanna Sami, Politecnico di Miano. Past Full Professor, Digital Systems. IEEE Life Member. Minerva prize for woman scientists, the Seymour Cray prize for contributions in the area of parallel processing, the Herbert A. Simon Gold Medal, assigned by the Society for Design and Process Science. Cavaliere della Repubblica Italiana (knight of the Italian republic). Member of the Italian National Science Academy (“Dei Quaranta”). Expert of high-performance, fault-tolerant array architectures for digital signal and image processing.

h) [7.7] by Professor Andrea Basso who has been active in ISO/IEC SC19 WG11 working group also known as MPEG for almost 25 years. Since 1990 while in EPFL in the group of Murat Kunt and then in AT&T Bell labs Holmdel NJ and AT&T Research Laboratories, Middletown, NJ, USA in the group of Barry Haskell, Professor Basso directly contributed the development of several MPEG standards starting from MPEG1 and MPEG2 till the most recent HEVC standard.

i) [7.8] by Allan Alcorn Apple Fellow in the late 80s in the Advanced Technology Group at Apple Computer, Cupertino USA

l) [7.9] Endorsement by Jeff Cooper, former Senior R&D Engineer at Thomson Multimedia - RCA in the 90s responsible for DirectTV MPEG2 set top box design based on STi3500 that was 1st MPEG2 decoder critical for launching DirecTV service.

m) [7.10] Endorsement by Shigenobu Minami, former R&D team leader at Toshiba for MPEG4 standardization and LSI planning in the 90s.

15) In the following years the MPEG products were the key enabling factor of a rich value chain digital consumer ecosystem composed by many operators, equipment manufacturers, software developers and conditional access companies as shown in the next figure 6

Figure 8 Worlds wide level ecosystem for digital multimedia services created thanks to STi family

16) As written in the EETimes article [2.4] “In the first shift or wave (referred to in the set-top box market) , which began about 1994, the boxes were simple channel-hopping devices for satellite and cable TV providers, Hatch explained. (ST's MPEG-2 video decoders were used then in the boxes manufactured by Thomson's RCA subsidiary.)” which clearly witness ST was ahead.

17) As written in the EDN article [2.5] on 1996 “Dataquest has confirmed SGS-THOMSON Microelectronics (Lincoln, MA) as the world's leading supplier of MPEG decoder ICs in 1995.” which witnessed ST leadership on MPEG decoders.

18) as written into a public article STMicroelectronics NV History [2.6] quoting “ ST is the world's leading manufacturer of analog ICs (integrated circuits) and MPEG-2 decoder chips, used to provide video decompression for DVD players and digital television set-top boxes.” and “Another area in which SGS Thomson became an early player was its development of MPEG decompression chips. The company's dedication to MPEG technology was risky, as a number of other digital video decompression schemes were competing for what promised to be a huge market in the future. SGS Thomson gained an early lead in the MPEG market, however, releasing its first Motion Estimation Processor in 1990. By 1993, the company had debuted its "multimedia" chip, capable of decompressing digital video files for display on a television set. This chip helped the company take a major position in the new set-top box market, starting with supplying the chip for the Hughes digital satellite television set-top box.” It helps to also state that ST was dominating the MPEG domain winning over the risks due to its early investments.

19) ST was also not only pioneer of MPEG2 in satellite digital multimedia business with DirecTV but also in Cable business with Scientific Atlanta (which was acquired by Cisco later). Thanks to the collaboration the MPEG chips named 5600 and 5610 were the first chips to integrate graphics engine with multimedia processors [1.21], [1.22].

What obstacles (technical, political, geographic) needed to be overcome?

One key initial obstacle was represented by the 2.0 and 1.25 um CMOS silicon technology that prevented to super integrate in a single chip many complex hardwired and micro controllers blocks. Therefore, hardware designers had to carefully decide what to accelerate with respect to what to assign to a software implementation, for example on 68020 micro controllers used in Apple computers or on ST8 micro controller. Therefore as result of an accurate design, STV3200 needed only 115,000 transistors because of the hardware optimized design involving multiple technical dimensions such as.

1) data precision of internal calculations, as minimized between 8, 12 to 16 bits

2) datapath branches to compensate low precision compute of some parts of the low bit depth circuitry to be compliant with IEEE 1180-1990, the IEEE Standard Specifications for the Implementations of 8x8 Inverse Discrete Cosine Transform,

3) internal memory distribution, transposition [4.1][4.2] of read and write memory data access to avoid using costly additional SRAM blocks.

These dimensions were put in the chip despite the need to support various block formats such as 16x16, 8x8, 4x4, 16x8, 8x4, 8x16, 4x8 that will be specified many years later by MPEG-1, 2, 4, H.264 and HEVC video standards. DCT coefficients were minimized to use only 12bits reducing furthermore internal memory footprint [4.6]. Internal data have been processed at 16 bits to minimize loss of precision instead of using 32 bits integer arithmetic. floating points was not considered as too transistor and silicon area demanding. Integrated memory was only 4Kbits and be transposable in the usage [4.1] [4.2] in real time by the internal controller avoiding unnecessary duplications and pipelining. Also, DCT mathematical separability to 1D was exploited to save silicon resources [4.7], [4.8], [4.9].

Similar optimizations were applied to STi3220 motion estimator based on block matching between block of pixels; by limiting the search window to 256 positions the silicon complexity of the block matcher was reduced, shifting the random-access memory bottleneck into proper pixel burst access to avoid the costly penalties of opening and closing memory pages (to access frame buffers) at pixel level with associated deadly loss of efficiency of the processor. Further innovative algorithms and hw design to achieve independence from the block based search windows estimation, therefore reducing computational complexity and be able to accurately estimate real life motion [1.27] [1.28]. Cache based motion estimation was introduced due to the recursive algorithms we invented and was designed to avoid block based data fetching with highly on chip data reusability therefore reducing memory bandwidth. [4.12] [4.13]

The software computing obstacle (and bottleneck) had to be addressed in the most efficient manner.

1) By hardwiring DCT and motion estimation/compensation processing functions, the most silicon area demanding function, hundreds of powerful micro controllers (such as Motorola 68020 or equivalent) were not anymore needed, while only an external 8bit controller using only 8Kbytes of ROM was capable to initiate decoding and searching operations once for any picture that was under decompression without being on the decoding critical path nor performing pixel level operations. That removed the need for any time critical handshake between the external micro controller and the integrated circuits, avoiding hardware pipeline stalls that could result in picture freeze or annoying block artefacts affecting picture quality.

Another challenge, as the silicon technology advanced for STi3240, STi3400, STi3500, STi3520, STi7000, STi5500 and successive derivative chips, was the memory bottleneck that became even more urgent to address due to the unified memory addressing space used by different accelerators. DRAM was costly, power hungry and severely limited in storage space and bandwidth which in turn prevented the super integration since required to store full resolution I, P, B pictures and MPEG compressed bitstream. Decoding PAL and NTSC resolutions into only 16 Mbits of memory, to keep low the system costs, implied several tricks to be implemented such as MPEG-2 B pictures decoding on the fly [4.4] and frame buffer on the fly compression [4.3] [4.5] [1.11] applied to both standard and high-definition decoders across the chip family. Novel memory reductions technique fixed the obstacle allowing MPEG SD decoding in only 8Mbits and HTDV in 32Mbits one of the most impressive achievement at that time. The memory bottleneck was solved. In figure 9 the picture of the first consumer high definition MPEG video chip. The Motion compensation unit with memory reduction is the block on the bottom right corner, with a raw of memory at the bottom and two almost parallel rows on the top. These are mostly due to memory related to data management for decompression

Figure 9 STi7000 layout with memory reduction integrated

The energy consumption obstacle needed to be addressed to avoid power hungry systems. SoC optimizations allowed to keep the power consumption challenge between 0.5 and 1 W because higher power consumption would have reduced dramatically the reliability of the chips and the costs. Chips were not using any heat sink other than a cheap plastic package. This allowed to increase speed and functionality up to full High-Definition pictures within a single chip, also addressing the engine orchestration challenge due the on-chip CPU, video, audio and graphics concurrent executions and later on, super integrating also the micro controller (8 to 32bits), which resulted into dramatically minimized bill of material and greatly improved performances, as compared with competition multi-chip implementations.

The MPEG family initially consisted of: STV3200 (DCT), STV3220 (Motion Estimator), STi3240, STi3400 (MPEG 1 video decoder), STi3500 (MPEG2 Video decoder), STi3520 (MPEG 2 audio and video decoder) , STi5600, STi5610 (MPEG 2 audio/video decoder with integrated powerful 2D graphics) and STi7000 (HD MPEG 2 video decoder and HD Display processor), STi5000 (integrating micro controllers) and many many others which we are not listing here for sake of brevity since spanning 30 years of developments.

Using low pin count and plastic packages without heat sinks mitigated the manufacturing cost obstacle and increased reliability of those chips that also necessitated from a simple 8-bit micro controller to a more complex 32bits CPU as companion first and then integrated in the SoC.

In term of addressing the acceleration obstacle, by using such a deep hardware optimized design approach vs a full software one, the achieved performances were incomparable to state of art micro controllers such as 68020 which was not an option to be used due the excessive two digits number of them (to implement the decoding function) since 68020 was not optimized to process 60 or 120 million of pixels per second such the one required by high-definition MPEG 2 decoding. This need was further exacerbated later by the need to use on screen graphics to enhance used experience.

Another obstacle to address was user experience which required high quality images produced with real time graphics processing. This was addressed by super integrating incrementally powerful image blitters, on screen display processor, 3D and 2D vector graphics engines to support user interfaces, internet browsers, gaming and 3D TV. ST was the 1st in bringing 3D graphics to the digital consumer market by implementing OpenGL-ES and OpenVG standards from the Khronos group.

Figure 10 graphic images rendered by STi family to enhance user experience

  Organizational obstacles

At the beginning in 1986 the digital multimedia applications and services were not existing because of the analog TV transmissions were dominating in everyone entertainment life, while the PC one was marginal for that domain. By consequence, the company management was reluctant to start investing in a full roadmap of MPEG chips with an unpredictable return on investments. The internal investments were purely R&D without any revenues, just losses in term of money discouraging further actions. The seminal cooperation between Rennes Electronics Laboratory, ENST Paris and Thomson Semiconductor under DAII sponsorship since 1986 first, then the development opportunities with Apple computer and then RCA/DirecTV removed those barriers and were instrumental for the company to be motivated to engage 30 years of efforts and into a roadmap of chips, manufacture them and to boost the chip production motivating subsequent investments and multimedia applications and service flourishment.

The architecture of the DCT chip was invented at ENST, [4.10] [4.11] [7.0], and the DCT industrial chip that was designed within the ENST laboratory (Alain Artieri, a key ST architect, inventor and design engineer of the MPEG family spent a year in ENST team) using a specific CAD tool that was also developed within ENST, that made possible to generate this Hardware-Optimized solution. At that time, the head of the IC research group at ENST was Professor Francis Jutand, and Nicolas Demassieux was assistant professor. Nicolas Demassieux was personally participating to the MPEG1 (and later MPEG2) standardization efforts, and this work was fueled by EC-funded projects (such as the VADIS project). Both Prof Jutand and Demassieux, with Alain Artieri came to SGS-Thomson, to convince the company managers to adopt DCT architecture and CAD tool. At that time, the internal ST design team was proposing 12 chips solutions, using gate-arrays. It took nearly a year to convince the manager of ST design team that Prof Jutand and Demassieux had a credible solution, and a contract was signed between ENST and ST to a) licensing the intellectual property of two patents to ST, and 2) set-up a joint-design team that would design the DCT chip. Alain Artieri came for a year in the ENST team, and designed together (with help of a couple of younger students) the layout of chip, while all the verifications (DRC, logical simulation....) were carried out in ST at Grenoble. Both Alain Artieri and Nicolas Demassieux designed the chip as equals in the collaboration.

The ST team in Grenoble should be hailed for this adventure, and for being open-minded to change their "usual way of working" and accept this innovative idea of a joint academic/industry design team, to deliver this world's first STV3200 chip. This was an unique case (to our best knowledge) of an industrial successful chip being designed by a joint academic/industry team, through the collaboration between ENST and ST. Initial research work at ENST in 1986, described in [1.24]. [4.10] [4.11] patents granted to ENST in 1986 resulting from this research. Licensing and contractual agreement between ST and ENST during 1986 with final meeting to close the deal on Nov 12, 1986.

Key top management directive (from Aldo Romano, CEO and President of the product division with almost 47 years of tireless entrepreneurial work in ST) was to seek the market interest on those chips and to start working with an initial customer: it was Apple Computer with whom SGS-Thomson specified the first MPEG1 ST3240 chip using available STV3200 implementing DCT. Secondly it was RCA, acquired by Thomson Consumer Electronics, was massively investing on Digital Satellite broadcast but its end-to-end system was lacking affordable decoding technologies. Company management decision was to take the risk on the decoder development without any commitment from RCA to buy those chips, with potential loss of development costs if the product would go in production too late. Thanks to the deep hard work of key ST engineers such as Alain Artieri and his colleagues, it was released to RCA, one and half year ahead of C-Cube therefore being the first decoder being adopted into DirectTV digital multimedia broadcast. The business was granted by ST ! The business started to be self sustainable and went on for 30 years !

What features set this work apart from similar achievements?

[1.29] is the report of the meeting with DCT manufacturers invited to report chip maturity to CCITT SGXV on 24-25 May 1988. Annex I reported a summary of documents and presentations on 24 May 1988. S. Kritter was describing the Thomson chip and performances. It is reported that current chip design available June 88. it was the only one ready for production. New version with improved performance available mid 89. J Guichard (CNET) was examinating IDCT error when using RM5 at 64kbit/s and the Thomson chip in the decoder concluding that there was no visible mismatch error when using the Thompson chip at the decoder (Miss America and Clare sequences). Guichard was reporting also the maturity of the SGS-Thomson chip. Inmos declared 1st samples on march 1989. Toshiba had a dsp (80MIPS) not a dedicated chip implementation for real time video. Fujitsu described an architecture and did not comment on chips. Telettra had working chip, but this did not meet current CCITT performance requirement therefore a new chip with improved performance was needed on 1989. ANT was not working at rates greater than 2Mbit/s and was for still pictures not for video rates. No specific conclusions from ATT. As reported into [1.9] many solutions for image and compression ICs were existing in 1992 however we claimed our developments started since 1986. The innovations have been outlined in this IEEE Milestone proposal, have created and provided significantly advancements in the field of digital multimedia, can be concisely summarized as follows:

1) The silicon development started on 1986 between ENST Paris, Thomson Semiconductor and Rennes Electronics Laboratory, under DAII sponsorship

2) The 1st chip for mass production for forward and inverse DCT was STV3200 in 1988.

3) The 1st chip for mass production for motion estimator was STV3220 in 1989. search window independence was further achieved with [1.27] [1.28] which mitigated the memory bottleneck by introducing cache based estimation

4) Their hardwired implementations replaced software solutions which would need hundreds of Motorola 68020 which were unable to achieve mass production and deployment into end user Set top boxes and digital TV.

5) They incrementally implemented innovative techniques to reduce memory bandwidth such as block based efficient burst memory read and write, picture decoding on the fly, frame buffer on the fly compression to minimize external RAM needs and enable super integration of 1 bit transistor memory RAM cells in an on per requirement fashion.

6) Gave the designer the freedom to use more advanced CMOS silicon lithography such as 0.5, 0.35um and subsequent in order to achieve super integration of more and more functionalities, to meet affordable mass production being the last one 25nm FDSOI offering low power high density integration capabilities

7) Integrate more heterogeneous functions such as micro controller (8 bits, 32bits), audio decoding and 3D OpenGL-ES and OpenVG graphics

8) System on Chip integration including faster processor cores (high instruction per cycles and frequency), caches (to handle code density and latency), memory controllers (for minimal latency communication with DDR), assisted by tool chains (with compilers adopted by a wide developer community)

9) Addressing HDTV/UDTV decoding, Video encoding and Transcoding in real time

10) Super integrating MPEG2 transport processing capabilities using STM32 bit microprocessor.

11) Low dissipation <1W (figure 11) and high throughput efficiency

Figure 11 Power dissipation minimized by STi family under 1W

12) A continued investment on R&D and production started on 1986 till 2016 for 30 years of break-through unwavered product commitment that shaped the digital multimedia domain (figure 12).

13) STi3500 was the enabler of the first Digital TV broadcast ever, that was the key to the worldwide leadership of SGS-THOMSON in digital multimedia market.

Figure 12

Supporting texts and citations to establish the dates, location, and importance of the achievement: Minimum of five (5), but as many as needed to support the milestone, such as patents, contemporary newspaper articles, journal articles, or chapters in scholarly books. 'Scholarly' is defined as peer-reviewed, with references, and published. You must supply the texts or excerpts themselves, not just the references. At least one of the references must be from a scholarly book or journal article. All supporting materials must be in English, or accompanied by an English translation.

1. TECHNICAL ARTICLES, CONFERENCE PAPERS & BOOKS

[1.1] A. Artieri, S. Kritter, F. Jutand and N. Demassieux, "A one chip VLSI for real time two-dimensional discrete cosine transform," 1988., IEEE International Symposium on Circuits and Systems, 1988, pp. 701-704 vol.1, doi: 10.1109/ISCAS.1988.15022.

[1.2] STV3200 DISCRETE COSINE TRANSFORM (DCT) Datasheet

[1.3] A. Artieri and F. Jutand, "A versatile and powerful chip for real-time motion estimation," International Conference on Acoustics, Speech, and Signal Processing, 1989, pp. 2453-2456 vol.4, doi: 10.1109/ICASSP.1989.266964.

[1.4] STi3220 MOTION ESTIMATOR PROCESSOR Datasheet

[1.5] APPLICATION NOTE STi3220 MOTION ESTIMATION PROCESSOR CODEC

[1.6] STi3400 MPEG/H.261 VIDEO DECODER DATASHEET

[1.7] STi3500 MPEG2 CCIR 601H.261 VIDEO DECODER DATASHEET

[1.8] Image and Video Compression standards: algorithm and architectures, Bhaskaran, Vasudev, Konstantinides, Konstantinos, ISBN 978-1-4615-6199-6, 1995

[1.9] K. Konstantinides and V. Bhaskaran, "Monolithic architectures for image processing and compression," in IEEE Computer Graphics and Applications, vol. 12, no. 6, pp. 75-86, Nov. 1992, doi: 10.1109/38.163627.

[1.10] A. Artieri and O. Colavin, "A chip set core for image compression," in IEEE Transactions on Consumer Electronics, vol. 36, no. 3, pp. 395-402, Aug. 1990, doi: 10.1109/30.103150.

[1.11] A. Chimienti, M. Lucenteforte, D. Pau and R. Sannino, "A novel co-decoding scheme to reduce memory in MPEG-2 MP@ML decoder," 1998 URSI International Symposium on Signals, Systems, and Electronics. Conference Proceedings (Cat. No.98EX167), 1998, pp. 272-277, doi: 10.1109/ISSSE.1998.738080.

[1.12] STi70000 press release

[1.13] STi71000 datasheet

[1.14] STi7200 datasheet

[1.15] STi7108 datasheet

[1.16] STi7108 processor with 3D graphics made public.

[1.17] A chip set for MPEG-2 video encoding; Proceedings of the IEEE 1995 Custom Integrated Circuits Conference, 1995

[1.18] STi3240 MPEG1/H261 datasheet

[1.19] STi3208 DCT chip datasheet

[1.20] STi3520A MPEG2 chip datasheet

[1.21] STi5600 MPEG2 chip datasheet

[1.22] STi5610 MPEG2 chip datasheet

[1.23] R. Bruni, A. Chimienti, M. Lucenteforte, D. Pau and R. Sannino, "A novel adaptive vector quantization method for memory reduction in MPEG-2 HDTV decoders," in IEEE Transactions on Consumer Electronics, vol. 44, no. 3, pp. 537-544, Aug. 1998, doi: 10.1109/30.713161.

[1.24] F. Jutand, N. Demassieux, G. Concordel, J. Guichard and E. Cassimatis, "A single chip video rate 16×16 discrete cosine transform," ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing, 1986, pp. 805-808, doi: 10.1109/ICASSP.1986.1169147.

[1.25] N. Demassieux, G. Concordel, J. Durandeau and F. Jutand, "An optimized VLSI architecture for a multiformat discrete cosine transform," ICASSP '87. IEEE International Conference on Acoustics, Speech, and Signal Processing, 1987, pp. 547-550, doi: 10.1109/ICASSP.1987.1169851.

[1.26] Tomasko, M., Buchhauser, D., Bushroe, M. et al. The Descent Imager/Spectral Radiometer (DISR) Experiment on the Huygens Entry Probe of Titan. Space Science Reviews 104, 469–551 (2002). https://doi.org/10.1023/A:1023632422098

[1.27] An innovative, high quality and search window independent motion estimation algorithm and architecture for MPEG-2 encoding; FS Rovati, D Pau, E Piccinelli, L Pezzoni, JM Bard IEEE transactions on consumer electronics 46 (3), 697-705

[1.28] A complexity-bounded motion estimation algorithm A Chimienti, C Ferraris, D Pau IEEE Transactions on image processing 11 (4), 387-392

[1.29] Report of the meeting of the DCT chip manufacturers in Martlesham UK 24-25 May 1988, CCITT SGXV Document #349, Working Party XV/1 21 June 1988, Specialists Group on Coding for Visual Telephony

[1.30] An Integrated MPEG-1 and MPEG-2 Decoder, IEEE International Conference on Consumer Electronics, 21-23 June 1994 https://ieeexplore.ieee.org/document/582280

2. ONLINE INFORMATION AND CITATIONS

[2.1] https://en.wikipedia.org/wiki/Motorola_68020

[2.2] https://techmonitor.ai/technology/sgs_thomson_debuts_one_chip_hdtv_system

[2.3] Leonardo Chiariglione awards https://leonardo.chiariglione.org/public-life/awards/

[2.4] see pdf and https://www.eetimes.com/apis-help-stmicroelectronics-grow-set-top-box-business/#

[2.5] see pdf and https://www.edn.com/sgs-thomson-microelectronics-named-number-one-mpeg-decoder-supplier-by-dataquest/

[2.6] see pdf and http://www.fundinguniverse.com/company-histories/stmicroelectronics-nv-history/

[2.7] https://books.google.it/books?id=C-bgBwAAQBAJ&pg=PA234&lpg=PA234&dq=stv3200+sgs-thomson&source=bl&ots=HrOQA0LEvS&sig=ACfU3U0vpZ7K5HJzJ1qJWjjlk4yfzPW82w&hl=it&sa=X&ved=2ahUKEwj2lZ2p35HwAhWBzKQKHVrmBBMQ6AEwEnoECBUQAw#v=onepage&q=stv3200%20sgs-thomson&f=false

[2.8] http://www.quretec.com/u/vilo/edu/2002-03/Tekstialgoritmid_I/Loengud/Loeng7_Compression/www.faqs.org/faqs/compression-faq/part1/preamble.html

[2.9] https://books.google.fr/books?id=UsH8CAAAQBAJ&lpg=PA501&ots=OJr_PN0LKo&dq=stv3200%20DCT&hl=fr&pg=PA501#v=onepage&q=stv3200%20DCT&f=false

[2.10] http://ciclops.org/ir_index/208/In-Orbit?js=1

[2.11] https://en.wikipedia.org/wiki/Allan_Alcorn

[2.12] https://www.amazon.com/dp/B096G6TSF9/

[2.13] Computer History Museum, Oral History of Allan Alcorn Recorded: April 26, 2008 and May 23, 2008 Mountain View, California https://archive.computerhistory.org/resources/access/text/2012/09/102658257-05-01-acc.pdf


3. STMicroelectronics documents

[3.1] De la Transformée en Cosinus Discrète aux Processeurs Applicatifs Multimédia, Alain Artiéri, Septembre 2007, STMicroelectronics presentation

4. PATENTS

[4.1] Integrated signal processing circuit for online and column summation of digital value matrices https://patents.google.com/patent/FR2608802B1/en?inventor=alain+artieri&oq=alain+artieri&sort=old

[4.2] Transposition memory for data processing circuit https://patents.google.com/patent/FR2617621B1/en?inventor=alain+artieri&oq=alain+artieri&sort=old

[4.3] Method and apparatus for addressing a memory area of an MPEG decoder4.6

[4.4] publication number US6081298A MPEG decoder with reduced memory capacity

[4.5] ITVA960016D0 Metodo di ricompressione e decompressione adpcm di un flusso di dati digitali costituente un segnale video digitale e stimatore https://patents.google.com/patent/ITVA960016D0/it?inventor=danilo+pau&oq=danilo+pau&sort=old

[4.6] publication number 0 298 002 of 29.06.1988 Memoire de transposition pour circuit de traitement de donnes

[4.7] publication number FR2649226A1 BREWING CIRCUIT OF DATA

[4.8] publication number FR2650462B1 Device for converting a line scanning into a scanning in vertical saw teeth by bands; https://patents.google.com/patent/FR2650462B1/en?q=H04N3%2f30&inventor=alain+artieri&sort=old

[4.9] publication number EP0368731B1 Process and circuit for image representation signal filtration

[4.10] Circuit pour effectuer une transformation linéaire sur un signal numérique Patent issuer and number eu EP 0241352 A1 https://patents.google.com/patent/EP0241352A1

[4.11] Additionneur binaire comportant un opérande fixé, et multiplieur binaire parallèle-série comprenant un tel additionneur Patent issuer and number eu EP 0262032 B1 https://patents.google.com/patent/EP0262032B1

[4.12] VLSI architecture, in particular for motion estimation applications F Rovati, D Pau, L Fanucci, S Saponara, A Cenciotti, D Alfonso US Patent 6,724,823

[4.13] Motion estimation process and system D Pau, E Piccinelli, F Rovati US Patent 6,891,891

5. Honors

[5.1] Two European IT Prizes Awarded to STMicroelectronics

6. MPEG mentions

[6.1] Chiariglione and the birth of MPEG https://www.chiariglione.org/public-life/media/leonardo_ieee/prof.htm

7. Letters of support

[7.0] Endorsement by Nicolas Demassieux, Senior Vice President of Research, Orange, assistant professor at ENST (now Telecom ParisTech) mid 80s.

[7.1] Endorsement by Leonardo Chiariglione, PhD Eng, the father of MPEG

[7.2] Endorsement by Professor Emeritus Fabio Rocca, Politecnico di Milano, pioneer of motion estimation in 1969

[7.3] Endorsement by Hisafumi Yamada, former Sony US TV CTO

[7.4] Endorsement by Sylvain Kritter, Product Director VIZYON

[7.5] Endorsement by Professor Emeritus Virginio Cantoni, University of Pavia. Also Fellow of the IAPR since 1994 and Life Fellow of the IEEE (he was Fellow since 1997). In 2006 the President of the Italian Republic conferred to Professor Virginio Cantoni the title of ‘Commander of the Order of Merit of the Italian Republic’.

[7.6] Endorsement by Professor Emeritus Mariagiovanna Sami, Politecnico di Miano. Past Full Professor, Digital Systems. IEEE Life Member. Minerva prize for woman scientists, the Seymour Cray prize for contributions in the area of parallel processing, the Herbert A. Simon Gold Medal, assigned by the Society for Design and Process Science. Cavaliere della Repubblica Italiana (knight of the Italian republic). Member of the Italian National Science Academy (“Dei Quaranta”).

[7.7] Endorsement by Professor Andrea Basso who directly contributed to MPEG standards since 90s while in EPFL and in AT&T Bell labs and AT&T Research Laboratories, Middletown, NJ, USA

[7.8] Endorsement by Allan Alcorn, Apple Fellow in the late 80s in the Advanced Technology Group at Apple Computer, Cupertino USA

[7.9] Endorsement by Jeff Cooper, former Senior R&D Engineer at Thomson Multimedia - RCA in the 90s responsible for DirectTV MPEG2 set top box design based on STi3500.

[7.10] Endorsement by Shigenobu Minami, former R&D team leader at Toshiba for MPEG4 standardization and LSI planning in the 90s.

Supporting materials (supported formats: GIF, JPEG, PNG, PDF, DOC): All supporting materials must be in English, or if not in English, accompanied by an English translation. You must supply the texts or excerpts themselves, not just the references. For documents that are copyright-encumbered, or which you do not have rights to post, email the documents themselves to ieee-history@ieee.org. Please see the Milestone Program Guidelines for more information.


Please email a jpeg or PDF a letter in English, or with English translation, from the site owner(s) giving permission to place IEEE milestone plaque on the property, and a letter (or forwarded email) from the appropriate Section Chair supporting the Milestone application to ieee-history@ieee.org with the subject line "Attention: Milestone Administrator." Note that there are multiple texts of the letter depending on whether an IEEE organizational unit other than the section will be paying for the plaque(s).