Difference between revisions of "Milestone-Proposal:MPEG integrated circuits"
|Line 344:||Line 344:|
7. Letters of support
7. Letters of support
[7.0] Endorsement by
[7.0] Endorsement by Nicolas Demassieux, Senior Vice President of
Research, Orange, assistant professor at ENST (now Telecom ParisTech) mid 80s.
Research, Orange, assistant professor at ENST (now Telecom ParisTech) mid 80s.
Revision as of 16:26, 12 May 2021
To see comments, or add a comment to this discussion, click here.
This proposal has been submitted for review.
To the proposer’s knowledge, is this achievement subject to litigation? No
Is the achievement you are proposing more than 25 years old? Yes
Is the achievement you are proposing within IEEE’s designated fields as defined by IEEE Bylaw I-104.11, namely: Engineering, Computer Sciences and Information Technology, Physical Sciences, Biological and Medical Sciences, Mathematics, Technical Communications, Education, Management, and Law and Policy. Yes
Did the achievement provide a meaningful benefit for humanity? Yes
Was it of at least regional importance? Yes
Has an IEEE Organizational Unit agreed to pay for the milestone plaque(s)? Yes
Has an IEEE Organizational Unit agreed to arrange the dedication ceremony? Yes
Has the IEEE Section in which the milestone is located agreed to take responsibility for the plaque after it is dedicated? Yes
Has the owner of the site agreed to have it designated as an IEEE Milestone? Yes
Year or range of years in which the achievement occurred:
Title of the proposed milestone:
Multimedia Integrated Circuits for MPEG, 1986
Plaque citation summarizing the achievement and its significance:
SGS-THOMSON (now STMicroelectronics) pioneered the family of multimedia integrated circuits, accelerating Moving Picture Experts Group (MPEG) standards. Discrete Cosine Transform designed with ENST (now Telecom ParisTech) was the first. Innovative memory reduction techniques, highly optimized dedicated hardware accelerators, super-integration of heterogeneous processor units achieved significant performances and cost reduction with minimized energy consumption. These integrated circuits were the key enabler of digital multimedia services to entertain end-users in everyday life.
In what IEEE section(s) does it reside?
IEEE Organizational Unit(s) which have agreed to sponsor the Milestone:
IEEE Organizational Unit(s) paying for milestone plaque(s):
Senior Officer Name: Claire LAJOIE-MAZENC
IEEE Organizational Unit(s) arranging the dedication ceremony:
Unit: France with STMicroelectronics
Senior Officer Name: Claire LAJOIE-MAZENC
IEEE section(s) monitoring the plaque(s):
IEEE Section: France
IEEE Section Chair name: Claire LAJOIE-MAZENC
Proposer name: Danilo Pau
Proposer email: Proposer's email masked to public
Proposer name: Jean-Michel Moutin
Proposer email: Proposer's email masked to public
Please note: your email address and contact information will be masked on the website for privacy reasons. Only IEEE History Center Staff will be able to view the email address.
Street address(es) and GPS coordinates of the intended milestone plaque site(s):
Describe briefly the intended site(s) of the milestone plaque(s). The intended site(s) must have a direct connection with the achievement (e.g. where developed, invented, tested, demonstrated, installed, or operated, etc.). A museum where a device or example of the technology is displayed, or the university where the inventor studied, are not, in themselves, sufficient connection for a milestone plaque.
Please give the address(es) of the plaque site(s) (GPS coordinates if you have them). Also please give the details of the mounting, i.e. on the outside of the building, in the ground floor entrance hall, on a plinth on the grounds, etc. If visitors to the plaque site will need to go through security, or make an appointment, please give the contact information visitors will need. The Grenoble site is of premier importance for STMicroelectronics since was owned by THOMSON SEMICONDUCTEURS and then merged with SGS in 1987 to form SGS-Thomson (now STMicroelectronics). Nowadays is the biggest R&D site of STMicroelectronics, hosting many product divisions from the three Product Groups of the company, including silicon and software design, test, and advanced packaging developments. The site were MPEG chips were conceived and designed
Are the original buildings extant?
Yes, even if part of them were restructured they still exists @ STMicroelectronics, 12, rue Jules Horowitz F-38000 Grenoble, France
Details of the plaque mounting:
Plaques will be installed in a public place at the main entrance as shown below. It will be close to ST security personnel who are is monitoring entrance 24 hours in a day, every day in a year. Very close as indicated there are other public roads. Plaque will be placed at 12, rue Jules Horowitz F-38000 Grenoble, France. GPS coordinates are: 45°12’12” N ; 05°41’45” E (45.203333, 5.695833). Therefore, visitors will not need to be escorted by ST guards to look the plaque or to take a snapshot of it. The milestone plaque will be installed right in front of STMicroelectronics, main entrance access. It is where employees, visitors and customers must pass to get into the ST Grenoble offices on daily basis. It is monitored continuously 24h/7d by ST security human resources and surveillance camera. It is a public place, right outside ST site restricted perimeter.
How is the site protected/secured, and in what ways is it accessible to the public?
It is protected by ST security human resources and with surveillance camera, 24h/7d. It is publicly and easily accessible from 12, rue Jules Horowitz F-38000 Grenoble, France. The site can be reached also from Lion Airport, close to Grenoble as shown in figure 2.
Who is the present owner of the site(s)?
STMicroelectronics (Grenoble 2) SAS
What is the historical significance of the work (its technological, scientific, or social importance)?
The technology objective of STMicroelectronics was to conceive, ahead of MPEG standard, and introduce innovative hardware solutions able to achieve several order of magnitude speed acceleration, compared to unfeasible full software implementation of MPEG decoding and encoding functionalities, by introducing for the first time memory compressions and reductions, dedicated hardware implementation in order to save silicon area per technology node, super integrate new functionalities in a less than a watt power envelope. The DCT circuit, the first in the family, was so reliably designed and produced that was subsequently used in the Cassini-Huygens probe for transmitting compressed images of Saturn during 20 years.
Specific objectives, that motivated STMicroelectronics (short name ST) to pioneer Mpeg family of System on a Chips and associated CMOS silicon technology development to implement it and starting since 1987 and onward, were:
1) To create reliable and low complex CMOS building (integrated circuits) blocks at production maturity to dramatically accelerate real time MPEG video decoding and encoding functionalities at increasing image resolutions and frame rates.
2) To super-integrate such building blocks in more complex integrated circuits and SoCs to accelerate MPEG 1, 2, 4, H264, HEVC and subsequent standardized specifications, at the same time they were under development. In other words before the standards were officially frozen and went public.
3) To achieve lowest on chip memory footprint and bit-depth of computing data paths to dramatically save silicon area given the existing (µm) CMOS technology constraints for mass production and reliable manufacturability with marginal chip defectivity.
4) To achieve minimal power consumption, to and below 1W across different silicon technologies generations and starting from 1.2um silicon gate manufacturing processes. Successive lower dimension lithography, were needed considering an increasing need to super integrate functionalities, heterogeneous hardware accelerators, microprocessors, graphic engines, crypto engines, a rich set of peripherals and interfaces etc.
5) To remove any need of costly and bulky heat-sinks as well as complex cooling mechanisms to offer an easier viable integration into thinner video equipment manufactured systems.
6) To dramatically reduce as low as possible video communication bandwidth between the chip and the external DRAM memories to minimize memory requirements of those chips and associated costly power consumption.
7) To control those chips with very simple and low computing power 8bits micro controllers and incrementally as requirements will grew with 32 bits: this to support flexibility that requires further software computing needs to offer un precedented user experience. Including the use of advanced graphics to ease content accessibility and fruition.
8) To enable viable, widely adopted, broad range of diversified digital multimedia applications and services, implying affordable software development and deployment to a vast range of end users at worldwide level with different needs.
In the following there is the summary of achievements by STi family in the present milestone proposal. To meet the above-mentioned challenging objectives and long-term technology vision, ST, since 1986, has developed following key technologies and chips. A few but very relevant technologies and chips will be described, because the family is too huge to report completely in this document.
1) Seminal work on ST CMOS technologies, started at ENST [1.24] [1.25] [7.0] with Thomson Semiconductors (now STMicroelectronics), produced the architecture of an economically reasonable chip implementing various sizes DCT 16x16, 16x8, 8x8, 8x4, 4x4. The main features of the chip under development with THOMSON SEMICONDUCTEURS in 2 µm first and then 1.25 µm CMOS technology were 70,400 Transistors, 25 mm2, Internal clock 13.5 MHz, Direct/Reverse DCT, 8 bits pixels, 16 bits internal accuracy. The chip was sampled by THOMSON SEMICONDUCTEUR in October 1987.
2) STV3200 [1.1] [1.2] was a dedicated integrated circuit to accelerate the discrete cosine transform (DCT). The two-dimensional forward DCT (FDCT) and inverse DCT (IDCT) was implemented for various image block sizes and the pixel rate was up to 15.0 MHz The circuit architecture was fully bidirectional with a 9-bit magnitude pixel data bus and a 12-bit magnitude coefficient data bus programmed as input or output depending on the selection of Forward DCT or Inverse DCT function. DCT was so complex that to be implemented on a CPU required about 1 billion additions per second (1 giga Hertz operations per second) to process the moving pictures in real time. CPUs and CMOS silicon process were unable to achieve such a very high frequency, therefore implying the computing power of fifty Motorola 68020 [2.1], one of the faster micro-processors in the 1980s. The integrated circuit went officially in production at the end of 1987 and was a world premiere product. Since 1988, this component was successfully sold by SGS-Thomson. It implemented key patents [4.1][4.2] for rearranging, permuting or selecting data according to predetermined rules, for changing the order of data flow, e.g. matrix transposition, LIFO buffers; Overflow or underflow handling that were key for DCT and IDCT implementations.
Innovative memory footprint reduction and distribution techniques were used since the hardware design did not use the traditional architecture (such as a unique multiplier that would read/write in a large memory), but a bit-serial pipelined architecture, where memory was distributed across the chip, to feed the 16 bit/serial dedicated multipliers that the chip had.
The STV3200 chip was used in the DISR (descent imager) [1.26] [2.9] [7.0] of Cassini-Huygens probe that was launched in 1997 and has transmitted magnificent images of Saturn until 2017. All the wonderful images that are available in this site [2.10] have been processed through the chip!
Design of the STV3200 happened during the December 1986 to August 1987 period (product sampling around October 1987) as mentioned at the end of [1.25]. As said commercialization of the STV3200 started in 1988 while on November 1988 there was the publication of the H261 standard (1st standard using DCT for real time video coding) and on May 1988 there was the 1st meeting of the MPEG standard (that was published in 1993).
3) STi3220 [1.3] [1.4] [1.5] was the second member of the family, a real time motion estimation integrated chip dedicated to motion estimation at video rates. The chip was optimized to compute the displacement vector of 8x4n or 16x4n pixel blocks in a search window, between temporally distant pictures, defined by a maximum horizontal and vertical displacement of +7/-8 pixels corresponding to 256 different vectors. The chip computed 256 distortion for each block according to the MAE (mean absolute error) criteria. The motion estimation required about 4 billion of additions per second (4 giga Hertz operations per second). If it would be implemented on CPU it would require the equivalent computing power of several hundred of Motorola 68020. Therefore, it was unfeasible in software. Its development started in 1988, went in production and be sold by SGS-Thomson since 1989. This component was another world premiere product in conjunction with STV3200. Furthermore, the Motion estimation processor was a key building block to accelerate more complex MPEG-2 Video Encoders [1.17]
4) Super integration of those fundamental functions with others fundamental building blocks (e.g. scalar quantization and de-quantization, entropy coding and decoding, motion compensation, unified memory management), was the next vital step required to manufacture reliable system solution for mass production while before they were consisting of few chips. In this respect [1.20] presents the deployment of STV3200 and STi3220 in a complete video codec hardware architecture for high pixel rates addressing standard (SD) and high definition (HD) picture formats.
5) STi3240 [1.18] was the 1st MPEG1/H261 chip with STV3208 (DCT processor) [1.19] supporting video decoding schemes up to 10 Mbit/s of the ISO/MPEG future (at that time under active definition) video standard and CCITT/H261 recommendation at video rate (352x288 pixels at 30fps). It required between 512 KBytes up to 4Mbytes of external DRAM with a 8/16 bits microprocessor interface. External DRAM was a scarce and costly resource, dramatically impacting the overall power consumption and bill of material. Therefore, the use of memory reduction techniques to support high chip calculations throughput and minimized data bandwidth between the chip and the memory were conceived for the first time and implemented.
6) Then the super integration of way started with active development of System of Chip. Earliest members of the STi digital multimedia chip family for mpeg moving picture processing were: STi3400, STi3500, STi3520A. Other early examples were STi7000, STi5000 (and the very many others that followed). They were the first able to offer a dramatic reduction in silicon and package size, power consumption, with higher level of transistors integration, functionalities, and lowest cost of equipment’s. Such equipment’s were made previously by many un-optimized discrete signal processors (DSP) and which were placed into many more boards. They far from being manufacturable viably and affordably for MPEG-1 and MPEG-2 standards deployment at end user side. STi chips were very instrumental to accelerate the introduction to the end users of CD-ROM, DVDs, Set Top Box and digital TVs that became ubiquitously adopted throughout the entertainment world and in everyone home (and mobile phones) across 3 decades.
7) Inside those chips’, innovative and advanced reduction techniques of external memory bandwidth were conceived and implemented first. They were key to lower power consumption to and under 1W. In that respect several key methods are described in the following subsections:
a) block based pixel memory addressing and communication [4.3] allowed to minimize the cost of external memory page openings and closures, distributing the costly overhead per block size (composed by several pixels) instead of per single or per few pixels. In turn this allowed to avoid hardware pipeline stalls resulting in using peak performances and resulting in a full motion interactive frame rates, clear picture decoding, free from block artefacts
b) picture decoding on the fly [4.4] was a break-through technology to decompress and motion compensate at the same time while displaying MPEG B-pictures. This to avoid inefficient and further storage of them in the external frame buffer, thus resulting in a dramatic reduction of the external memory to only 16 Mbits for SD definition [1.20]
c) frame buffer compression [1.11] [4.5] was another key technology used to co-decompress on the fly I , P and B MPEG pictures by halving the associated external DRAM memory and the memory bandwidth without annoying picture artefacts. Indeed, in STi solution the quantization noise was cleverly masked by MPEG coding noise, resulting in not human perceivable loss of quality. This allowed to reach 8 Mbits DRAM footprint in SD enabling super integration of 1 bit transistor per cell DRAM on the same chip, in a more competitive way than integrating 6 transistor per cell SRAM. Since then STi did not require any additional external memory for the MPEG SD video decompression task. Another kind of memory reduction was applied to HDTV decoders with even further benefits. [1.23] was novel method to reduce the external memory needed by STi MPEG-2 HDTV decoder architecture. The total amount of memory was reduced from 96 to 32 Mbits preserving a good picture quality. Furthermore, it required low hardware complexity increases of less than 5%, in 0.35 pm technology, the total decoder silicon area with respect to the standard decoder.
8) The STi implementation of the ST comprehensive vision to create the digital Multimedia domain was made possible with those and many other chips.
However, the pioneering development work began since 1986 [1.24] [1.26] among different and key French partners:
1) Thomson Semiconductor (merged later into SGS-THOMSON, now STMicroelectronics) launched innovative circuit architecture studies for integrated for the Discrete Cosine Transform since 1986
2) ENST Paris (now Telecon ParisTech) was very active and interested since 1986 in integrated circuit architectures for DCT [1.24] [1.25] single chip for video rates.
3) Rennes Electronics Laboratory, expert in the field of image compression, built image transmission equipment between studios and digital recording.
Under the finance support of the DAII (French State Agency) these three laboratories collaborated to form a team responsible for realizing the first DCT component (STV3200) and the Motion Estimation chip (STi3220). The DCT and Motion estimation chips existed thanks to the strong collaboration between SGS-Thomson and the public research lab of ENST.
Thus, on 1988, SGS-THOMSON finalized these seminal developments in production (but with background developments since 1986 [1.24] [1.26] in Thomson Semiconductor) therefore creating the wave of key circuits and SoC for MPEG. By consequence it mastered at industrial level initially the 2 key components which made possible and economically viable the realization of more complex, reliable, ready for mass production, yet low cost and low power MPEG-1 and MPEG-2 encoders and decoder systems for the (mid 80s) for the not existing digital multimedia television markets.
9) Apple Computer was looking for a technology partner in the field of image compression. Since aware of above-mentioned developments, contacted SGS-THOMSON in 1988. The 68020 microprocessors was used from March 1987 to March 1992 and only used twice, once in the Macintosh II as a high-end processor, and then again with the Macintosh LC, as a low-end processor. However, it was too slow and inadequate to support video decoding functions as for MPEG decoding. From this key technical limit, a collaboration with SGS-Thomson was started, the first in its kind, to study and develop highly integrated set of components to provide video functions for Apple's microcomputers. That was a very advanced concept for that time. Therefore SGS-THOMSON entered a partnership with Apple. Thus, based on SGS-THOMSON understanding of 68020 limits and the contribution of innovative chip solutions from SGS-THOMSON, the first architecture of an integrated system for image compression was defined to support the MPEG1 standard: the STi3240 [1.18], STi3400 [1.6]. STi3400 was a real time video, super integrated with DCT, decompression integrated processor supporting MPEG-1 and H.261 standards. The digital output was for PAL 50Hz and NTSC 60 Hz interlaced displays.
10) Next leveraging his expertise, SGS-THOMSON started a background work with RCA, which ended in 1992 with the development of an MPEG2 video decoder, STi3500 [1.7] and subsequent STi3520 [1.8]. Those were other essential components at the heart of the revolutionary multimedia digital television services introduced firstly from the USA Digital Satellite DirectTV firm. SGS-THOMSON succeeded in delivering the first MPEG2 decoder in the world, much ahead of the Californian start-up C-CUBE. It must be noticed that C-Cube was funded on 1988 when SGS-THOMSON had already above-mentioned chips in production and when Thomson semiconductor started on 1986.
11) STi3500 was a real time video decompression integrated circuit supporting MPEG-1 and MPEG-2 standards at video rates up to 720x480x60Hz or 720x576x50Hz requiring minimal support from an external 8bit microcontroller (ST8) used only to initialize the chip, reducing the complexity of software and associated code footprint. It required only 1W of power consumption. The STi3500 was a success since achieved a turnover greater than $ 100M in less than 3 years as a sign of market adoption. In that chip STV3200 (which was super integrated) represented only 10% of the total silicon area. STi3520 on top of STi3500, used only 16Mbits memory and integrated audio decoder compliant with MPEG layers I and II supporting sampling rates of 32, 44.1 and 48 KHz. STI3520A [1.20] was a single chip for MPEG MP@ML video decoding in only 16 Mbit RAM integrated with STi4500 MPEG L1 and L2 Audio decoder and integrating on screen display generator. The chip was using 0.5µm CMOS SGS-THOMSON technology
12) STi7000 [1.12] [2.2] was an integrated system for High-Definition Television (HDTV), which combined an MPEG-2 decoder with a display and format converter onto one chip in 1998. As already introduced in point 5) the most advanced trick was the frame buffer memory reduction through tiny on chip compression engine based on scalar adaptive quantization allowing full HDTV pictures to be decoded and displayed with only 64 Mbits of external memory instead of 128 Mbits. Designed for use in HDTV and other digital TV receivers, set top boxes and PCs, the STi7000 was developed in collaboration with Thomson Multimedia, a strategic partner with whom SGS-THOMSON shared a joint design center in Grenoble, France. The chip incorporated all the 18 video formats defined by the ATSC (Advanced Television Systems Committee) and Grand Alliance specifications. STi7000 supported video rates of up to 1920 x 1088 x 30Hz interlaced or 1280 x 720 x 60Hz progressive. Built in 0.35-micron SGS-Thomson HCMOS6 silicon technology, the STi7000 also included interfaces for a host microcontroller, local SDRAM, standard or high-definition video output and D1 digitized video input. Next versions STi7100 [1.13] STi7200 [1.14] also integrated graphics engine and power powerful micro controllers. STi7108 had dual ST40-300 CPU host processors linked to a 256K L2 cache giving up to 2000 DMIPS performance and a total of 4000 DMIPS. A 3D graphics engine [1.15] enabled advanced Internet content and high-performance gaming. It was the first set-top box IC [1.16] in the market to combine 3D OpenGL-ES 2.0 graphics, Ethernet, USB and e-SATA interfaces to connect Internet devices, DVR storage or external Flash or hard-disk (HDD) drives.
13) The STi5500 was the first member of back-end decoders for set-top boxes and DVD players. Compliant with MPEG-2 to decode its transport stream and convert it into sounds and images that sent to loudspeakers, monitors, TV screens and similar human interfaces. The STi5500 was the first integrated circuit in the world to incorporate not only all the circuitry required to handle all the back-end functions but also a 32-bit microprocessor. The built-in microprocessor had sufficient processing power to handle system-level functions as well as the managing the MPEG decoding function, so it eliminated the need for a separate system controller in most applications. Its super integration was the crucial factor and achieved immediate customers adoption. Since then, ST has developed even more powerful members including the STi5505, which is the first silicon chip in the world to integrate all the back-end functions of a DVD player. As demonstration of the innovation it brought in the field of digital multimedia, STi5500 won [5.1] from a field of nearly 300 nominations, the prestigious European IT PrizesPrize awarded by the European Commission in conjunction with Euro-CASE, the European Council of Applied Science and Engineering, for the development of products that are expected to play fundamental roles in helping European industry to increase its share of world markets.
Summarizing SGS-THOMSON STi gained its leadership position in the field of digital consumer multimedia decoder-type products for digital satellite TV and DVD players. Early members such as STV3200, STi3220, STi3240, STi3400, STi3500, STi3520A, STi7000, STi7100, STi7108, STi5500 SoC through the 80s, 90s and 2000 years continued for the development of digital multimedia applications at heart of the worldwide digital multimedia convergence of the TV services for everyday users, the world of telecommunications and the world of the PC.
Many other generations were manufactured across the years as shown in the next figure 5 even supporting complex operative systems such as Linux.
Interestingly many decades later, the pioneering work that conducted to STV3200 represented less than 0.1% of the surface of the SoC (figure 6).
Until STMicroelectronics has sold over 500,000,000 (cumulated) of those integrated systems worldwide. This figure undoubtedly demonstrates the pervasive of STi family through the world. Figure 7 shows cumulative shipments of ST products per year basis.
14) As proof of the high relevance of the proposed milestone a number of letter of supports were provided by key and well know experts in the field
a) A letter of endorsement [7.0] to ST milestone is provided by Nicolas Demassieux, Senior Vice President of Research, Orange [2.3], and assistant professor at ENST (now Telecom ParisTech) working on very optimized VLSI implementations for signal processing, included the DCT processor.
b) A letter of endorsement [7.1] to ST milestone is provided by Leonardo Chiariglione [2.3], the father of MPEG and the driving force behind MPEG standards for digitized video. Leonardo is unanimously considered a genius since created a new way of enjoying music, with Leonardo Chiariglione’ s MP3 standard.
As mentioned in the public article [6.1] with Leonardo Chiariglione, SGSThomson has been a big MPEG driver since it was the world's leading producer of MPEG-related ICs and supplied close to 70 percent of the world's MPEG-2 decoder chips. That was the result of the audiovisual vision that was shared by the leaders of SGS-Thomson, Thomson Multimedia SA, Paris, and France's government. Also, Chiariglione noted, Thomson's U.S. subsidiary RCA was in the need of MPEG chips for a huge order of 1st generation satellite TV decoders--a million set-top boxes for the Hughes-RCA DirecTV newborn system--and SGS-Thomson Microelectronics got the contract, as proof of its leading integrated circuits technology started since 1986.
c) A letter of endorsement [7.2] to ST milestone is provided by prof Fabio Rocca. Prof Rocca on 1969 was the pioneer on motion estimation and compensation technologies.
d) A letter of endorsement [7.3] to ST milestone is provided by Hisafumi Yamada, past Sony USA TV CTO that prove the high innovation STi HDTV chips provided to ATV USA television.
15) In the following years STi products on MPEG where the key enabling factor of a huge digital consumer ecosystem composed by many operators, equipment manufacturers, software developers and conditional access companies as shown in the next figure 6
16) As written in the EETimes article [2.4] “In the first shift or wave (referred to in the set-top box market) , which began about 1994, the boxes were simple channel-hopping devices for satellite and cable TV providers, Hatch explained. (ST's MPEG-2 video decoders were used then in the boxes manufactured by Thomson's RCA subsidiary.)” which clearly witness how ST was ahead.
17) As written in the EDN article [2.5] on 1996 “Dataquest has confirmed SGS-THOMSON Microelectronics (Lincoln, MA) as the world's leading supplier of MPEG decoder ICs in 1995.” which witnessed ST leadership on MPEG decoders.
18) as written into a public article STMicroelectronics NV History [2.6] quoting “ ST is the world's leading manufacturer of analog ICs (integrated circuits) and MPEG-2 decoder chips, used to provide video decompression for DVD players and digital television set-top boxes.” and “Another area in which SGS Thomson became an early player was its development of MPEG decompression chips. The company's dedication to MPEG technology was risky, as a number of other digital video decompression schemes were competing for what promised to be a huge market in the future. SGS Thomson gained an early lead in the MPEG market, however, releasing its first Motion Estimation Processor in 1990. By 1993, the company had debuted its "multimedia" chip, capable of decompressing digital video files for display on a television set. This chip helped the company take a major position in the new set-top box market, starting with supplying the chip for the Hughes digital satellite television set-top box.” It helps to also state that ST was dominating the MPEG domain winning over the risks due to its early investments.
19) SGS-THOMSON was also not only pioneer of MPEG2 in satellite digital multimedia business with DirecTV but also in Cable business with Scientific Atlanta (which was acquired by Cisco later). Thanks to the collaboration the STi chips named 5600 and 5610 were the first chips to integrate graphics engine with multimedia processors [1.21], [1.22].
What obstacles (technical, political, geographic) needed to be overcome?
One key initial obstacle was represented by the initially used 1.2 um silicon technology that prevented to super integrate in a single chip too many hardwired and micro controller bocks. Therefore, hardware designers had to carefully decide what to accelerate with respect to a software implementation, for example on 68020 micro controllers used in Apple computers. In fact, STV3200 needed only 115,000 transistors because of a deep hardware optimized design involving multiple technical dimensions.
1) data precision of internal calculations, as minimized between 8, 12 to 16 bits
2) datapath branches to compensate low precision compute of some part of the low bit depth circuitry to be compliant with IEEE 1180-1990, the IEEE Standard Specifications for the Implementations of 8x8 Inverse Discrete Cosine Transform,
3) internal memory distribution, transposition [4.1][4.2] of read and write memory data access to avoid using costly additional SRAM blocks.
These dimensions were met in the chip despite the need to pioneer the support of various block formats such as 16x16, 8x8, 4x4, 16x8, 8x4, 8x16, 4x8 that will be used many years later by MPEG-1,2, 4, H.264 and HEVC video standards. DCT coefficients were minimized to use only 12bits reducing furthermore internal memory footprint [4.6]. Internal data have been processed at 16 bits to minimize loss of precision instead of using 32 bits integer arithmetic. Integrated memory was only 4Kbits and be transposed [4.1] [4.2] in real time by the internal controller avoiding unnecessary duplications. Also, DCT separability to 1D was exploited to save silicon resources [4.7], [4.8], [4.9].
Similar optimizations were applied to STi3220 motion estimator based on block matching between block of pixels; by limiting the search window to 256 positions the silicon complexity of the block matcher was reduced, shifting the random-access memory bottleneck into proper burst access to avoid the costly penalties of opening and closing memory pages (to access frame buffers) at pixel level with associated deadly loss of efficiency of the processor.
The software computing obstacle had to be addressed in the most efficient manner.
1) By hardwiring DCT and motion estimation/compensation processing functions, the most silicon area demanding function, hundreds of powerful micro controllers (such as Motorola 68020 or equivalent) were not anymore needed, while only an external 8bit controller using only 8Kbytes of ROM was capable to initiate decoding and searching operation once any picture was under decompression without being on the decoding critical path nor perform pixel level operations. That removed the need for any handshake between the external micro controller and the integrated circuits, avoiding hardware pipeline stalls that could result in picture freeze or annoying block artefacts affecting picture quality.
Another challenge, as the silicon technology advanced for STi3240, STi3400, STi3500, STi3520, STi7000, STi5500 and successive derivative chips, was the memory bottleneck that became even more urgent to address due to the unified addressing space used by different accelerators. DRAM was costly and severely limited in space and bandwidth which in turn prevented the super integration since required to store I, P, B pictures and MPEG compressed bitstream. Decoding PAL and NTSC resolutions into only 16 Mbits of memory, to keep low the costs, implied several tricks to be implemented such as MPEG-2 B pictures decoding on the fly [4.4] and frame buffer compression [4.3] [4.5] [1.11] applied to both standard and high-definition decoders across the STi family. Novel memory reductions technique fixed the obstacle allowing MPEG SD decoding in only 8Mbits and HTDV in 32Mbits one of the most impressive achievement at that time.
The energy consumption obstacle needed to be addressed to avoid power hungry systems. SoC optimizations allowed to keep the power consumption challenge between 0.5 and 1 W because higher power consumption would have reduced dramatically the reliability of the chips which were not using any heat sink other than a plastic package. This allowed to increase speed and functionality up to full High-Definition pictures within a single chip, also addressing the engine orchestrating obstacle due the on-chip CPU, video, audio and graphics concurrent executions and later on, super integrating also the micro controller (8 to 32bits), which resulted into dramatically minimized costs and greatly improved performances, as compared with competition multi-chip implementations.
The STi family consisted of: STV3200 (DCT), STV3220 (Motion Estimator), STi3240, STi3400 (MPEG 1 video decoder), STi3500 (MPEG2 Video decoder), STi3520 (MPEG 2 audio and video decoder) and STi7000 (HD MPEG 2 video decoder and 3D graphics), STi5000 (integrating micro controllers) and many many others which we are not listing here for sake of brevity.
Using low pin count and plastic packages without heat sinks mitigated the manufacturing cost obstacle and increased reliability of those chips that also necessitated from a simple 8-bit micro controller to a more complex 32bits CPU as companion first and then integrated in the SoC.
In term of addressing the acceleration obstacle, by using such a deep hardware optimized design approach vs a full software one, the achieved performances were incomparable to state of art micro controllers such as 68020 which was unthinkable to be used due the excessive two digits number of them (to implement the decoding function) since 68020 was not optimized to process 60 or 120 million of pixels per second such the one required by high-definition MPEG 2 decoding. This need was further exacerbated later by the need to use on screen graphics to enhance used experience.
Another obstacle to address was user experience which required high quality images produced with real time graphics processing. This was addressed by super integrating incrementally powerful image blitters, on screen display processor, 3D and 2D vector graphics engines to support user interfaces, internet browsers, gaming and 3D TV. ST was the 1st in bringing 3D graphics to the digital consumer market by implementing OpenGL-ES and OpenVG standards from the Khronos group.
At the beginning in 1986 the digital multimedia applications and services were not existing because of the analog TV transmissions were dominating in everyone entertainment life, while the PC one was marginal for that domain. By consequence, the SGS-THOMSON management was reluctant to invest initially in a full generation of STi chips with an unpredictable return on investment a-head. The internal investments were purely R&D without any revenues, just losses in term of money discouraging further investments. Moreover, the seed cooperation between Rennes Electronics Laboratory, ENST Paris and Thomson Semiconductor under DAII sponsorship since 1986 first, then with Apple computer and RCA removed those barriers and was instrumental for SGS-Thomson to create a roadmap of chips, manufacture them and to boost the chip production motivating subsequent investments and multimedia applications and service flourishment.
The architecture of the DCT chip was invented at ENST, [4.10] [4.11] [7.0], and the DCT industrial chip that was designed within the ENST laboratory (Alain Artieri spent a year in ENST team) using a specific CAD tool that was also developed within ENST, that made possible to generate this Hardware-Optimized solution. At that time, the head of the IC research group at ENST was Professor Francis Jutand, and Nicolas Demassieux was assistant professor. Nicolas Demassieux was personally participating to the MPEG1 (and later MPEG2) standardization efforts, and this work was fueled by EC-funded projects (such as the VADIS project). Both of Jutand and Demassieux came to SGS-Thomson, to convince the company managers to adopt DCT architecture and CAD tool. At that time, the internal ST design team was proposing 12 chips solutions, using gate-arrays. It took nearly a year to convince the manager of ST design team that Jutand and Demassieux had a credible solution, and a contract was signed between ENST and ST to a) licensing the intellectual property of two patents to ST, and 2) set-up a joint-design team that would design the DCT chip. Alain Artieri came for a year in the ENST team, and we designed together (with help of a couple of younger students) the layout of chip, while all the verifications (DRC, logical simulation....) were carried out in ST at Grenoble. Both Alain Artieri and Nicolas Demassieux designed the chip as equals in the collaboration.
The ST team in Grenoble should be hailed for this adventure, and for being open-minded to change their "usual way of working" and accept this innovative idea of a joint academic/industry design team, to deliver this world's first STV3200 chip. This was a unique case (to our best knowledge) of an industrial chip being designed by a joint academic/industry team, through the collaboration between ENST and ST. Initial research work at ENST in 1986, described in [1.24]. [4.10] [4.11] patents granted to ENST in 1986 resulting from this research. Licensing and contractual agreement between ST and ENST during 1986 with final meeting to close the deal on Nov 12, 1986.
A key management directive was to seek the market interest on those chips and an initial customer which was Apple Computer with whom SGS-Thomson specified the first MPEG1 ST3240 chip using available STV3200 implementing DCT. Secondly RCA, acquired by Thomson Consumer Electronics, was massively investing on Digital Satellite broadcast but its end-to-end system was lacking affordable decoding technologies. SGS-Thomson management decision was to take the risk on the decoder development without any commitment from RCA, with potential loss of development costs. Thanks to the deep hard work of SGS-Thomson a key engineer like Alain Artieri, it was released to RCA, one and half year ahead of C-Cube therefore being the first decoder being adopted into DirectTV digital multimedia broadcast.
What features set this work apart from similar achievements?
As reported into [1.9] many solutions for image and compression ICs were existing in 1992 however SGS-Thomson started since 1986. The ST innovations, which have been outlined in this IEEE Milestone proposal and which have created and provided significantly advancements in the field of digital multimedia, can be concisely summarized as follows:
1) The silicon development started on 1986 between Rennes Electronics Laboratory, ENST Paris and Thomson Semiconductor under DAII sponsorship a-head of anybody else
3) The 1st chip for mass production for forward and inverse DCT was STV3200 in 1988.
4) The 1st chip for mass production for motion estimator was STV3220 in 1989.
5) Their hardwired implementations replaced software solutions which needed hundreds of Motorola 68020 which were unable to achieve production and deployment into end user Set top boxes and digital TV.
6) They incrementally implemented innovative techniques to reduce memory bandwidth such as block based efficient burst memory read and write, picture decoding on the fly, frame buffer compression to minimize external RAM needs and enable super integration of 1 bit transistor memory RAM cells in an on per requirement fashion.
7) Gave the designer the freedom to use more advanced CMOS silicon lithography such as 0.5, 0.35um and successive in order to achieve super integration of more and more functionalities, to meet affordable mass production.
8) Integrate more heterogeneous functions such as micro controller (8 bits, 32bits), audio decoding and 3D OpenGL-ES graphics
9) System on Chip integration including faster processor cores (high instruction per cycles and frequency), caches (to handle code density and latency), memory controllers (for minimal latency communication with DDR), assisted by tool chains (with compilers adopted by a wide developer community)
10) Addressing HDTV decoding, Video encoding and content Transcoding in real time
11) Super integrating MPEG2 transport processing capabilities using STM32 bit microprocessor.
12) Low dissipation <1W (figure 10) and high throughput efficiency
13) A continued investment on R&D and production started on 1986 till 2016 for 30 years of break-through products that shaped the digital multimedia domain (figure 11).
Supporting texts and citations to establish the dates, location, and importance of the achievement: Minimum of five (5), but as many as needed to support the milestone, such as patents, contemporary newspaper articles, journal articles, or chapters in scholarly books. 'Scholarly' is defined as peer-reviewed, with references, and published. You must supply the texts or excerpts themselves, not just the references. At least one of the references must be from a scholarly book or journal article. All supporting materials must be in English, or accompanied by an English translation.
1. TECHNICAL ARTICLES, CONFERENCE PAPERS & BOOKS
[1.1] A. Artieri, S. Kritter, F. Jutand and N. Demassieux, "A one chip VLSI for real time two-dimensional discrete cosine transform," 1988., IEEE International Symposium on Circuits and Systems, 1988, pp. 701-704 vol.1, doi: 10.1109/ISCAS.1988.15022.
[1.2] STV3200 DISCRETE COSINE TRANSFORM (DCT) Datasheet
[1.3] A. Artieri and F. Jutand, "A versatile and powerful chip for real-time motion estimation," International Conference on Acoustics, Speech, and Signal Processing, 1989, pp. 2453-2456 vol.4, doi: 10.1109/ICASSP.1989.266964.
[1.4] STi3220 MOTION ESTIMATOR PROCESSOR Datasheet
[1.5] APPLICATION NOTE STi3220 MOTION ESTIMATION PROCESSOR CODEC
[1.6] STi3400 MPEG/H.261 VIDEO DECODER DATASHEET
[1.7] STi3500 MPEG2 CCIR 601H.261 VIDEO DECODER DATASHEET
[1.8] Image and Video Compression standards: algorithm and architectures, Bhaskaran, Vasudev, Konstantinides, Konstantinos, ISBN 978-1-4615-6199-6, 1995
[1.9] K. Konstantinides and V. Bhaskaran, "Monolithic architectures for image processing and compression," in IEEE Computer Graphics and Applications, vol. 12, no. 6, pp. 75-86, Nov. 1992, doi: 10.1109/38.163627.
[1.10] A. Artieri and O. Colavin, "A chip set core for image compression," in IEEE Transactions on Consumer Electronics, vol. 36, no. 3, pp. 395-402, Aug. 1990, doi: 10.1109/30.103150.
[1.11] A. Chimienti, M. Lucenteforte, D. Pau and R. Sannino, "A novel co-decoding scheme to reduce memory in MPEG-2 MP@ML decoder," 1998 URSI International Symposium on Signals, Systems, and Electronics. Conference Proceedings (Cat. No.98EX167), 1998, pp. 272-277, doi: 10.1109/ISSSE.1998.738080.
[1.12] STi70000 press release
[1.13] STi71000 datasheet
[1.14] STi7200 datasheet
[1.15] STi7108 datasheet
[1.16] STi7108 processor with 3D graphics made public.
[1.17] A chip set for MPEG-2 video encoding; Proceedings of the IEEE 1995 Custom Integrated Circuits Conference, 1995
[1.18] STi3240 MPEG1/H261 datasheet
[1.19] STi3208 DCT chip datasheet
[1.20] STi3520A MPEG2 chip datasheet
[1.21] STi5600 MPEG2 chip datasheet
[1.22] STi5610 MPEG2 chip datasheet
[1.23] R. Bruni, A. Chimienti, M. Lucenteforte, D. Pau and R. Sannino, "A novel adaptive vector quantization method for memory reduction in MPEG-2 HDTV decoders," in IEEE Transactions on Consumer Electronics, vol. 44, no. 3, pp. 537-544, Aug. 1998, doi: 10.1109/30.713161.
[1.24] F. Jutand, N. Demassieux, G. Concordel, J. Guichard and E. Cassimatis, "A single chip video rate 16×16 discrete cosine transform," ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing, 1986, pp. 805-808, doi: 10.1109/ICASSP.1986.1169147.
[1.25] N. Demassieux, G. Concordel, J. Durandeau and F. Jutand, "An optimized VLSI architecture for a multiformat discrete cosine transform," ICASSP '87. IEEE International Conference on Acoustics, Speech, and Signal Processing, 1987, pp. 547-550, doi: 10.1109/ICASSP.1987.1169851.
[1.26] Tomasko, M., Buchhauser, D., Bushroe, M. et al. The Descent Imager/Spectral Radiometer (DISR) Experiment on the Huygens Entry Probe of Titan. Space Science Reviews 104, 469–551 (2002). https://doi.org/10.1023/A:1023632422098 2. ONLINE INFORMATION AND CITATIONS
[2.3] Leonardo Chiariglione awards https://leonardo.chiariglione.org/public-life/awards/
3. STMicroelectronics documents
[3.1] De la Transformée en Cosinus Discrète aux Processeurs Applicatifs Multimédia, Alain Artiéri, Septembre 2007, STMicroelectronics presentation
[4.1] Integrated signal processing circuit for online and column summation of digital value matrices https://patents.google.com/patent/FR2608802B1/en?inventor=alain+artieri&oq=alain+artieri&sort=old
[4.2] Transposition memory for data processing circuit https://patents.google.com/patent/FR2617621B1/en?inventor=alain+artieri&oq=alain+artieri&sort=old
[4.3] Method and apparatus for addressing a memory area of an MPEG decoder4.6
[4.4] publication number US6081298A MPEG decoder with reduced memory capacity
[4.5] ITVA960016D0 Metodo di ricompressione e decompressione adpcm di un flusso di dati digitali costituente un segnale video digitale e stimatore https://patents.google.com/patent/ITVA960016D0/it?inventor=danilo+pau&oq=danilo+pau&sort=old
[4.6] publication number 0 298 002 of 29.06.1988 Memoire de transposition pour circuit de traitement de donnes
[4.7] publication number FR2649226A1 BREWING CIRCUIT OF DATA
[4.8] publication number FR2650462B1 Device for converting a line scanning into a scanning in vertical saw teeth by bands; https://patents.google.com/patent/FR2650462B1/en?q=H04N3%2f30&inventor=alain+artieri&sort=old
[4.9] publication number EP0368731B1 Process and circuit for image representation signal filtration
[4.10] Circuit pour effectuer une transformation linéaire sur un signal numérique Patent issuer and number eu EP 0241352 A1 https://patents.google.com/patent/EP0241352A1
[4.11] Additionneur binaire comportant un opérande fixé, et multiplieur binaire parallèle-série comprenant un tel additionneur Patent issuer and number eu EP 0262032 B1 https://patents.google.com/patent/EP0262032B1
[5.1] Two European IT Prizes Awarded to STMicroelectronics
6. MPEG mentions
[6.1] Chiariglione and the birth of MPEG https://www.chiariglione.org/public-life/media/leonardo_ieee/prof.htm
7. Letters of support
[7.0] Endorsement by Nicolas Demassieux, Senior Vice President of Research, Orange, assistant professor at ENST (now Telecom ParisTech) mid 80s.
[7.1] Endorsement by Leonardo Chiariglione, the father of MPEG
[7.2] Endorsement by Prof Rocca, pioneer of motion estimation in 1969
[7.3] Endorsement by Hisafumi Yamada, former Sony US TV CTO
Supporting materials (supported formats: GIF, JPEG, PNG, PDF, DOC): All supporting materials must be in English, or if not in English, accompanied by an English translation. You must supply the texts or excerpts themselves, not just the references. For documents that are copyright-encumbered, or which you do not have rights to post, email the documents themselves to email@example.com. Please see the Milestone Program Guidelines for more information.
Please email a jpeg or PDF a letter in English, or with English translation, from the site owner(s) giving permission to place IEEE milestone plaque on the property, and a letter (or forwarded email) from the appropriate Section Chair supporting the Milestone application to firstname.lastname@example.org with the subject line "Attention: Milestone Administrator." Note that there are multiple texts of the letter depending on whether an IEEE organizational unit other than the section will be paying for the plaque(s).