Processor Watch

August 30, 2004

Editor: Tom R. Halfhill  

In this issue:

Benchmarking the Benchmarks
Tom R. Halfhill - Senior Editor  {08/30/2004}

We live in a season of divisive partisan politics: endless bickering, blame games, finger pointing, strident propaganda, arguments over strategy, and embarrassing scandals. And that’s just the politics of microprocessor benchmarking.

An outsider might reasonably wonder what all the fuss is about. Isn’t measuring the speed of a microprocessor as easy as measuring the speed of an automobile? Just use the electronic equivalent of a speedometer and be done with it. Of course, it isn’t that simple. If the marketing vice presidents from three different processor vendors were randomly selected and put in the same room, they probably wouldn’t agree on whether the best instrument of measurement is a speedometer, a tachometer, or a barometer.

For years, the embedded-processor industry limped along with the Dhrystone MIPS (DMIPS) benchmark, the descendant of a relatively simple 1984 Ada program whose reference machine was a DEC VAX 11/780 minicomputer. Even its author, Reinhold Weicker, has disavowed the benchmark’s usefulness for evaluating anything designed in the past 10 years. The greatest leap beyond Dhrystone has been the Embedded Microprocessor Benchmark Consortium (EEMBC, pronounced “embassy”). Founded in 1997 by Markus Levy, EEMBC currently has 58 member companies, including almost all the biggest names in the business. EEMBC revolutionized microprocessor benchmarking with its strict bylaws, democratic business model, application-oriented test suites, provisions for optimizing benchmark code, and rigorous score-certification process. (See MPR 5/1/00-02, “EEMBC Releases First Benchmarks.”)

However, even EEMBC hasn’t been a panacea. Seven years after the consortium was founded, fewer than half its members have published any benchmark scores for public scrutiny. Most companies prefer to keep their scores private or to share them only with customers, under an NDA. Two relatively new benchmark suites—one for measuring Java performance, another for testing 8/16-bit microcontrollers—have stagnated after garnering a mere handful of published scores, despite years of internal development. Creating new benchmarks and revising existing suites have proved to be arduous tasks, because everything must be decided by committees of member companies that fiercely compete with each other in the marketplace. Some new suites have been under development or delayed for as long as three years.

Recently, a Texas entrepreneur connected with EEMBC decided to fill a different benchmarking niche using a less cumbersome business model. Alan R. Weiss’s Austin-based startup, Synchromesh Computing, has introduced a new benchmark suite for testing x86-compatible processors—specifically, x86 processors suitable for low-end PCs, thin clients, high-end set-top boxes, and Internet appliances. Those applications are at the crossroads between PC processors (which already have a plethora of popular benchmark programs) and embedded processors, which are EEMBC’s traditional territory. Although Synchromesh Computing calls its benchmark suite the Embedded Processor Rating System (EPRS), its focus on system-level tests and higher-end applications differs from EEMBC’s mission.

Synchromesh Computing has stitched together a composite suite by choosing off-the-shelf benchmark programs whose source code is publicly available and by writing some new tests. As a for-profit private company, not an industry consortium, Synchromesh Computing is unimpeded by committee meetings and a multivendor board of directors. Weiss’s connection to EEMBC is that he also owns EEMBC Certification Laboratories (ECL), EEMBC’s exclusive lab for certifying EEMBC benchmark scores. However, EEMBC and Synchromesh Computing are completely separate legal entities.

Weiss’s first client for EPRS benchmarking was AMD, which funded development of the new suite. The lab’s tests of AMD’s x86-compatible Geode processors have provoked howls of protest from VIA, a rival x86 vendor chasing the same markets that AMD does. Controversy seems inseparable from the science and art of benchmarking, often derided as “benchmarketing.”

Even Microprocessor Report isn’t immune from the politics. In the interest of full disclosure, note that both Alan Weiss and EEMBC president Markus Levy are members of the MPR editorial board. Furthermore, the author of this article, when employed at ARC International, was a voting representative on the EEMBC board of directors.

Microprocessor Report readers can access the full story (14 pages, 4 graphics) here: www.mdronline.com/mpr/h/2004/0830/183501.html. To find out more about Microprocessor Report, please visit: www.mdronline.com.

How Is Intel Really Doing?
Jim McGregor -    {08/23/2004}

At the beginning of 2004, Intel looked to rebound from a three-year industry and economic downturn. Intel began the year with production on a new 90nm manufacturing process, using 300mm wafers, combined with the introduction of the Prescott and Dothan processors for desktop and mobile computing, respectively. In April, Intel kept the momentum going with presentations to the entire analyst community that highlighted improving conditions for flash memory and the first glimpse at the 1.7 billion-transistor Montecito processor—the next-generation dual-core Itanium processor for high-end servers. For the grand finale, Intel introduced two new chipsets, Grantsdale and Alderwood, the start of a new generation of computing using PCI Express.

Despite these accomplishments, Intel has been plagued by a number of production and product miscues, leading to slips in product introductions, changes in product roadmaps, and adjustments in future strategies. The problems are highlighted in a companywide email from CEO Craig Barrett that indicates frustration with the company’s execution on commitments. In addition, AMD has gained market share and made significant strides in capturing new opportunities with its Opteron processor. Despite the problems, however, Intel is on track to extend its manufacturing leadership in the semiconductor industry and achieve record revenue and profit levels.

Microprocessor Report readers can access the full story (4+ pages, 1 graphic) here: www.mdronline.com/mpr/h/2004/0823/183401.html. To find out more about Microprocessor Report, please visit: www.mdronline.com.

Most Recent Processor Watch Articles

Past Processor Watch Articles