If you drive down highway 880 from Oakland, California, take an exit about 30 miles south, and snake past a long line of car dealerships, you'll find an ordinary office building that belongs to a company you've never heard of. And if you're allowed to walk inside -- past the receptionist and the cubicles, through another door, around the security guard, and into the warehouse -- you'll find some technicians assembling and testing server hardware for some of the biggest names on the internet.
This includes Facebook and Rackspace and at least one or two other names that even your grandmother knows about it.
The warehouse belongs to a company called Synnex -- an outfit that spent the last 30 years buying and selling computers, hard drives, chips, memory, and all sorts of other hardware. But Synnex isn't the one assembling and testing all that internet server hardware -- at least not officially. Those technicians work for a brand new Synnex division called Hyve.
Hyve Solutions was created to serve the world's "large-scale internet companies" -- companies increasingly interested in buying servers designed specifically for their sweeping online operations. Because their internet services are backed by such an enormous number of servers, these companies are looking to keep the cost and the power consumption of each system to a minimum. They want something a little different from the off-the-shelf machines purchased by the average business. "What we saw was a migration from traditional servers to more custom-built servers," says Hyve senior vice president and general manager Steve Ichinaga. "The trend began several years ago with Google, and most recently, Facebook was added to the ranks of companies who want this kind of solution."
The net's biggest names have caused a tectonic shift in the worldwide server market. These are the companies that need more servers than anyone else on the planet, and they're moving away from traditional server makers such as Dell and HP, embracing Hyve, various manufacturers in Taiwan, and other little known companies that can help them build servers for their particular needs. In response, Dell and HP are now doing custom work as well. But the Hyves of the world are here to stay.
Be Like Google
Hyve doesn't count Google as a customer -- or at least it doesn't seem to. But it's serving many of the internet companies that are imitating Google. Unhappy with the cost and design of traditional servers from the likes of Dell and HP, Google designs its own servers, and it contracts with companies in Taiwan to build them. Facebook has now followed suit. Its no-frills servers are built by Taiwanese "original device manufacturers" (ODMs) Quanta and Wistron, and then they're shipped to Hyve in Fremont, California, where technicians load them into racks, hook up the required networking equipment, test them, and ship them off to Facebook's data centers.
Google treats its latest server designs as trade secrets, but earlier this year, Facebook open sourced its designs under the aegis of the Open Compute Project, sharing them to anyone who wants them. And this led Synnex to create Hyve. Hyve is a place where internet giants can go if they want Open Compute servers. But even before Hyve was created, Synnex was working for the big internet names. It has long provided custom machines for Rackspace -- the San Antonio, Texas company that offers infrastructure services across the net as a scale rivaled only by Amazon -- and though Synnex won't identify its other customers, it will say that these are companies everyone knows. "They're household names," Ichinaga says.
Hyve is just one of the under-the-radar server companies feeding these big internet names. SGI -- the company once known as Rackable (not to be confused with Rackspace) -- has spent years building custom servers for the likes of Amazon and Microsoft. And a New Jersey-based outfit known as ZT Systems is building servers for similar internet outfits -- though it won't say who. Hyve and ZT say very little about how their operations work, but both seem to have very close ties to manufacturing outfits in China and Taiwan, where so much of the world's IT hardware is built. This means they can provide custom servers while still keeping prices down.
"We have very long term relationships with the key vendors," says Ichinaga. "We already sell billions of dollars' worth of components and other IT equipment, and that basically allows us to leverage our relationships with our partners when we serve our [internet] customers." Tim Symchych, the director of supply chain operations at Rackspace, confirms that his company can get lower prices from Hyve than he can from Dell and HP (though the company continues to buy from Dell and HP as well). Jason Hoffman -- the chief technology officier of Amazon- and Rackspace-rival Joyent -- says there will be cases when his company can actually get a better price from a traditional server maker such as Dell. But the point is the Hyves and ZT Systems of the world are competing with the big boys, and in many cases, they're winning.
With Facebook sharing its designs through the Open Compute Project this trend will only continue. According to Ichinaga, Hyve has already received orders for Open Compute servers from multiple companies. And that doesn't include Facebook.
The Hyve Mind
Joyent CTO Jason Hoffman visited Hyve one afternoon in early December. Joyent typically buys its servers from Dell or Sun Microsystems (now part of Oracle), but he's exploring other options. He spent an hour discussing Hyve's services with a company sales rep, and then he took a brief tour of the warehouse. We tagged along, and though Hyve was careful not to expose its other customers, it did show off some of the Open Compute server racks it's putting together for Facebook.
Hoffman and Joyent don't want Open Compute servers. Facebook's designs are meant for web serving and memcaching -- a way of storing caching data in server memory, so it can be quickly accessed -- and he's looking for something more robust. "Facebook is really just running one applications," he says. "We're supporting different applications for all our customers."
But Hyve says it can give him what he wants. Though Facebook's servers are built in Taiwan and shipped to Hyve's warehouse in Fremont, Hyve tells us that in most cases, it builds servers on its own, pulling parts from partners across the globe. Hyve says it can work with a company like Joyent to design a server that suits its particular needs. "It starts with collaboration," Ichinaga says. "We figure out what the customer wants in terms of server workload and the physical environment and everything else. If you need something different, we can do it."
Ichinaga won't go into detail about Hyve's business model. But he says the company is only able to do this because Synnex has spent thirty years distributing computer hardware to thousands of resellers across the globe. It has close relationships, Ichinaga says, with the likes of chip giant Intel and hard drive maker Seagate, as well as ODMs such as Quanta and Wistron. "Compared to a traditional OEM companies," he says, referring to original equipment manufacturers such as Dell and HP. "We have a very low S,G, and A [selling, general and administrative expenses]. It's just a very efficient model."
Synnex is a US company. Its Fremont, California, offices are the official headquarters. But the company's founder, Robert Huang, was born in Taiwan and studied electrical engineering in Japan. And its ties to Asia run deep. The company has offices in both China and Japan.
Somewhere in New Jersey
ZT Systems seems to operate in much the same way. And it too builds servers for some of the biggest names on the net.
It would appear that these names include Amazon. Amazon job listings seeking technicians for its data centers sometimes request engineers that have "hands-on experience with Hewlett-Packard, Dell, Rackable, or ZT Systems." It's no secret that Rackable builds for Amazon. As a public company, Rackable (now SGI) must name customers that account for more than 10 percent of its revenues -- and Amazon does. But Amazon's relationship with ZT is still very much under the radar.
Amazon has not responded to requests to discuss ZT Systems, and ZT declines to name any of its customers. But a company spokesman will say that like Hyve, it builds custom servers for "very large data center operators." And he says it runs sales offices in Fremont, California, near Hyve, and in Seattle, Washington. Amazon is headquartered in Seattle. AMD -- Intel's chief chip rival -- has sold chips to ZT Systems for at least six years, according to John Fruehe, director of product marketing for server and embedded products at AMD and former business development man at the chip maker. Fruehe won't name ZT's customers, but he confirms that it serves "mega-data-center customers."
Like Hyve, ZT is a US company, and according to a company spokesman, it builds all its servers in New Jersey. But it too has close ties to Asia. That said, ZT insists it shouldn't be compared to Hyve, because it has a longer history as a full-fledged OEM. "ZT Systems is a full-featured server OEM with robust engineering capabilities, and extensive experience designing and building computers in the USA for over 17 years," says the company spokesman. "Our systems contain the highest quality components, many of which are produced in Asia."
Uh, Who Makes These Things?
The server supply chain is a complicated thing. At times, it's difficult to tell who's actually building the machines and who's merely sprucing them up. Traditionally, ODMs such as Quanta and Wistron build the systems. Then OEMS such as Dell and HP add some additional hardware and ensure the systems meet certain standards. And these OEMs work with VARs -- value added resellers -- who will sell gear to the end user and may add some extra, well, value.
But the lines between these various layers are rather blurry, and a company like Hyve blurs them further still. Synnex sells hardware to resellers and distributes hardware on behalf of OEMs like HP, but now, with the rise of Hyve, it's turning into a server maker. With Facebook's servers, Hyve isn't actually building the systems. But with other customers -- such as Rackspace -- it says that it is.
Even Jason Waxman -- general manager of high-density computing in Intel’s data center group and a board member of the Open Compute Project -- finds it hard to classify a company like this. "I'm still trying to understand some of their models," he says. "[With an company like Hyve], I don't know if they allow the end user to take responsibility for the design or if they're really becoming an OEM themselves. OEM is actually building a fully configured system and supporting it."
What's more, there are cases where component companies such as AMD are selling parts directly to the big internet names, and these names will then go to companies like ZT Systems or the Taiwanese ODMs and ask them to actually put these components into servers. Google buys chips directly, and at least some others have followed. "This was all the rage about two years ago," says AMD's Fruehe. "There was one extremely large web company that was doing this, and they still to this day build their own systems. Everybody looked at this and said 'This is great. I've been going to an OEM to have them build my systems, but I can really cut down my costs if I do it myself and go straight to Taiwan."
Intel sells chips in this way as well. But Jason Hoffman says this is not the norm. "Our goal is that we almost exclusively work with OEMs," he says.
As AMD's Fruehe points out, there are still reasons for even the biggest internet companies to lean on OEMs like Dell and HP and ZT. "The OEM is providing the system integration, the tests, the qualifications, all the low-level systems code," he says. "You start to realize this is a whole lot of work, and if you build your own servers, you have to hire a whole lot of people do what you could have hired HP or Dell or ZT or another partners to do."
In addition to building the systems, ZT will help support them. The company runs "integration facilities" around the world, which are essentially support offices that serve the data centers of the big internet players. In some cases, says a company spokesman, ZT will have someone working inside a customer's data center. This is quite different from the way Hyze works with Facebook. Facebook does get on-site support from the company, but it does use Hyve to put its servers into racks, hook them up, test them, and deliver them.
The big internet names are approaching the task in many different ways. And generally, each is dealing with multiple server vendors. Rackspace uses Dell, HP, and Hyve. Facebook is moving to Quanta, Wistron, and Hyze, but it continues to use Dell and HP. According to job listings, Amazon is using HP, Dell, Rackable, and ZT, and it may be using other companies as well.
Beyond the Net
None of this is new. But in open sourcing its new server designs, Facebook is pushing the market in a new direction. It's now easier for server buyers to go directly to Taiwan. Or if they want a bit more hand-holding, they can go to Hyve. Facebook recently released version 2.0 of its Open Compute server designs, and in the new year, Hyve will be shipping servers based on these designs not only to Facebook but to other big name internet outfits.
Both Amazon and Apple attended Facebook recent Open Compute summit in New York. "There's folks from Google and Apple and other folks here today as well," Facebook's Frank Frankovsky told Wired. "Even though we didn't highlight the contributions from those companies, everybody's included. And people can just consume the technology and not contribute back."
Sun founder Andy Bechtolsheim -- another Open Compute Project board member -- believes that Apple will at least consider Open Compute servers. "Apple wants to build a big iCloud," he told us. "Obviously, they want to minimize their power consumption and cost. I'm pretty sure they will look at this. The point is they couldn't have looked at this until it became an open spec. Their other choice was to build their own version."
And the internet giants aren't the only ones. According to Ichinaga, all sorts of other outfits -- government organizations, telecoms, and large financial organizations -- have ordered Open Compute servers. Chris Kemp -- the CEO of Nebula, a startup that's selling appliances that run OpenStack, an open source platform that mimics Amazon's web services -- has been involved with the Open Compute Project from the beginning, and he echoes Ichinaga.
"Every company in the world is going to take a close look at what they’re all about," he says. "If you’re a financial or biotech company, you’re gonna to start looking at the cost of your infrastructure, and you’re going to start behaving a lot like Google and Amazon and Facebook. In today's world, everybody’s business is becoming computationally intensive."
In other words, he says, the business world will move even further away from the Dells and the HPs.
Additional reporting by Eric Smalley.
