Today marks the launch of Amazon EBS (Elastic Block Store), the long awaited persistent storage service for EC2. Details can be found on the EC2 detail page and the press release.
With the launch of the Elastic Block Store we complete an important milestone in offering a complete suite of storage solutions as part of the Amazon Infrastructure Services. Back in the days when we made the architectural decision to virtualize the internal Amazon infrastructure one of the first steps we took was a deep analysis of the way that storage was used by the internal Amazon services. We had to make sure that the infrastructure storage solutions we were going to develop would be highly effective for developers by addressing the most common patterns first. That analysis led us to three top patterns:
1. Key-Value storage. The majority of the Amazon storage patterns were based on primary key access leading to single value or object. This pattern led to the development of Amazon S3.
2. Simple Structured Data storage. A second large category of storage patterns were satisfied by access to simple query interface into structured datasets. Fast indexing allows high-speed lookups over large dataset. This pattern led to the development of Amazon SimpleDB. A common pattern we see is that secondary keys to objects stored in Amazon S3 are stored in SimpleDB, where lookups result in sets of S3 (primary) keys.
3. Block storage. The remaining bucket holds a variety of storage patterns ranging special file systems such as ZFS to applications managing their own block storage (e.g. cache servers) to relational databases. This category is served by Amazon EBS which provides the fundamental building block for implementing a variety of storage patterns.
I have written before about the basic features of Amazon EBS:
- Amazon EBS will be offered in the form of storage volumes which you can mount into your EC2 instance as a raw block storage device. It basically looks like an unformatted hard disk. Once you have the volume mounted for the first time you can format it with any file system you want or if you have advanced applications such as high-end database engines, you could use it directly.
- Developers can create multiple volumes, in size ranging from 1 GB to 1TB. This volume will be created within a specified Availability Zone and will be accessible by your EC2 instances running in that Availability Zone. As to be expected with a volume abstraction only one instance can have the volume mounted at any given time. Volumes can migrate and be reattached to other instances if necessary for failure handling or application migration reasons.
- The consistency of data written to this device is similar to that of other local and network-attached devices; it is under control of the developer when and how to force flush data to disk if you want to bypass the traditional lazy-writer functionality in the operating systems file-cache. Because of the session oriented model for access to the volume you do not need to worry about eventual consistency issues.
However Amazon EBS isn’t just a massive volume storage array within an Availability Zone, it provides a unique feature that allows for the creation of novel storage management scenarios: the ability to create snapshots and store those snapshots into Amazon S3. These snapshots can then be used as the starting point for creating new volumes within any availability zone.
We see developers use this feature for long term backup purposes, for use in rollback strategies, for (world-wide) volume re-creation purposes. Snapshots also play an important role in building fault-tolerance scenarios when combined with managing applications using Elastic IP addresses and Availability Zones.
Congratulations to the EBS team for delivering a great service that will help a lot of EC2 customers managing their storage efficiently.
Read the rest of this entry »
Posted in Internet, Software | No Comments »
“Innovation” has been thrown around so often in technology circles that to some, it’s a four-letter word.
At one tech company, innovation can mean bringing a dazzling new product to store shelves. At another, it can translate to a tiny new button on a Web site. That’s why, executives say, the word itself has been overused and devalued.
Still, new cutting-edge products mean everything to a successful tech company.
Executives from eBay, Hewlett-Packard, Microsoft, and others were here at SDForum’s first Corporate Innovation and Research Fair on Friday to talk about their techniques for staying creative. Each company has its own style, with some strategies that overlap. But they all acknowledged it’s not easy to innovate, especially considering that large corporate cultures can be a curse to fresh ideas.
Max Mancini, eBay’s senior director of Platform and Disruptive Innovation, went so far as to say that Silicon Valley venture capitalists wouldn’t make so much money on start-up investments if tech companies were better at developing new products.
“Venture capital firms thrive on inefficiencies in large organizations,” said Mancini, who spoke at the gathering held at the Computer History Museum.
His counterpart at HP added to the idea by saying that demands from Wall Street and senior management can stifle innovation. “If you’re a larger company, there’s high probability you have creative people (in your organization). But creative people get impatient,” said Rich Friedrich, director of HP’s Enterprise Systems and Software Lab.
That means that these companies either must invest billions in research and development units, or bake in policies to ensure that people dream up new products. Google, of course, asks engineers to spend 20 percent of their time on pet projects. Microsoft, in contrast, employs more than 800 researchers in labs around the world.
A bottom-up style
Roy Levin, Microsoft’s director of research in Silicon Valley, said that one reason the labs have proven helpful to Microsoft, including bringing products like Windows Media to consumers, is their bottom-up style. The labs’ researchers pick projects themselves and collaborate with each other. They’re also not beholden to profit-and-loss goals or managers, he said.
“Every time you introduce (managerial) hierarchy, you introduce barriers to collaboration; and collaboration is key,” Levin said.
But once a technology is ready, transferring it to a product group or bringing it to market can be highly difficult, he said. That’s why so-called technology transfers are “a contact sport,” he said. Researchers must travel a lot to get new ideas and prototypes in front of the right people, Levin said.
eBay’s Mancini said that the auction company does two big things to promote creativity. The first is operating a technology platform that mirrors the eBay framework so that its engineers can experiment with new tools. That way, developers can test products outside of the company’s rigid software development process, he said.
The other method is to invite third-party developers into the fold through application programming interfaces. He said that in the last year developers have created an estimated 12,000 applications for eBay, producing as many as 60 percent of the listings on the site. “That’s innovation we probably couldn’t afford,” he said.
“Innovation is about the ecosystem, either removing barriers internally or allowing third parties to help meet the needs of your customers in ways you can’t afford to do (or have the time to do),” Mancini said.
Similarly, HP’s Friedrich said that one of his company’s strategies is to partner with outsiders on projects. “All of the innovative people don’t work for your company,” he said.
HP, for example, teamed up with DreamWorks years ago to work on technology for life-like animation and “cloud” services that were used to produce the movie Shrek. Last week, HP also teamed up with Intel and Yahoo to create six large-scale computing centers that would allow outsiders to test technology.
Cloud services are one of several areas of research for HP, which invests about $3.6 billion annually in R&D, Friedrich said. It’s also looking at projects in sustainability and managing data. On a broader level, HP is trying to shift the company from a hardware maker to a software company; and it’s doing that largely through acquisitions.
Oracle’s Marie-Anne Neimat, vice president of development for embedded databases, also pointed to acquisitions as a way to evolve, beyond Oracle’s multibillion dollar annual investment in R&D.
“It’s new blood,” she said.
Finally, some technology companies have turned into venture capitalists, too.
Ike Nassi, SAP’s executive vice president of research for the Americas and China, said it recently started a venture capital incubator. It solicits ideas from internal employees and external start-ups; and if it’s a good idea, SAP will help form a new business unit, fold the start-up into an existing product line, or spin it out as a new company, he said.
“If you have an interesting idea and don’t want to go the VC route, we provide seed funding,” Nassi said.
That’s similar to other technology companies. Intel, Google, Motorola, Amazon, and Comcast run venture capital units either formally or informally.
What about the word innovation?
“It’s completely devalued,” Nassi said. “The thing we need to look at is managing risk–whether placing an investment on this versus that, and what’s the payoff of that investment.”
Read the rest of this entry »
Posted in General, Ideas, Internet | No Comments »
Despite a recent high-profile vulnerability that showed the net could be hacked in minutes, the domain name system — a key internet infrastructure — continues to suffer from a serious security weakness, thanks to bureaucratic inertia at the U.S. government agency in charge, security experts say.
If the complicated politics of internet governance continue to get in the way of upgrading the security of the net’s core technology, the internet could turn into a carnival house of mirrors, where no URL or e-mail address could be trusted to be genuine, according to Bill Woodcock, research director at the nonprofit Packet Clearing House.
“The National Telecommunications and Information Administration, an agency of the Department of Commerce, is the show-stopper here,” Woodcock said.
At issue is the trustworthiness of the domain name system, or DNS, which serves as the internet’s phone book, translating queries such as wikipedia.org into the numeric IP address where the site’s server lives.
Just weeks ago, security researcher Dan Kaminsky announced he’d discovered a way for hackers to feed fake info into DNS listings, which would allow hackers to redirect web traffic at will — for example, routing every person attempting to log in to the Bank of America to a fake site controlled by the attacker.
Kaminsky quietly worked with large tech companies to build patches for the net’s name servers to make the attack more difficult. But security experts, and even the NTIA, say those patches are just temporary fixes; the only known complete fix is DNSSEC — a set of security extensions for name servers.
Those extensions cryptographically sign DNS records, ensuring their authenticity like a wax seal on an letter. The push for DNSSEC has been ramping up over the last few years, with four regions — including Sweden (.SE) and Puerto Rico (.PR) — already securing their own domains with DNSSEC. Four of the largest top-level domains — .org, .gov, .uk and .mil, are not far behind.
But because DNS servers work in a giant hierarchy, deploying DNSSEC successfully also requires having someone trustworthy sign the so-called “root file” with a public-private key. Otherwise, an attacker can undermine the entire system at the root level, like cutting down a tree at the trunk. That’s where the politics comes in. The DNS root is controlled by the Commerce Department’s NTIA, which thus far has refused to implement DNSSEC.
The NTIA brokers the contracts that divide the governance and top-level operations of the internet between the nonprofit ICANN and the for-profit VeriSign, which also runs the .com domain.
“They’re the only department of the government that isn’t on board with securing the Domain Name System, and unfortunately, they’re also the ones who Commerce deputized to oversee ICANN,” Woodcock said.
“The biggest difference is that once the root is signed and the public key is out, it will be put in every operating system and will be on all CDs from Apple, Microsoft, SUSE, Freebsd, etc,” says Russ Mundy, principal networking scientist at Sparta, Inc, which has been developing open-source DNSSEC tools for years with government funding, He says the top-level key is “the only one you have to have, to go down the tree.”
A European networking group known as RIPE called in June 2007 for the root to be signed, with Swedish and British representatives echoing the call in October. But NTIA is not moving quickly enough to sign the root, given the looming threat, even after the final technical problems have been resolved, according to Woodcock and others.
“A few years ago, there were still technical hurdles to actually signing and using DNSSEC, but in the past few years, a lot of software tools, both commercial and open-source, have come out, and now it’s a completely solved problem,” Woodcock said. “All that’s left is the far less tractable, purely political problem.”
“Arguing over who gets to hold the cryptographic keys in the long run [should] wait until we’re not facing a critical threat,” Woodcock said.
But the NTIA insists it is moving at just the right pace.
“We are committed to taking no action that would have the potential to adversely affect the operational stability of the DNS,” says spokesman Bart Forbes. “While there is increasing pressure to secure the DNS, NTIA must work with all stakeholders and consider all possible solutions.”
Olaf Kolkman, a Dutch networking export, says there’s no time to waste. The only way for DNSSEC to work is for the top-level zone file — which lists the specifics for top-level domains like .gov — to be signed by a trusted authority.
“Currently DNSSEC is the only mechanism known to protect against the Kaminsky attack,” Kolkman said. “It is not clear that other solutions will provide the same level of protection as DNSSEC.”
Without such extensions, a hacker eager for trade secrets could hijack the DNS listing for Apple’s e-mail server and insert the number for a server he controls instead. He could then keep a copy of every message sent to the company and forward them all. No one would likely to be any wiser until a human looked closely at the mail headers.
Still, even DNSSEC’s most fervent backers admit that signing the root won’t instantly secure the net. Installing the extensions internet-wide will be costly and time-intensive, but proponents say that getting the root signed will turbocharge the process.
The Internet Assigned Numbers Authority — which coordinates the internet — has been prototyping a system to sign the root-zone file for the last year, but they can’t do the same for the internet’s top servers without approval from the Department of Commerce.
That’s where the rub is, according to Kolkman.
“Then the issue becomes political because there seems to be the perception that the introduction of a key guardian changes the current policies,” Kolkman said
That could also simplify how top-level zone files are created, according to Richard Lamb, a technical expert at IANA. Currently companies that manage top-level domains like .com submit changes to ICANN, which then sends them to NTIA for approval, before they’re forwarded to VeriSign. VeriSign actually edits the root file and publishes it to the 13 root servers around the world.
“We would want to bring the editing, creation and signing of the root zone file here,” to IANA, Lamb said, noting that VeriSign would likely still control distribution of the file to the root servers, and there would be a public consultation process that the change was right for the net.
But changing that system could be perceived as reducing U.S. control over the net — a touchy geopolitical issue. ICANN is often considered by Washington politicians to be akin to the United Nations, and its push to control the root-zone file could push the U.S. to give more control to VeriSign, experts say.
VeriSign did not respond to a request for comment, but its CTO said earlier this year that it was creating its own root-zone file-signing test bed.
The root-zone file, which contains entries for the 300 or so top-level domains such as .gov and .com, changes almost every day, but the number of changes to the file will likely increase radically in the near future, since ICANN decided in June to allow an explosion of new top-level domain names.
Woodcock isn’t buying the assurances of NTIA that it is simply moving deliberatively.
“If the root isn’t signed, then no amount of work that responsible individuals and companies do to protect their domains will be effective,” Woodcock said. “You have to follow the chain of signatures down from the root to the top-level domain to the user’s domain. If all three pieces aren’t there, the user isn’t protected.”
Read the rest of this entry »
Posted in Domains, Internet, Security | 1 Comment »
