Microsoft to revamp Windows encryption keys in face of Flame malware

Starting next month even updated machines may face issues when browsing websites with SSL certificates that are below the new minimum of 1,024 bits

Starting next month, updated Windows operating systems will reject encryption keys smaller than 1,024 bits, which could cause problems for customer applications accessing websites and email platforms that use the keys.

The cryptographic policy change is part of Microsoft's response to security weaknesses that came to light after Windows Update became an unwitting party to Flame Malware attacks, and affects Windows XP, Windows Server 2003, Windows Server 2003 R2, Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2 operating systems, according to the Windows PKI blog written by Kurt L. Hudson, a senior technical writer for the company.

[ Discover what's new in business applications with InfoWorld's Technology: Applications newsletter. | Get the latest insight on the tech news that matters from InfoWorld's Tech Watch blog. ]

BACKGROUND: Flame's Windows Update hack required world-class cryptanalysis, researchers say
MORE: Price tag for Microsoft piece of Flame malware $1M, researcher says

"To prepare for this update, you should determine whether your organization is currently using keys less than 1,024 bits," Hudson writes. "If it is, then you should take steps to update your cryptographic settings such that keys under 1,024 bits are not in use."

Even with preparation, updated machines may face issues such as error messages when browsing to websites with SSL certificates that are below the minimum 1,024. They may also face problems enrolling for certificates when certificate requests use a 1,024 or less key, the blog says. Installing Active X controls signed with 1,024-bit or less signatures will also fail.

The same is true for installing applications signed with less-than 1,024-bit signatures. The exception is those applications signed before Jan. 1, 2010, which will be allowed by default, the blog says.

The use of cryptographic keys shorter than 1,024 bits makes them too vulnerable to brute-force attacks, Microsoft says, something that is widely recognized and dealt with, but not universally.

The biggest challenge for businesses getting ready for the change will likely be with legacy, in-house applications that interact with Windows platforms, says John Pironti, president of IP Architects and the security track leader for Interop.

Microsoft and many other software vendors can readily update the rules under which they accept certificates, he says. It may not be that easy to alter the rules used by custom applications, and in some cases IT security pros may not recall all the places where smaller key sizes are used. "That box just works and nobody thinks about it," he says. "A lot of cases will be, 'Oh, we forgot,' or 'We don't know how to upgrade that cert."

Dealing with such cases manually will require time and money, he says. In addition to changing settings, some hardware may need to be replaced because larger keys sap more processing power. On maxed-out machines, the added computation could cause unacceptable delay.

Overall, though, the transition should be more of an annoyance than anything else, Pironti says. As certificates issued to businesses expire, they are generally replaced with certs using longer keys, he says, so there might not be so many that remain in use.

There are commercial tools for finding and automatically replacing certificates that are too short, Pironti says. Among them is Director made by Venafi, which contributed to the latest NIST Information Technology Laboratory bulletin on certificate authority compromise and fraudulent certificates.

NIST currently has set a deadline of Dec. 31, 2013 for when entities ought to stop using 1,024-bit RSA and DSA encryption. "However, since such keys are more and more likely to be broken as the 2013 date approaches, the data owner must understand and accept the risk of continuing to use these keys to generate digital signatures," according to a special publication called "Transitions: Recommendation for Transitioning the Use of Cryptographic Algorithms and Key Lengths" published in 2011.

Microsoft is updating its operating systems in the wake of the Flame malware used to spy on networks in Iran. Flame exploited Micrsoft's use of the MD5 hashing algorithm in authenticating Windows Update. Microsoft officially disallowed its use in 2009 but failed to weed it out of its own products, particularly Terminal Server Licensing Service. Researchers figured out how to compromise MD5 using what they call collision attacks to obtain fraudulent certificates that are accepted as real.

Since Flame was publicized, Microsoft has started a campaign not only to shut down use of MD5 but also beef up other areas that have not fallen victim to attackers.

The August update will follow on yesterday's security advisory revoking trust for 28 certificates that fail the company's own recently upgraded security standards for the public key infrastructure underpinning Windows Update.

(Tim Greene covers Microsoft for Network World and writes the Mostly Microsoft blog. Reach him at tgreene@nww.com and follow him on Twitter https://twitter.com/#!/Tim_Greene.)

Read more about wide area network in Network World's Wide Area Network section.

This story, "Microsoft to revamp Windows encryption keys in face of Flame malware" was originally published by Network World.

Copyright © 2012 IDG Communications, Inc.