UPDATE: InternetNZ has this afternoon apologised for a security upgrade gone haywire that took some academic sites offline yesterday, then many apps and services of all kinds, from banking apps to media to some government sites, overnight and this morning.
“We sincerely apologise for the problems it caused New Zealanders who couldn’t access some web applications and sites today and to the industry partners working hard on resolving it,” said the non-profit, which administers NZ web domains.
“We will thoroughly investigate what exactly has happened to prevent it from happening again. We will review our processes and systems to ensure a robust service to the Internet users of Aotearoa. We’ll be engaging with the industry and stakeholders to communicate any potentially impactful updates to the system in the future.”
InternetNZ confirmed the glitch related to a refresh of the Domain Name System Security Extensions (DNSSEC) - a system designed as a safeguard against DNS spoofing, or hackers maliciously redirecting users to a fake website.
“To simplify, think of these keys like the keys to a house. Every so often, for security reasons, you might change your house’s locks and get new keys. Similarly, in DNSSEC, InternetNZ must periodically change keys to maintain the system’s security.”
InternetNZ said it could not tell how many websites affected. It has not specified the exact nature of the foul-up, but said it occurred during what is usually a “standard annual procedure”.
It had fixed the issue at its end at its end, but internet service providers needed to make changes to make sure the changes were implemented and the websites and apps they hosted became accessible to the public again. It was working with ISPs to make sure the fixes flowed through, had not estimated time for all normal service to resume.
EARLIER: An internet glitch rendered banking apps and a number of .co.nz websites inaccessible for some users this morning.
It appears to be related to an attempt by InternetNZ - the non-profit that administers local web domains - to rollout a better system for protecting users from fake versions of websites.
“Our apologies, we’re aware that certain Internet Service providers are encountering issues this morning. This means some of our customers will have issues accessing FastNet Classic and ASB mobile,” ASB posted this morning on its Facebook page.
And after Sheri Ngaha complained on Kiwibank’s Facebook page “Why can’t we get into the app or ring this morning. This is so annoying, I’m needing to transfer money but can’t,” the bank replied: “We’re currently experiencing an issue for some customers when trying to access our App or Internet Banking. Our teams are looking into this at the moment and we hope to have this resolved soon.”
On Twitter, Hamish Mack posted: “NZ sites RNZ, New World shopping online and Kiwibank sites are not working? What the heck??”
And Rebecca McMillan said the outage did not seem to have affected Govt.nz but all NZ apps and websites she used were down.
“Can’t even listen to @radionz because the mobile app is down. Yikes. Time to get a transistor radio.”
A service bulletin from InternetNZ late yesterday noted technical problems that hit .ac.nz (education) addresses yesterday then spread to other local domains from 10.45pm last night. InternetNZ today said all types of local internet addresses were affected. An update at 9.21am this morning said, “The issue will resolve over time”.
Responding to a Herald query on Twitter, cloud computing engineer Simon Lyall said, “InternetNZ was changing the key they use to sign .nz and made a mistake. So DNS [domain name server] queries are getting a certificate error.” In other words, it seems a change designed to boost security went haywire, rendering some sites inaccessible. It seems the change related to a measure to prevent “DNS spoofing” - or maliciously redirecting a user to a fake version of a website.
InternetNZ acknowledged the issue when approached by the Herald. More information is pending.
Mack said at 10.30am that his internet connection was “all good now”.
Stuart Laing posted earlier this morning: “Anybody having problems accessing .co.nz sites.nz sites seem to be ok,” but told the Herald just after 9am that his connections were now “sorted”.
One customer of One NZ (formerly Vodafone NZ) said he had issues accessing multiple .co.nz internet addresses from around 11.30pm last night.
A spokesman for One NZ told the Herald: “We had an issue that occurred in the wee small hours where some customers couldn’t access .nz domains when using a fixed connection, but this has been resolved.”
Major internet service providers and banks have been approached for comment. 2degrees and One referred the Herald to InternetNZ.
So what was the change that InternetNZ was trying to implement?
Technology writer Juha Saarinen explained that “The original domain name system (DNS) - that translates between links like http://nzherald.co.nz and Internet Protocol (IP) address like 104.18.2.137 assigned to network hosts - had no security features.
“This led to major security problems such as DNS ‘cache poisoning’ which meant malicious people were able to redirect users to bogus websites.”
The changes introduced overnight were designed to make it easier to authenticate if a website was the real deal. They involved Domain Name System Security Extensions (DNSSEC).
Some people posted that clearing their web browser’s cache resolved the issue, but Lyall cautioned that might not work if there are problems further up the food chain. It’s better to wait until your internet service provider has followed InternetNZ’s instructions to flush its own cache (temporary storage of websites and related data, designed to speed up loading).
Take your Radio, Podcasts and Music with you
