Pierluigi Paganini August 28, 2017

A widespread internet disruption occurred in Japan on Friday, Google error it the main cause of the Internet outage that caused the panic in the country.

Google error it the root cause of a widespread Internet outage in Japan that lasted for about an hour on Friday, August 25. The incident was caused by a BGM route hijack that began at 12:22 PM local Japan time and was fixed by 1:01 PM.

Border Gateway Protocol (BGP) is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the Internet. It interconnects the networks of major Internet service providers that announce which IP addresses are available on their networks.

This is what has happened on Friday when Google wrongly advertised that IP blocks belonging to a Japanese ISPs were found on its network.

BGPMon who analyzed the event confirmed that Google hijacked the traffic NTT Communications Corp., a major ISP who also supports two smaller ISPs named OCN and KDDI Corp. NTT has more than 8 million customers, 7.67 million home users, and 480,000 companies.

“Beginning at 2017-08-25 03:23:34, we detected a possible BGP hijack.

Prefix 210.171.224.0/20, Normally announced by AS7527 Japan Internet Exchange Co., Ltd.

Starting at 2017-08-25 03:23:34, a more specific route (210.171.224.0/24) was announced by ASN 15169 (Google Inc.).

This was detected by 17 BGPMon peers.” reported BGPMon.“

Other ISPs started routing traffic that was destined for Japan to Google’s systems, many services online in Japan went down, including the Nintendo networks.

“A widespread internet disruption hit Japan on Friday, blocking access to banking and train reservation services as well as gaming sites.

The disruption was limited to the networks of NTT Communications Corp. and KDDI Corp.

NTT Communications, which runs the OCN internet service, said the problem occurred because an overseas network service provider that OCN uses suddenly switched internet routes.” reported the Japan Times.

It was chaos in the country, the Internal Affairs and Communications Ministry promptly started an investigation into the incident.

“Nintendo Co. said on its website that some Switch, Wii U and 3DS clients started complaining about poor connectivity at about 12:25 p.m.” continnues the media outlet.

Clients of Resona Bank, Saitama Resona Bank and Kinki Osaka Bank said they had trouble logging in to their accounts, as did customers of online banker Jibun Bank Corp. and Rakuten Securities Inc.

East Japan Railway Co. (JR East) said some users of its Suica mobile payment system complained they weren’t able to use the service, which lets them board trains and buy products using their smartphones.”

Google confirmed the incident and provided further details on it.

“We set wrong information for the network and, as a result, problems occurred. We modified the information to the correct one within eight minutes. We apologize for causing inconvenience and anxieties,” the Google spokesperson told The Asahi Shimbun.

BGPMon published a detailed analysis of the incident, it confirmed that Google hijacked over 135,000 network prefixes, from all over the world, of which over NTT 24,000 prefixes.

“In total we saw over 135,000 prefixes visible via the Google – Verizon path. Widespread outages, particularly in Japan (OCN) were because of the more specifics, causing many networks to reroute traffic toward verizon and Google which likely would have congested that path or perhaps hit some kind of acl, resulting in the outages. Many BGPmon users would have seen an alert similar like the one below, informing them new prefixes were being originated and visible global.” concluded BGPMon.

According to the BGPMon engineer Andree Toonk it’s easy to make configuration mistakes that can lead incidents like this.

“In this case it appears a configuration error or software problem in Google’s network led to inadvertently announcing thousands of prefixes to Verizon, who in turn propagated the leak to many of its peers,” said Toonk.

“Since it is easy to make configurations errors, it clearly is a necessity to have filters on both sides of an EBGP session. In this case it appears Verizon had little or no filters, and accepted most if not all BGP announcements from Google which lead to widespread service disruptions,” Toonk added. “At the minimum Verizon should probably have a maximum-prefix limit on their side and perhaps some as-path filters which would have prevented the wide spread impact.”

Pierluigi Paganini

(Security Affairs – Japan outage, Google hacking)

[adrotate banner=”5″]

[adrotate banner=”13″]