Wednesday, 10 August 2016

Census fail

The census in Australia is held every 5 years, this year the Australian Bureau of Statistics first attempted to take a population census online. It ended up with the website crashing and it is a major IT failure. We the tax payers paid over $10 million for this disaster!

http://www.itnews.com.au/news/ibm-wins-96m-to-host-ecensus-in-2016-397613




Contracts clearly for load testing of Census 2016
CN IDAmountDescription
CN3356295$54,367.50Load Testing Services for Census 2016
CN3344945$90,000.00Load Testing Services for Census 2016
CN3344757$82,225.00Software Licences to support Laod Testing services for Census IT...
CN3315542$34,650.00Load Testing Script Development Services for CENSUS
CN3312448$325,000.00Licenses for Census Load testing
Sub-total:$586,242.00
Other contracts
CN IDAmountDescription
CN3368531$78,650.00Testing Services
CN3340072$56,760.00Common Authoring Tool (CAT) Performance Testing
CN3332236$65,115.00Agricultural Census Load and Performance Testing
CN3327530$17,160.00testing strategy for the Common Authoring Tool (CAT) Extended for 2...
CN3326273$86,302.00Load and performance testing on ActiveVOS
CN3325298$51,975.002016 Census Script Development
CN3323467$33,000.00Undertake an evaluation of search engines for the MRR
CN3322301$127,710.00Senior Performance Test Engineer
CN3322295$25,300.00Compatibility Testing Services
CN3312447$40,000.00performance testing of recruitment systems
Sub-total:$581,972.00

Grand Total: $1,168,214.00

It has been a disaster, now on the second day Census site census.abs.gov.au still not working:


http://www.theregister.co.uk/2016/08/10/a_failure_of_planning_dumb_ddos_kneecaps_australian_census/


Looks like IBM's Softlayer is not coping well:
|19:31:16|zorang@centos6:[tmp]> wget http://census.abs.gov.au
 <head>
      <meta charset="UTF-8">
      <meta name="robots" content="noarchive, nofollow, noindex" />
      <meta name="applicationVersion" content="5.48" />
      <meta name="viewport" content="width=device-width, initial-scale=1" />
      <meta name="msapplication-config" content="none"/>
      <meta name="format-detection" content="telephone=no" />
      <link rel="icon" href="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/favicon.ico" type="image/x-icon" />
      <title>Census 2016 home page - Australian Bureau of Statistics</title>
      <link rel="stylesheet" href="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/ss/be.css" type="text/css" />
      <script type="text/javascript" src="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/js/be.js"></script>
   </head>
   <body onload="EC.welcome.init({absHttpUrl:'http://help.census.abs.gov.au/',absHttpsUrl:'https://196d9.https.cdn.softlayer.net/80196D9/content.census.abs.gov.au/',vipUrl:'https://stream10.census.abs.gov.au',vipAffinityPort:':8443',primaryUrl:'https://stream11.census.abs.gov.au',ajaxReqTimeout:'20000',ajaxAnimDelayTime:'2000',ajaxPingTestTimeout:'5000'});">
      <div class="WC">
         <div class="headerContainer">
            <div class="header">
               <div id="stlContainer" tabindex="0"><a href="#content" class="skipTo" tabindex="0">Skip to content</a></div>
               <div class="logoContainer"><a href="http://www.abs.gov.au"><img src="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/images/logo.png" alt="Australian Bureau of Statistics logo"></a></div>
               <div class="menuContainer"><button id="btnMenu">Menu<span class="sr"> - expand</span><span class="sr"> - collapse</span></button></div>
               <div id="searchContainer">
                  <form action="//abs-search.clients.funnelback.com/s/search.html" method="get" autocomplete="off">
                     <div><span><input type="text" name="query" id="query" title="Search"></span><input type="hidden" name="collection" value="abs-census"><button id="btnSearch" type="submit"><img id="btnSearchImg" src="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/images/magnifyingglass.png" alt="Search" /></button></div>
                  </form>
               </div>
            </div>
         </div>
         <div id="topLinksContainer">
            <div id="topLinks" class="topLinks mobile">
               <span class="triangle"></span>
               <ul id="tl_m">
                  <li><a id="tlHome_m" tabindex="0" href="#" onclick="window.scrollTo(0, 0);">Census home<span class="triangle"></span></a></li>
                  <li><a id="tlAbout_m" tabindex="0" href="http://help.census.abs.gov.au/about">About the Census<span class="triangle"></span></a></li>
                  <li><a id="tlNews_m" tabindex="0" href="http://help.census.abs.gov.au/news">News &amp; media<span class="triangle"></span></a></li>
                  <li><a id="tlHelp_m" tabindex="0" href="http://help.census.abs.gov.au/help">Help<span class="triangle"></span></a></li>
               </ul>
            </div>
            <div id="topLinks_d" class="topLinks desktop">
               <span class="triangle"></span>
               <ul>
                  <li><button id="tlHome" data-navurl="#">Census home</button></li>
                  <li>
                     <button id="tlAbout">About the Census</button>
                     <div id="foAbout" class="fo about">
                        <div></div>
                        <span class="triangle"></span>
                        <div class="container4">
                           <div class="container3">
                              <div class="container2">
                                 <div class="container1">
                                    <div class="col1">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/about/what">
                                          <h2>What is the Census?</h2>
                                          <p>Find all the latest information about the Census in the one place</p>
                                       </a>
                                    </div>
                                    <div class="col2">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/about/privacy">
                                          <h2>Your privacy</h2>
                                          <p>Find out how we protect the privacy and security of your information</p>
                                       </a>
                                       <hr>
                                       <a tabindex="0" href="http://help.census.abs.gov.au/about/complete">
                                          <h2>Ways to complete the Census</h2>
                                          <p>Find out how to complete the Census, including requesting extra Logins or paper forms</p>
                                       </a>
                                    </div>
                                    <div class="col3">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/about/do">
                                          <h2>Do I have to do the Census?</h2>
                                          <p>Everyone in Australia must complete the Census, wherever they are on Census night</p>
                                       </a>
                                    </div>
                                    <div class="col4"><a class="fo_img" tabindex="0" href="http://help.census.abs.gov.au/about/complete/understanding"><img alt="About the Census information image" src="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/images/fo_atc.png"><span>Need help understanding your Census materials?</span></a></div>
                                 </div>
                              </div>
                           </div>
                        </div>
                     </div>
                  </li>
                  <li>
                     <button id="tlNews">News &amp; media</button>
                     <div id="foNews" class="fo news">
                        <div></div>
                        <span class="triangle"></span>
                        <div class="container4">
                           <div class="container3">
                              <div class="container2">
                                 <div class="container1">
                                    <div class="col1">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/news/news">
                                          <h2>News</h2>
                                          <p>All the latest Census news</p>
                                       </a>
                                       <hr>
                                       <a tabindex="0" href="http://help.census.abs.gov.au/news/media">
                                          <h2>Media</h2>
                                          <p>Media releases and enquiries</p>
                                       </a>
                                    </div>
                                    <div class="col2"><a class="fo_img" tabindex="0" href="http://help.census.abs.gov.au/news/campaign/special"><img alt="News &amp; media information image" src="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/images/fo_nm.png"><span>See how we're reaching everyone in Australia</span></a></div>
                                    <div class="col3">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/news/campaign">
                                          <h2>Campaign materials</h2>
                                          <p>Useful resources including help in your language, information for Aboriginal and Torres Strait Islanders, and more</p>
                                       </a>
                                    </div>
                                    <div class="col4">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/news/interactive">
                                          <h2>Interactive</h2>
                                          <p>Join in the fun and find out what Census can tell you about Australia's people and their housing</p>
                                       </a>
                                    </div>
                                 </div>
                              </div>
                           </div>
                        </div>
                     </div>
                  </li>
                  <li>
                     <button id="tlHelp">Help</button>
                     <div id="foHelp" class="fo help">
                        <div></div>
                        <span class="triangle"></span>
                        <div class="container4">
                           <div class="container3">
                              <div class="container2">
                                 <div class="container1">
                                    <div class="col1"><a class="fo_img" tabindex="0" href="http://help.census.abs.gov.au/help/information/general"><img alt="Help information image" src="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/images/fo_h.png"><span>Need help? Check our help pages</span></a></div>
                                    <div class="col2">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/help/popquestions">
                                          <h2>Popular questions</h2>
                                          <p>Find answers to all your popular questions</p>
                                          <hr>
                                       </a>
                                       <a tabindex="0" href="https://stream10.census.abs.gov.au/eCensusWeb/contact.jsp">
                                          <h2>Contact us</h2>
                                          <p>Can't find the information you're looking for? Get in touch with us</p>
                                       </a>
                                    </div>
                                    <div class="col3">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/help/information">
                                          <h2>Helpful information</h2>
                                          <p>Help information on the Census process, filling out your form and connecting with the Census</p>
                                       </a>
                                    </div>
                                    <div class="col4">
                                       <a tabindex="0" href="http://help.census.abs.gov.au/help/answer">
                                          <h2>How to answer questions on your form</h2>
                                          <p>Find out how to answer Census questions correctly</p>
                                       </a>
                                    </div>
                                 </div>
                              </div>
                           </div>
                        </div>
                     </div>
                  </li>
               </ul>
            </div>
         </div>
         <div id="content">
            <div id="lowerContent">
               <div class="validation">
                  <div class="errors" id="pageLevelErrors">
                     <div><img src="https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/images/icon_error.png" alt="Error Icon" /></div>
                     <div>
                        <ul></ul>
                     </div>
                  </div>
               </div>
               <div class="banner">
                  <div id="dynamicContent1Container">
                     <h1 id="dynamicContent1Title">Census of Population and Housing</h1>
                     <p id="dynamicContent1">Tuesday, 9 August 2016</p>
                  </div>
               </div>
               <div id="dynamicContent2Container">
                  <div id="startContentContainer">
                     <div id="startContentCol">
                        <form id="welcomeForm" name="welcomeForm" action="https://stream10.census.abs.gov.au/eCensusWeb/login.jsp" method="post" autocomplete="off" target="eCensusKiosk">
                           <div id="buttonContainer">
                              <input type="submit" id="btnStart" value="Complete my Census" name="12" title="Opens in new window" />
                              <div></div>
                           </div>
                           <input type="hidden" name="vf" id="vf" value="A" /><input type="hidden" name="pcsf" id="pcsf" value="false" /><input type="hidden" id="vipUrl" name="vipUrl" value="https://stream10.census.abs.gov.au" /><input type="hidden" id="primaryStreamNumber" name="primaryStreamNumber" value="1" /><input type="hidden" id="primaryServerNumber" name="primaryServerNumber" value="1" /><input type="hidden" id="primaryUrl" name="primaryUrl" value="https://stream11.census.abs.gov.au" /><input type="hidden" id="currentUrl" name="currentUrl" />
                        </form>
                     </div>
                     <div id="dynamicContent2Col">
                        <h2 id="dynamicContent2Title">Help shape Australia's future</h2>
                        <p id="dynamicContent2">The Census provides a snapshot of Australia's people and their housing. It helps estimate Australia's population, which is used to distribute government funds and plan services for your community.</p>
                     </div>
                  </div>
               </div>
            </div>
         </div>
      </div>
      <div id="footer" class="bkd">
         <div id="bottom_links">
            <ul id="bottom_links_options">
               <li><a tabindex="0" href="http://help.census.abs.gov.au/privacy">Privacy</a></li>
               <li><a tabindex="0" href="http://help.census.abs.gov.au/security">Security</a></li>
               <li><a tabindex="0" href="http://help.census.abs.gov.au/disclaimer">Disclaimer</a></li>
               <li><a tabindex="0" href="http://help.census.abs.gov.au/copyright">&#169; Copyright</a></li>
               <li><a tabindex="0" href="https://stream10.census.abs.gov.au/eCensusWeb/contact.jsp">Contact us</a></li>
               <li><a tabindex="0" href="http://help.census.abs.gov.au/sitemap">Sitemap</a></li>
            </ul>
            <ul class="social_media_links">
               <li class="fb"><a tabindex="0" target="_blank" title="Follow us on Facebook (opens in new window)" href="https://www.facebook.com/CensusAustralia"></a></li>
               <li class="tw"><a tabindex="0" target="_blank" title="Follow us on Twitter (opens in new window)" href="https://twitter.com/abscensus"></a></li>
               <li class="in"><a tabindex="0" target="_blank" title="Follow us on Instagram (opens in new window)" href="https://instagram.com/abscensus"></a></li>
               <li class="yt"><a tabindex="0" target="_blank" title="Follow us on YouTube (opens in new window)" href="https://www.youtube.com/user/CensusAustralia"></a></li>
            </ul>
         </div>
      </div>
   </body>
</html>

So lets try downloading the simple stylesheet from Softlayer CDN:
|19:45:42|zorang@centos6:[tmp]> wget https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/ss/be.css
--2016-08-10 19:51:36--  https://18294.ssl.cdn.softlayer.net/8018294/stream00.census.abs.gov.au/eCensusWeb/ss/be.css
Resolving 18294.ssl.cdn.softlayer.net... 192.229.144.93
Connecting to 18294.ssl.cdn.softlayer.net|192.229.144.93|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 52397 (51K) [text/css]
Saving to: âbe.css.4â

100%[===================================================================================================================================================================================================>] 52,397      3.97K/s   in 13s

2016-08-10 19:51:49 (3.97 KB/s) - âbe.css

13 seconds to download a 51K stylesheet from CDN!!?...essentially CDN is not coping or throttling. If the caching layer is slow what can you expect when traffic hits actual webservers? Massive fail for IBM!

See here for more details:
https://www.reddit.com/r/australia/comments/4x02pr/lets_investigate_censusfail
https://dperrysvendsen.wordpress.com/2016/11/27/census-2016-a-comedy-of-errors/

Update 30/0/2018 - KPMG review of ABS census IT systems 'compromised', whistleblower says:
https://www.reddit.com/r/australia/comments/8uqlyc/kpmg_review_of_abs_census_it_systems_compromised/

Update 3/5/2019 - Census 2021 digital services handed to PwC, AWS:
IBM did not even bother submitting a bid
https://www.itnews.com.au/news/census-2021-digital-services-handed-to-aws-pwc-524631

No comments:

Post a Comment