Tuesday, 15 March 2016

Dropbox moved 90% of its files from Amazon's S3 to data centers designed by its own engineers over two-and-a-half years

In a blog post on Monday, Akhil Gupta, Dropbox's VP of engineering, said his company has been an early adopter of Amazon S3 to store bulk data – the firm has always kept metadata and web services in-house – but was now handling 90 per cent of its own storage needs. They called this system Magic Pocket and Dropbox moved 500PB into this storage system.

In essence, they built their own Amazon S3—except they tailored their software to their own particular technical problems. “We haven’t built a like-for-like replacement,” Agarwal says. “We’ve built something that is customized for us.”

S3 compatible software:
http://basho.com/products/riak-s2/ (commercial/open source)
http://www.skylable.com/products/sx/ (commercial/open source)
https://www.swiftstack.com/product (commercial/open source)
http://www8.hp.com/us/en/cloud/helion-eucalyptus-support.html (commercial/open source)
https://minio.io/ (open source)

The chassis looks similar to Supermicro SC946ED-R2KJBOD with 90 x 3.5" disks: 
http://www.supermicro.com/products/chassis/4U/946/SC946ED-R2KJBOD.cfm
http://www.wiredzone.com/supermicro-racks-kvm-chassis-power-server-chassis-4u-rackmount-cse-946ed-r2kjbod-10024922
 
Hence 90 x 6TB = 540TB for $33,606.91 or $0.062/GB...now assuming 5 year deployment this equates to $0.00103/GB/month. Yet this does not include data centre facilities, redundancy, networking, management...lets be very conservative and multiply by 10 hence $0.0103/GB/month to include these costs. Still significantly lower cost than AWS S3 at $0.0275/GB/month (US East when using over 5000TB/month).

Important to note that Dropbox did not choose storage from expensive vendors vendors like EMC, HDS, IBM, HP or Netapp. Very hard to compete with storage costs based on commodity hardware and open source.


Update 23/2/2018 - Dropbox saved almost $75 million over two years by building its own infrastructure:

No comments:

Post a Comment