Google cluster architecture
Download as PPTX, PDF11 likes11,444 views
Google's search infrastructure uses a cluster architecture with thousands of machines to handle high query volumes. It employs techniques like PageRank to measure page importance, replicates data for reliability and availability, uses low-cost commodity hardware, and manages power usage. Key systems include the Google File System for storage and MapReduce for distributed processing of large datasets.
Google cluster architecture
- 1. Cluster ArchitectureWeb Search for a Planet and much more….Abhijeet [email protected]
- 2. 2
- 3. Google Query Serving InfrastructureElapsed time: 0.25s, machines involved: 1000s+3
- 4. PageRankPageRankTM is the core technology to measure the importance of a pageGoogle's theory – If page A links to page B # Page B is important # The link text is irrelevant – If many important links point to page A # Links from page A are also important4
- 5. 5
- 6. Key Design PrinciplesSoftware reliabilityUse replication for better request throughput and availability Price/performance beats peak performanceUsing commodity PCs reduces the cost of computation 6
- 7. The Power ProblemHigh density of machines (racks) – High power consumption 400-700 W/ft2 # Typical data center provides 70-150 W/ft2 # Energy costs – Heating # Cooling system costsReducing power – Reduce performance (c/p may not reduce!) – Faster hardware depreciation (cost up!)7
- 8. ParallelismLookup of matching docs in a large index --> many lookups in a set of smaller indexes followed by a merge stepA query stream --> multiple streams (each handled by a cluster)Adding machines to a pool increases serving capacity8
- 9. Hardware Level Consideration Instruction level parallelism does not helpMultiple simple, in-order, short-pipeline coreThread level parallelismMemory system with moderate sized L2 cache is enoughLarge shared-memory machines are not required to boost the performance9
- 10. GFS (Google File System) Design Master manages metadata
- 11. Data transfers happen directly between clients/chunk servers
- 12. Files broken into chunks (typically 64 MB)10
- 13. GFS Usage @ Google• 200+ clusters• Many clusters of 1000s of machines• Pools of 1000s of clients• 4+ PB Filesystems• 40 GB/s read/write load – (in the presence of frequent HW failures)11
- 14. The Machinery12
- 15. Architectural view of the storage hierarchy13
- 16. Clusters through the years“Google” Circa 1997 (google.stanford.edu)Google (circa 1999)14
- 17. Clusters through the yearsGoogle (new data center 2001)Google Data Center (Circa 2000)3 days later15
- 18. Current Design• In-house rack design• PC-class motherboards• Low-end storage and networking hardware• Linux• + in-house software16
- 22. Comparison Between Custom built & High-end Servers 20
- 23. Implications of the Computing EnvironmentStuff Breaks• If you have one server, it may stay up three years (1,000 days)• If you have 10,000 servers, expect to lose ten a day• “Ultra-reliable” hardware doesn’t really help• At large scale, super-fancy reliable hardware still fails, albeit less often – software still needs to be fault-tolerant – commodity machines without fancy hardware give better performance/$• Reliability has to come from the software• Making it easier to write distributed programs21
- 24. Infrastructure for Search SystemsSeveral key pieces of infrastructure: – GFS – MapReduce – BigTable22
- 25. MapReduce• A simple programming model that applies to many large-scale computing problems• Hide messy details in MapReduce runtime library: – automatic parallelization – load balancing – network and disk transfer optimizations – handling of machine failures – robustness – improvements to core library benefit all users of library!23
- 26. Typical problem solved by MapReduce• Read a lot of data• Map: extract something you care about from each record• Shuffle and Sort• Reduce: aggregate, summarize, filter, or transform• Write the results• Outline stays the same, map and reduce change to fit the problem24
- 27. ConclusionsFor a large scale web service system like Google – Design the algorithm which can be easily parallelized – Design the architecture using replication to achieve distributed computing/storage and fault tolerance – Be aware of the power problem which significantly restricts the use of parallelism25
- 28. References1. Luiz André Barroso , Jeffrey Dean , UrsHölzle, Web Search for a Planet: The Google Cluster Architecture, IEEE Micro, v.23 n.2, p.22-28, March 2003 [doi>10.1109/MM.2003.1196112]2. S. Brin and L. Page, “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” Proc. Seventh World Wide Web Conf. (WWW7), International World Wide Web Conference Committee (IW3C2), 1998, pp. 107-117.3. “TPC Benchmark C Full Disclosure Report for IBM eserverxSeries 440 using Microsoft SQL Server 2000 Enterprise Edition and Microsoft Windows .NET Datacenter Server 2003, TPC-C Version 5.0,” http://www.tpc.org/results/FDR/TPCC/ibm.x4408way.c5.fdr.02110801.pdf.4. D. Marr et al., “Hyper-Threading Technology Architecture and Microarchitecture: A Hypertext History,” Intel Technology J., vol. 6, issue 1, Feb. 2002.5. L. Hammond, B. Nayfeh, and K. Olukotun, “A Single-Chip Multiprocessor,” Computer, vol. 30, no. 9, Sept. 1997, pp. 79-85. 6. L.A. Barroso et al., “Piranha: A Scalable Architecture Based on Single-Chip Multiprocessing,” Proc. 27th ACM Int’l Symp. Computer Architecture, ACM Press, 2000, pp. 282-293.7. L.A. Barroso, K. Gharachorloo, and E. Bugnion, “Memory System Characterization of Commercial Workloads,” Proc. 25th ACM Int’l Symp. Computer Architecture, ACM Press, 1998, pp. 3-14. 26