CS5102 High Performance Computer Systems Warehouse-Scale ...

CS5102 High Performance Computer Systems Warehouse-Scale ...

CS5102 High Performance Computer Systems Warehouse-Scale Computers Prof. Chung-Ta King Department of Computer Science National Tsing Hua University, Taiwan (Slides are from textbook, Prof. A.W. Moore, Prof. J. Urbain, Dr. A.E. Evrard) National Tsing Hua University Main Reference The Datacenter as a Computer: An Introduction to the Design of WarehouseScale Machines, 2nd Ed., Luiz Andr Barroso, Jimmy Clidaras, Urs Hlzle, Morgan

& Claypool Publishers, 2013. National Tsing Hua University 2 Internet Services Becoming Prevalent Internet services: - Webmail, search, social network, maps, video/picture sharing, storage, video streaming, language translation, ... The trend toward server-side or cloud computing - Moving away from PC-like clients to smaller, often mobile, devices combined with large-scaled internet services - Many Internet services were previously resided in the client, but now are moved to the cloud

Characteristics of internet services - Tens or more individual programs that interact to implement complex services to serve a massive amount of end users one service for many National Tsing Hua University 3 What Computers for Internet Services? WSC: warehouse-scale computer - Large-scaled internet services require a computing power that can only be provided by clusters of 100s or 1000s of machines, i.e. a warehouse full of computers - Hardware and software resources in such a facility must work in concert as would

Server and networking equipment a single computer datacenter is the computer Cooling towers (Ana Radovanovic) National Tsing Hua University Generators MW substation

4 Warehouse-Scale Computer (WSC) Program is an Internet service - May consist of tens or more individual programs that interact to implement complex end-user services such as email, search, or maps Software runs on hardware consisting of thousands of individual computing nodes with: - Networking and storage subsystems - Power distribution and conditioning equipment - Extensive cooling systems The enclosure for these systems is in fact a building

structure and often indistinguishable from a large warehouse National Tsing Hua University 5 Google Oregon WSC National Tsing Hua University 6 Warehouse-Scale Computer (WSC) Single facility for a single organization Designed for narrower set of needs, fewer requirements

Use a relatively homogeneous hardware and system software platform, often customized OS & software - Easier to maximize efficiency Share a common systems management layer Internet services must achieve high availability, typically aiming for at least 99.99% uptime (about an hour of downtime per year) National Tsing Hua University 7 Datacenter as a Computer National Tsing Hua University

8 Warehouse-Scale Computer (WSC) Different from HPC clusters (e.g., Top 500): - Clusters have higher performance processors and network - Clusters solve one problem for a small team - Clusters emphasize thread-level parallelism, while WSCs emphasize request-level parallelism Different from traditional datacenters: - Datacenters consolidate different machines and software into one location a co-location facility - Datacenters emphasize virtual machines and hardware heterogeneity in order to serve varied customers - Datacenters are often singly managed for many clients

with specialized needs, different (specific) requirements - Broad scope: multiple, standard OSs and application suites National Tsing Hua University 9 WSC vs Data Center vs HPC Cluster vs Server Homogeneous WSC High latency Low bandwidth High quality Expensive components

Server National Tsing Hua University Heterogeneous Data Center Low latency High bandwidth HPC Cluster 10

Design Factors for WSC Cost-performance (i.e., work done/$) - A WSC infrastructure can cost $150 M, a 10% reduction in capital cost saves $15 M small savings add up Energy efficiency - Power distribution and cooling are critical - Work done per joule has to be maximized due to scaling Dependability via redundancy: - At least 99.99% availability: down less than one hour/year - Multiple servers and WSCs for redundancy Network I/O: inter and intra WSC Interactive (search, social) and batch processing (indexing) workloads

National Tsing Hua University Just like server architecture 11 Design Factors for WSC Ample parallelism: finding parallelism is non-issue - Data-level parallelism: independent processing on data - Software as a Service: millions of independent users request-level parallelism: no coordinate or synchronize Operational costs count - WSCs have longer lifetimes: building, electrical/cooling are amortized over 10 or more years op. costs add up, e.g., energy, power distribution, cooling > 30% cost over 10 yr

Scale and its opportunities and problems - Can afford to build customized systems since WSC require volume purchase Unlike server - Flipside: failures architecture National Tsing Hua University 12 Outline Programming models and workloads for warehousescale computers (Sec. 6.2) Computer architecture of warehouse-scale computers (Sec. 6.3) Physical infrastructure and costs of warehouse-scale

computers (Sec. 6.4) Could computing: the return of utility computing (Sec. 6.5) National Tsing Hua University 13 Characteristics of Internet Services Ample parallelism: - Internet services often have large amount of parallelism stemming from both data- and request-level parallelism The problem is not to find parallelism but to manage and efficiently harness the explicit parallelism - Data parallelism: from the large data sets of relatively

independent records that need substantial processing can hide or tolerate comm. and synch. overheads - Request-level parallelism: from huge amount of requests per second received, which rarely involve readwrite sharing of data or synchronization across requests computation can be easily partitioned both within a request and across different requests National Tsing Hua University 14 Characteristics of Internet Services Workload churn: - Users of Internet services are isolated from the services implementation details by relatively well-defined and stable high-level APIs (e.g., simple URLs)

Easier to deploy new software quickly, e.g., weekly cycle By hundreds of developers making thousands of independent code changes per week - Good for rapid product innovation but hard for system designers to extract useful benchmarks and workload mix potentially affecting the optimal design point of WSCs - Hardware need not strive for good performance for immutable pieces of code, but consider software rewrites to take advantage of new hardware capabilities or devices National Tsing Hua University 15 Characteristics of Internet Services Platform homogeneity:

- Internet services typically require a small number of hardware and system software configurations a more homogeneous target for software development - Homogeneity simplifies cluster-level scheduling and load balancing and reduces maintenance burden for platforms software (kernels, drivers, etc.) - Homogeneity allows more efficient supply chains and repair processes (e.g., having more experience with fewer types of systems) National Tsing Hua University 16 Characteristics of Internet Services Fault-free operation:

- Internet services run on clusters of thousands of machines fault is expected every few hours or less - Internet services need to work in an environment where faults are part of daily life - Ideally, system software should provide a layer that hides most of fault complexity from application-level software National Tsing Hua University 17 Software Layers of WSC Platform-level software: - Common firmware, kernel, OS distribution, libraries that are present in individual servers to abstract the hardware of a single machine and provide basic server-level services

Cluster-level infrastructure: - Software that manages resources and provides services at cluster level, e.g., distributed file systems, RPC libraries, programming models (MapReduce, Hadoop, BigTable, ) OS of the datacenter Application-level software: - Software that implements a specific service: online services (search, mail), offline computations (build index) National Tsing Hua University 18 Platform-Level Software Similar to that on a regular server platform

But, firmware, drivers, or OS can be simplified - HW homogeneity with fewer combinations of devices allows streamline firmware and driver and testing - WSC server is deployed in a relatively well-known environment, leading to further optimizations, e.g., networking is within same building and incur low losses Virtualization is now popular in WSCs - Provides portable interface to manage security and performance isolation and to perform live migration, e.g., allowing infrastructure to be upgraded or repaired without impacting a users computation National Tsing Hua University 19

Cluster-Level Infrastructure Software OS of a WSC - Resource management: controls mapping of user tasks to HW resources, enforces priorities and quotas, considers power limitations and energy usage optimization - Hardware abstraction and basic services: e.g., reliable distributed storage, message passing, and cluster-level synchronization as reusable modules or services - Deployment and maintenance: e.g., software image distribution, configuration management, monitoring and debugging and optimization - Programming frameworks: e.g., MapReduce and BigTable improve programmability by automatically handling data partitioning, distribution, fault tolerance National Tsing Hua University

20 Programming Models Most prevalent programming model: MapReduce - Embarrassingly parallel operations performed on very large datasets, e.g., search on a keyword, aggregate a count over several documents - Application writer provides Map and Reduce functions that operate on key-value pairs Hadoop: - An open-source implementation of the MapReduce framework; makes it easy for users to write MapReduce programs without worrying about low-level task/data management

National Tsing Hua University 21 Programming Models Map: operates on a collection of records - A record is (say) a webpage or a facebook user profile - The records are in the file system and scattered across several servers - Thousands of map functions are spawned on thousands of computers to work on all records in parallel - Provides new set of key-value pairs as intermediate values Reduce: aggregates and sorts the results produced by the Mappers, also performed in parallel

National Tsing Hua University 22 Hadoop: MapReduce Open Source Hadoop jobs consist of two types of tasks: - Map and Reduce tasks - Reduce tasks can only start when all map tasks have finished Hadoop cluster: - Compute nodes: where map and reduce tasks execute - Each compute node has a number of slots for the execution of map and reduce tasks - Each compute node has a TaskTracker daemon that monitors the execution of map and reduce tasks

National Tsing Hua University 23 MapReduce Model M: number of map tasks of a job k: number of map slots per worker node c: number of compute nodes n = k x c concurrent tasks National Tsing Hua University 24

WordCount Example Goal: count the number of occurrences of each word in a large file Approach: - File is broken down into a large number of chunks - Each map task takes a chunk from the file and breaks it into words and emits a (key, value) pair consisting of the word and a value of 1 - The reduce tasks add the number of 1s for each word and output the word and the sum National Tsing Hua University 25 WordCount: Data Flow

National Tsing Hua University 26 WordCount: Simplified Code map(String key, String value): // key: doc name; value: doc contents for each word w in value EmitIntermediate(w,1); // Produce list of all words reduce(String key, Iterator values): // key: a word; value: list of counts int result = 0; for each v in values: result += ParseInt(v);

Emit(AsString(result)); National Tsing Hua University 27 MapReduce Execution MapReduce runtime manages MapReduce jobs - Assigns MAP tasks to nodes based on how fast the nodes execute balances the load - Replicates execution of tasks and lets the nodes race Improves completion time but (somewhat) waists resources - Counteracts hardware failures by rerunning failed tasks - Provides stable storage e.g., Google File System, Dynamo (Amazon), Big Table

- Replicates data Using erasure coding for more efficient storage Improves resilience and performance - Delivers storage consistency - Deals with variability in utilization (maybe 100% change) National Tsing Hua University 28 Outline Programming models and workloads for warehousescale computers (Sec. 6.2) Computer architecture of warehouse-scale computers (Sec. 6.3) Physical infrastructure and costs of warehouse-scale computers (Sec. 6.4)

Could computing: the return of utility computing (Sec. 6.5) National Tsing Hua University 29 WSC Building Blocks A datacenter contains one or more clusters, and has a network and a power topology Server: 1 high (1U) x 19 wide x 16-20 (e.g., 8 cores, 16 GB DRAM, 4x1 TB disk) Cluster: 16-32 server racks +

larger LAN switch (array switch) Rack: 40-80 servers + Ethernet switch (rack switch) National Tsing Hua University 30 WSC Building Blocks Server Cluster Rack National Tsing Hua University

31 WSC Building Blocks Low-end servers: - Typically in 1U (1 high x 19 wide) or blade format Rack: - Servers are mounted within a rack (typically 19 wide with 48 1U servers), interconnected with an Ethernet switch - Rack-level switches (1- or 10-Gbps links) have a number of uplink connections to one or more cluster-level (or datacenter-level) Ethernet switches Cluster (or array): - A collection of server racks

- The second-level, cluster-level switching domain can span more than ten thousand individual servers National Tsing Hua University 32 A Row of Servers in a Google WSC (2012) National Tsing Hua University 33 Typical System Parameters Lower latency to DRAM in another server than local disk Higher bandwidth to local disk than to DRAM in another server

Local Racks Servers Cores (Processors) DRAM Capacity (GB) Disk Capacity (GB) DRAM Latency (microseconds) Disk Latency (microseconds) DRAM Bandwidth (MB/sec) Disk Bandwidth (MB/sec) National Tsing Hua University Rack Array

-1 30 1 80 2400 8 640 19,200 16 1,280 38,400 4,000 320,000 9,600,000 0.1 100 300 10,000

11,000 12,000 20,000 100 10 200 100 10 34 WSC Building Blocks A rack switch is used for communication within and out of a rack A cluster switch connects a cluster of racks Latency grows if data is fetched from remote DRAM or disk

Bandwidth within a rack is much higher than between racks Hence, software must be aware of data placement and locality National Tsing Hua University 35 Google Has Its Own Way: Server (2009) Fig. 6.21 (http://news.cnet.com/8301-1001_3-10209580-92.html#ixzz0yo8bhTOH) National Tsing Hua University

36 Storage Options 1. Connect disk/flash devices directly to each server - Managed by a global distributed file system, e.g., GFS, or 2. Use Network Attached Storage (NAS) devices directly connected to cluster-level switching fabric Comparisons: - NAS may be simpler to deploy by outsourcing to vendor - NAS is easier to enforce QoS since it runs no compute jobs besides storage server - Direct disks reduce hardware costs (leveraging existing server enclosure) - Direct disks improve networking fabric utilization (each network port is dynamically shared between computing

tasks and the file system) National Tsing Hua University 37 Storage Replication model between attached disks and NAS - NAS provides extra reliability through replication or error correction capabilities within each appliance - Desktop-class attached disk drives with GFS: Low cost, but may not be designed for continuous operation Replica across different machines (at least 3 replica), thus use more network bandwidth to complete write operations Can keep data available even after the loss of an entire server enclosure or rack Allow higher aggregate read BW because same data can be

sourced from multiple replicas Solid State Drives (SSDs) is becoming popular for databases in Web services for their very high IO rates National Tsing Hua University 38 Networking Fabric Trade-off between speed, scale, and cost - 1-Gbps Ethernet switches with 48 ports are commodity, less than $30/Gbps per server to connect a single rack Uplink count varies (2-8) which gives oversubscription in terms of bandwidth (48/2 to 48/8) Software scheduler should be aware of the difference and aim at mapping sender and receiver to the same rack

- Switches with high port counts to connect WSC clusters are 10 times more expensive than commodity switches - Therefore the networking of WSCs is often organized as a two-level hierarchy Also need to trade off between spending on networking vs. on buying more servers or storage National Tsing Hua University 39 Two-Level Hierarchy of Switches in a WSC Such a hierarchical structure in turn affects programmers view of

memory and storage Fig. 6.5 National Tsing Hua University 40 Programmers View of Memory A server consists of a number of processor sockets, each with a multicore CPU and its internal cache hierarchy, local shared DRAM, and a number of directly attached disk drives A rack has DRAM and disk resources, accessible through the first-level rack switches (assuming some sort of remote procedure call API to them)

A cluster has accesses to all resources in all racks via the cluster-level switch The relative balance of various resources depends on the needs of target applications National Tsing Hua University 41 Storage Hierarchy National Tsing Hua University 42 WSC Memory Hierarchy Servers can access DRAM and disks on other servers

using a NUMA-style interface Fig. 6.6 National Tsing Hua University 43 WSC Memory Hierarchy Example I What is the average latency assuming that 90% of accesses are local to the server, 9% are outside the server but local to the rack , and 1% are outside the rack but within the array (cluster)? (90%x0.1)+(9%x100)+(1%x300)=12.09 msec National Tsing Hua University

44 WSC Memory Hierarchy Example II How long does it take to transfer 1000MB between disks within the server, between servers in the rack, and between servers in different racks of an array? - Within server: 1000/200=5 sec - Within rack: 1000/100=10 sec - Within array: 1000/10= 100 sec National Tsing Hua University 45 A Layer 3 Network for Linking Clusters To connect 50,000 servers, the WSC needs 20

clusters one more level of networking hierarchy Fig. 6.8 National Tsing Hua University 46 Implications of Memory Hierarchy Applications must deal effectively with the large discrepancies in latency, bandwidth, and capacity making it difficult to program a WSC A challenge for architects of WSCs is to smooth out these discrepancies in a cost-efficient manner Another challenge is to build cluster infrastructure and services that hide most of this complexity from application developers

Flash offers a viable option for bridging the cost and performance gap between DRAM and disks, especially for random reads (3 orders of magnitude) National Tsing Hua University 47 Power Usage Example breakdown of peak power usage of a datacenter using 2012 generation servers - Assumes two-socket x86 servers, 16 DIMMs and 8 disk drives per server, and an average utilization of 80% National Tsing Hua University 48

Outline Programming models and workloads for warehousescale computers (Sec. 6.2) Computer architecture of warehouse-scale computers (Sec. 6.3) Physical infrastructure and costs of warehouse-scale computers (Sec. 6.4) Could computing: the return of utility computing (Sec. 6.5) National Tsing Hua University 49 Warehouse for WSCs Building(s) to house computing and storage

infrastructure in a variety of networked formats - Deliver the utilities needed by housed equipment and personnel, e.g., power, cooling, shelter, and security Location considerations: proximity to ... - Internet backbone optical fibers, low cost electricity, low risk of environment disasters (earthquakes, floods, hurricanes, ), geographical vicinity of large population of users, real estate deals and low property taxes Main design challenge and costs: delivery of input energy (power distribution) and removal of waste heat (cooling) National Tsing Hua University 50

Datacenter Power and Cooling System National Tsing Hua University 51 Power Distribution Room for improvement is 11% Fig. 6.9 National Tsing Hua University 52 Power Distribution

High voltage lines at utility tower: 115 kv Substation switches 115 kv to 13.2 kv UPS (Uninterruptable Power Supply) with generators - Power conditioning, holding electrical load while generators start and come on line, and switching back to the electrical utility - Take a large space and account for 7% to 12% IT cost PDU (power distribution unit): convert to 480 v - One PDU unit handles around 10 racks One more level conversion to 208 v for servers Total efficiency: - 99.7% x 94% x 98% x 98% x 99% = 89% National Tsing Hua University

53 Cooling System National Tsing Hua University 54 Cooling System Computer room air-conditioning (CRAC) units - Cools the air using chilled water - Fans push warm air past a set of coils filled with cold water, and a pump moves the warmed water to external chillers to be cooled down e.g. 70,000 to 200,000 gallons per day for an 8 MW facility

- Cool air typically between 18C ~ 22C Improving energy efficiency - Keep temperature of IT equipment higher (closer to 22C) But increases failure rate and wear of components - Cooling tower can use the colder outside air to cool the water before it is sent to the chillers - Careful separation of cold and hot: alternating aisles National Tsing Hua University 55 Typical Heat Management CRAC unit

rack rack rack rack CRAC unit floor tiles Liquid supply National Tsing Hua University

56 Improving Energy Efficiency Unloaded servers dissipate large amount of power - Can approach energy-proportionality by turning on a few servers that are heavily utilized Fig. 6.3 National Tsing Hua University 57 Costs of Cooling System Power cost breakdown relative to IT equipment:

- Chillers: 30-50% of the power used by the IT equipment - Air conditioning: 10-20% of IT power, mostly due to fans How many servers can a WSC support? - Can get this number based on required power 1. Measure a single server under a variety of workloads 2. Divide available IT power by measured server power - Can oversubscribe cumulative server power by 40%, because not all servers running at peak at the same time - But need to monitor power closely National Tsing Hua University 58 Measuring Energy Efficiency of a WSC Power Utilization Effectiveness (PEU)

= Total facility power / IT equipment power - PEU less efficient - PEU > 1 (1.33 to 3.03 in 2006) - Median PUE: 1.69 cooling power is roughly half the power used by servers National Tsing Hua University 59 How to Improve Energy Efficiency? Datacenter-level (PUE) - Use passive cooling (NCSA Blue Waters, UC Berkeley CRTF)

Avoid UPS devices (lower Tier level) Avoid AC/DC conversion Use green power (Icelandic model) Rack-level (SPUE) - Avoid local power supplies - More efficient voltage regulation Processor/process-level - Speed-variable processors - Embedded, low-power processors - Smarter parallelization/distribution of works National Tsing Hua University 60

Measuring Performance of a WSC Latency is important metric because users concern Users satisfaction and productivity are tied to response time of a service - User productivity = 1 / time of interaction t(interaction) = t(human entry) + t(system response) + t(analysis of response) - t(response) reduced by 30% 70% less t(interaction), because people think continuously when not interrupted by a long delay - Bing study: users use search as response time 200 ms longer delay 500 ms longer time to next click Revenue and user satisfaction drops linearly with increasing delay National Tsing Hua University

61 Primary Concern: User Satisfaction Based on Internet studies... - Page load above few tens of milliseconds cause user to switch to another task - Page load time must be below 1s or it is deemed broken Users do not come back Quantifying influence of response delays - Use % requests below a latency threshold instead of avg. - SLO (Server Level Objective), SLA (Server Level Agreement) Example: 99% of requests must be below 100 ms delay - Amazon's Dynamo: 99.9% of key-value request must be below a threshold

Which is more important: average case or the tail (diminishing return)? National Tsing Hua University 62 Reliability and Availability Reliability (MTTF) & Availability (MTTF/MTTF+MTTR) are very important, given the large scale - MTTF: mean time to failure - MTTR: mean time to repair - A server with MTTF of 25 years 50K servers would lead to 5 server failures a day - Annual disk failure rate of 2-10% 1 disk failure per hour National Tsing Hua University

63 Outages and Anomalies Approximate frequencies of outages and anomalies in the first year of a new cluster of 2400 servers Fig. 6.1 National Tsing Hua University 64 Outages and Anomalies Example Calculate the availability of a service running the 2400 servers in Fig. 6.1, assuming -

Service cannot tolerate software or hardware failures Reboot time= 5 minutes Hardware repair time= 1 hour Ignore slow disks and power utility failures All events have equal probability Hours outage = 1192 hours = (4+250+250+250) x 1hr + (250+5000) x 5min Availability = (365x24-1192)/8760 = 86% service down average 1 day per week National Tsing Hua University 65 Cost of a WSC

Capital expenditures (CapEx) - Infrastructure costs for the building, power delivery, cooling, and servers, i.e., cost to build a WSC Operational expenditures (OpEx) - Cost to operate a WSC, e.g., monthly bill for energy, failures, personnel, etc. CapEx can be amortized into a monthly estimate - E.g., assume facilities will last 10 years, server parts will last 3 years, and networking parts will last 4 years Estimates of CapEx and OpEx give an idea of where to invest to cut costs: software, hardware, infrastructure National Tsing Hua University

66 CapEx/OpEx Case Study 8 MW facility cost $88M 46,000 servers cost $67M $13M for networking equipment PUE: 1.45 Monthly expense: $3.8M - (Fig. 6.13) Servers 53% (amortized CapEx) (Fig. 6.14) Networking 8% (amortized CapEx)

Power/cooling infrastructure 20% (amortized CapEx) Other infrastructure 4% (amortized CapEx) Monthly power bill 13% (true OpEx) Monthly personnel salaries 2% (true OpEx) National Tsing Hua University 67

Recently Viewed Presentations

  • Why do you think the Tudors had this

    Why do you think the Tudors had this

    Green and White A White Rose A Red Rose Why do you think the Tudors had this rose design? Why do you think the Tudors had this rose design? Green and White A White Rose A Red Rose
  • Poetic Devices: Diction - Liberty Union High School District

    Poetic Devices: Diction - Liberty Union High School District

    Poetic Devices: Diction English 1 Mrs. Foster Every Word Counts In 10 words or less, describe: The most beautiful thing you've ever seen In 10 words or less, describe: The ugliest thing you've ever seen Diction A writer's choice of...
  • Multiple regression refresher - University of Vermont

    Multiple regression refresher - University of Vermont

    A simple multivariate model. Another example: car price. Addressing multicollinearity. Intercorrelationof Xs. When excessive, SE of beta coefficients become large, hard to assess relative importance of Xs. ... Multiple regression refresher
  • Intermodal What You Need to Know Transportation &

    Intermodal What You Need to Know Transportation &

    Mission Statement. Utilize Integra's experienced personnel and its proprietary web based technology systems to deliver an unparalleled customer service experience to our clients and their customers
  • Youth employment - DOL

    Youth employment - DOL

    Minimum Wage of $7.25 per hour for all time worked. Tipped Employees: $2.13/hr cash wage + tips = $7.25. Deductions may not cut into minimum wage or overtime pay. Most training time is work time and must be paid. Overtime=time...
  • New Scouts Path To First Class: A Troop

    New Scouts Path To First Class: A Troop

    New Scouts' Path To First Class:A Troop Program. Bob Petraitis - UoS 2018 Why was this presentation developed? Retaining Scouts in the program is a challenge presented to every Troop.
  • Variations on a theme: extensions to Black-Scholes-Merton

    Variations on a theme: extensions to Black-Scholes-Merton

    Variations on a theme: extensions to Black-Scholes-Merton option pricing Dividends options on Forwards/Futures (Black model) currencies (Garman-Kohlhagen) S. Mann, 2010 Martingale pricing : risk-neutral Drift Risk-neutral pricing: we model prices as Martingales with respect to the riskless return.
  • The Last Epoch Cosmic Structure Formation  Today we

    The Last Epoch Cosmic Structure Formation Today we

    Star Formation. As H and He accrete, gas pressure increases limiting further growth unless there is sufficient matter; maximum star mass is about 60 M. sun.. The Jeans mass (≈ 105M. sun) constitutes the minimum amount of mass required for...