CS50 Lecture by Mark Zuckerberg - 7 December 2005

Main ideas, concepts, and lessons

1) How Mark Zuckerberg’s Harvard CS experience influenced Facebook’s growth

• Zuckerberg frames his talk around how specific decisions while building Facebook were informed by concepts/courses learned at Harvard.
• He highlights relevant coursework he took (e.g., CS121, CS161).
• He also notes a practical gap: he did not take CS50—instead, his roommate Dustin had.

2) Early engineering choices for launching and scaling from one school to many

Initial implementation

• Facebook was initially written in PHP (not a language he presents as central to his Harvard coursework).
• With a C background, he says PHP was easy to pick up.
• Launch: Harvard, February 2004.
  • It quickly reached a few thousand users.
  • Other colleges began requesting launches.

Scaling decision #1: distribute data by school

• Core scaling challenge: computing “connections” (friend-of-friend / shortest-path style queries).
• Computational growth (as degrees expand):
  • Friends-of-friends: roughly n²
  • Friends-of-friends-of-friends: roughly n³
  • More degrees out becomes increasingly hard and can kill performance.
• Example: predecessor site Friendster had outages when trying to compute paths “6 or 7 degrees out.”
• Architectural solution:
  • Assumption: most activity happens within a user’s own school (about 90% of interactions).
  • Create one MySQL database instance per school.
  • “Paths” are computed only within a school, shrinking the effective problem size:
    • Example given: compute over ~10,000 users instead of millions.

Scaling decision #2: grow infrastructure linearly

• Early setup:
  • A single machine running Apache (web server) and MySQL.
• Moving beyond the dorm:
  • He describes renting off-site servers instead of running everything in a dorm.
• Because data distribution was by school, scaling could be done by adding machines linearly (rather than exponentially).

Scaling decision #3: separate web and database servers

• After roughly 30–50 schools, performance problems from running MySQL and Apache on the same server became more visible:
  • If that server fails, both database and web serving fail (page not founds).
  • Load varies widely by school size:
    • Example: Penn State ~50,000 users, while many others were under 2,000.
• Solution:
  • Build a pool of load-balanced Apache web servers.
  • Keep the database layer consistent per school.

3) Performance at massive scale: caching + redundancy

Bottlenecks at extreme traffic

• He describes hitting bottlenecks as traffic grows (example timeframe: ~100 million pages a day).
• MySQL baseline:
  • Typical query latency: 2–4 ms
  • At extreme scale (e.g., “100 billion page views a day” with “30–50 queries per page view”), even tiny latencies add up massively.

Caching layer

• Introduced caching to reduce access time:
  • Memcache (initially): about 0.3–0.5 ms
• Memcache issue:
  • It’s intended to act like a distributed hash table conceptually.
  • In practice, he describes failure modes:
    • When nodes go down, there’s no redundancy.
    • Cache misses increase, overloading MySQL and harming performance.

Evolving beyond Memcache

• They “outgrew” the initial Memcache assumptions and added extra redundancy on top (implying custom engineering).
• Some of this was not open-sourced, for competitive/practical reasons.

4) Ongoing product/feature decisions: privacy and social constraints

Core privacy principle

• Information should be available only to people the user intends to see it, and within a meaningful context.

School-based visibility rule

• Only students at the same school can see each other’s profiles.
• Rationale:
  • Reduces misuse.
  • Lowers the likelihood of strangers accessing sensitive information.

User control over visibility

• Granular controls over sensitive fields (e.g., “who do you want to show this to?”).

Limits of user control

• He emphasizes that Facebook can’t fully control what happens after users share info:
  • Once someone has access, it’s effectively out of Facebook’s control.

Scraping and data aggregation defenses

• Defenses described include:
  • Making school email addresses/images (not plain text) to reduce scraping.
  • Detecting abnormal profile viewing patterns and using Bayesian filters to restrict what suspicious users can see.

5) How Facebook features evolved (examples)

Wall / “wiki-like” concept → redesigned

• Started as a quick concept to show contributors (hover-like, wiki-ish idea).
• Problems:
  • Mouse-over wasn’t accurate/useful (wrong attribution, incorrect highlighting).
• Redesign:
  • Better parser/decomposition.
  • Shift toward a simpler model: “add a post” that appears at the top.

Photo upload and performance

• Initial approach:
  • Distributed storage concept.
  • Slower stable disk plus caching layer holding thumbnails and frequently accessed images in RAM.
• Upload problem:
  • Network/router limitations.
  • Example: uploading a 30-photo album (~90 MB) created bottlenecks.
• Solution approach:
  • Client-side compression using Java applet and ActiveX control to reduce upload size.
  • Additional step: edge caching (Akamai-like) since photos are relatively static.

6) Organizational and engineering culture (“how decisions get made”)

• Meritocracy:
  • People who can implement cool solutions quickly with fewer bugs gain influence.
• Pairing/onboarding:
  • New hires are paired with strong engineers to learn company style and methods.
• Iteration mindset:
  • The company doesn’t need to be perfect on day one; it should iterate.
  • Comparisons made to:
    • Google beta releases
    • Microsoft’s product maturity across versions
• Architecture-first:
  • Get the architecture right, then improve implementation details iteratively.

7) Hiring philosophy and learning approach

Hiring priorities

• Don’t hire “because they have business skills.”
• Emphasize CS fundamentals and complexity/scale thinking, which he claims translates into scaling a business.

Learning method

• Use the internet as a primary learning tool.
• Hire younger/raw-intelligence talent who can learn quickly from available information.
• Background examples:
  • Dustin (roommate) wasn’t a CS major (economics) but could pick up technical skills.
  • Other strong hires from EE/math backgrounds are also effective at learning and building.

8) Future roadmap themes discussed

Coming near-term launches

• Aggregated “stats” (what’s hot/changing, including surprising estimates like political affiliation).
• A feature to clarify relationships beyond binary “friend” status.

Relationship-strength concept

• He rejects “rate your friendship 1–10” as socially pressured and likely meaningless.
• Proposed approach:
  • Infer closeness using bi-directional factual relationship categories (conceptually: “took a course together,” “lived in a house together”).
• Add nuance about time:
  • Relationship meaning depends on when it formed (more recent shared experiences imply different strength than older ones).

9) Audience Q&A highlights (technical + non-technical)

• Competition and leverage:
  • Individuals/young teams can do more because infrastructure is cheaper (rented hardware vs. earlier hardware-heavy eras).
  • Distributed cheap machines help redundancy.
• Legal/hiring/prioritization:
  • They have a full-time lawyer now.
  • Early legal work was limited; it caused “annoyance later.”
• Procrastination:
  • He acknowledges procrastination humorously, emphasizing the value of time spent on the site.

---

Methodology / lists of instructions (detailed bullet points)

A) Architectural method to make “connection” computations scalable

• Identify the expensive computation:
  • Friend connections require expanding through a graph of user relationships.
• Estimate growth:
  • Friends-of-friends ≈ n²
  • Friends-of-friends-of-friends ≈ n³
  • Degree expansion becomes exponentially difficult.
• Mitigate by reducing the effective graph size:
  • Assume most interactions are within each user’s school (~90%).
• Implement data distribution:
  • Split the system so that each school has its own MySQL database instance.
• Result:
  • Compute shortest/connection paths only within the school, avoiding global-degree computation over millions.

B) Performance engineering method used when traffic grows

• Measure bottlenecks:
  • Even small query latency (2–4 ms in MySQL) becomes costly at high request volume.
• Add caching:
  • Use Memcache to reduce latency (down to ~0.3–0.5 ms).
• Validate resilience:
  • If cache nodes fail, watch for lack of redundancy leading to cache misses and overload on the database.
• Extend/replace caching approach:
  • Move beyond Memcache-only assumptions.
  • Add redundancy and additional layers to prevent cache outages from overwhelming MySQL.

C) Scaling web serving

• Start with one machine running both web and database (Apache + MySQL).
• When reliability demands rise:
  • Failure mode: if web+DB share a server, server failure breaks everything.
  • Variance problem: large schools overload while smaller ones are much lighter.
• Split responsibilities:
  • Separate web servers from database servers.
  • Use a pool of load-balanced Apache servers.
  • Keep database instances stable and consistent per school.

D) Privacy/scraping defenses (non-technical + technical blend)

• Reduce exposure scope:
  • Only allow same-school profile viewing.
• Provide user controls:
  • Granular per-field visibility (“friends,” “school,” etc.).
• Make scraping harder:
  • Represent school emails in a way less suitable for scraping (e.g., image instead of plain text).
• Detect abnormal behavior:
  • Use Bayesian filtering to identify unusual profile viewing patterns.
  • Restrict content visibility to suspicious actors.

E) Photo upload performance approach (client + server + CDN)

• Build initial storage architecture:
  • Distributed smaller storage plus a caching layer with lots of RAM for thumbnails/frequently accessed images.
• Detect upload bottlenecks:
  • Network/router limitations make large raw uploads too slow.
• Offload work to the client:
  • Use Java applet / ActiveX control for client-side photo compression before upload.
• Add CDN-style caching:
  • Use edge caching (Akamai-like) for static photo content closer to users.

F) Engineering process for implementing ideas

• Use “meritocracy”:
  • People who can implement cool ideas quickly with few bugs should lead.
• Iterate instead of expecting perfection:
  • Release in phases; improve repeatedly.
• Onboard to scale teams:
  • Pair new hires with top engineers so they learn quickly.
• Reassure architecture-first:
  • If architecture is right, implementation can be iterated later.

---

Speakers / sources featured (identified)

• Michael D. Smith (host/introducer; also fields questions)
• Mark Zuckerberg (main speaker; founder of Facebook)
• Dustin Moskovitz (mentioned; Zuckerberg’s roommate early on; credited with CS50 completion and involvement)
• Andrew McCollum (identified in an audience/search-server discussion excerpt; contributes to custom search-related work)
• Audience / Students at Harvard (group of questioners; no individual names provided beyond a brief mention: “Will Chen”)

Companies/technology sources mentioned (not as direct speakers)

• Facebook.com, Friendster, Google, eBay, Yahoo
• Oracle (contrasted with MySQL features), MySQL
• Apache, Memcache (“Memcache boxes”)
• Akamai (edge caching), NetApp
• Microsoft, Linux
• Bayesian filters (Bayesian filtering concept; not a named external speaker)
• Cingular (example SMS/email gateway domain)