I moved my site, but some users still reach the old server
See the problem
Section titled “See the problem”Your shop runs on a server at 203.0.113.10. You move it to a bigger server at 198.51.100.20 and update the DNS record. You turn the old server off.
Within minutes, support gets messages: “The shop is down.” For others it works fine. A day later, the messages stop by themselves. What happened?
Define the task
Section titled “Define the task”- What you have: a name people type,
shop.example.com, and servers whose addresses change. - What you want: every user finds the right server, quickly, even right after a move.
- The hard part: billions of lookups a day can’t all go to your DNS server. So answers are copied and kept around the world, and old copies don’t know you moved.
Pause and think
Section titled “Pause and think”Hint 1 of 3
Your DNS server would get a question for every page view, from every user, everywhere.
Hint 2 of 3
Each answer comes with a number: how many seconds it may be kept.
Hint 3 of 3
What if you made that number small just before the move?
Understand the idea
Section titled “Understand the idea”DNS is the internet’s phone book. You know a name; you need a number (an IP address). Nobody keeps the whole book. Instead, each part of a name has its own keeper:
- A root server knows who keeps
.com. - The .com server (a top-level domain, or TLD, server) knows who keeps
example.com. - The authoritative server for
example.comknows the actual address ofshop.example.com.
Your browser doesn’t walk these steps itself. It asks a resolver, run by your internet provider or a public service like 1.1.1.1. The resolver does the walk once, then keeps the answer for its TTL (time to live), in seconds.
Watch one lookup walk the three servers, then see what the TTL does after a move. Try a TTL of 300 below the player.
One lookup
Parts:
- Browser
- Resolver
- Web server
- Root server
- .com server
- Shop’s DNS
Arrows:
- Browser sends requests to Resolver
- Resolver sends requests to Root server
- Resolver sends requests to .com server
- Resolver sends requests to Shop’s DNS
- Browser sends requests to Web server
Step 1 of 8. You type shop.example.com. The browser needs the server’s address (an IP address) before it can send anything.
The tests run this small model of a resolver with a cache:
"""A small model of DNS: a resolver that walks root -> TLD -> the site's ownDNS server, and keeps each answer for its TTL (time to live).
Time is passed in as `now` (seconds), so the tests can move the clock."""
from dataclasses import dataclass, field
@dataclassclass Record: address: str ttl: int # Seconds a resolver may keep this answer.
@dataclassclass Internet: """The authoritative answers: what each site's own DNS server says right now."""
records: dict[str, Record] = field(default_factory=dict)
def set(self, name: str, address: str, ttl: int) -> None: self.records[name] = Record(address, ttl)
@dataclassclass Resolver: """The resolver your internet provider (or 1.1.1.1, or 8.8.8.8) runs for you."""
internet: Internet # name -> (address, expires at) cache: dict[str, tuple[str, float]] = field(default_factory=dict) queries: int = 0 # Questions sent to other DNS servers.
def lookup(self, name: str, now: float) -> str: cached = self.cache.get(name) if cached and now < cached[1]: return cached[0] # Still fresh: no questions sent. # Ask the root, then the TLD server (.com), then the site's own DNS server. self.queries += 3 record = self.internet.records[name] self.cache[name] = (record.address, now + record.ttl) return record.addressRecord types you will meet
| Type | Points a name at | Example |
|---|---|---|
A | an IPv4 address | shop.example.com → 203.0.113.10 |
AAAA | an IPv6 address | shop.example.com → 2001:db8::10 |
CNAME | another name | www.example.com → shop.example.com |
MX | the mail servers | example.com → mail.example.com |
TXT | free text, often for proving you own the name | example.com → "v=spf1 …" |
NS | the authoritative servers for a name | example.com → ns1.dnshost.net |
Climb the ladder
Section titled “Climb the ladder”Each rung is a way to move the shop to a new server.
Hard-code the address
The idea: Put the server’s IP address in the app, the config files and the links you send.
This is how a home network often works: you type 192.168.1.20 to reach the printer.
- Lookup cost
nothing to look up- Weak spot
every move breaks every user- Why
- No name, no lookup. But the address is copied everywhere.
Where it breaks: the move. Every copy of the old address, in phones, bookmarks and emails, is now wrong. Nobody remembers numbers, and HTTPS certificates are made for names, not addresses.
A name, with a long TTL
The idea: Give the server a name. Resolvers walk root, .com and your DNS server once, then keep the answer for a day.
The waste in Okay: the address was copied into every client. Now only the name is copied, and DNS gives the address.
- Lookup cost
1 walk a day per resolver- Weak spot
a move takes up to a day to reach everyone- Why
- With a TTL of 86,400 seconds, each resolver asks once a day and answers everyone else from memory.
Where it breaks: the story at the top. A resolver that looked up the name just before the move keeps the old address for up to 24 hours. Turn the old server off, and those users see an error.
Lower the TTL first, then move
The idea: A day before the move, lower the TTL to 5 minutes. Move, keep the old server running for a while, then raise the TTL again.
The waste in Better: a one-day TTL is great on a normal day and terrible on moving day. The trick is to change it only around the move. Keep the old server up (or have it send people on) until the old copies have run out.
- Move takes
old answers gone in 5 minutes- Weak spot
more lookups while the TTL is low- Why
- Once every old one-day copy has run out, every copy is at most 5 minutes old.
When Best isn’t best: if you move servers every week, stop changing DNS at all. Point the name at something that never moves, like a load balancer, and swap servers behind it. Some resolvers also ignore short TTLs and keep answers longer, so never count on the TTL exactly.
Draw it yourself
Section titled “Draw it yourself”Step 1 finds the shop by name. Step 2 puts a stable address in DNS, so swapping servers needs no DNS change at all.
Step 1: Find the shop by its name
- People type shop.example.com to reach your shop.
- One app server and one database for now.
Solid arrow: sends requests toDashed arrow: copies data to
Watch the traffic
Send 10 requests through your board and see where each one goes. Then make a part fail and send them again.
How requests move
Requests start at the Client and follow your arrows. Reads try a cache first (4 in 5 are found there), then a read replica, then the database. Writes go to the database. Background jobs go on a queue, and a worker saves them. A load balancer skips a part that is down; nothing else can.
Compare
Section titled “Compare”| Way | A normal day | Moving day | Use it when |
|---|---|---|---|
| Hard-coded address | No lookups | Everyone breaks | Home networks, tests |
| Name, long TTL | Cheap: one walk a day per resolver | Up to a day of errors | Things that rarely move |
| Lower the TTL, then move | Cheap again after the move | 5 minutes of mixed answers | A planned move |
| Stable address (load balancer) in DNS | Cheap | No DNS change at all | Servers that change often |
Common mistakes and edge cases
Section titled “Common mistakes and edge cases”- A CNAME at the top of a domain.
example.comitself can’t be aCNAME, because it must also haveNSand other records. Many DNS hosts offer an “alias” or “flattening” record instead. - Lowering the TTL too late. If you lower it an hour before a move, resolvers that fetched it earlier still hold the one-day copy. Lower it at least one old TTL before.
- Browsers and operating systems cache too. A resolver is not the only copy; apps and phones keep answers for a while.
- Mail breaks after a move. Email follows
MXrecords, notArecords. Move them too. - DNS is not a load balancer. DNS can hand out several addresses, but it can’t tell when one is down, and copies keep pointing at it.
Say it in the interview
Section titled “Say it in the interview”Questions to ask first
- How often do the servers behind this name change?
- Do users come from one country or many?
- How quickly must a change take effect?
A 60-second walkthrough
“The browser asks a resolver. The resolver asks a root server, then the .com server, then the site’s authoritative server, and gets the address with a TTL. It keeps that answer for the TTL, so nearly every lookup is answered from memory. That is why DNS scales, and why changes are slow. A day before a planned move, I lower the TTL to a few minutes. Then I move, and keep the old server up until old copies run out. For servers that change often, I point DNS at a load balancer and change things behind it.”
Likely follow-ups
- Why not a TTL of 1 second always? Every resolver would ask on almost every visit. Lookups get slower, and your DNS servers get far more traffic.
- How does DNS send users to a nearby region? The DNS host can answer with a different address depending on where the resolver is (often called GeoDNS). It works with health checks to stop handing out a failed region.
- What is anycast? The same IP address is announced from many places, and the network sends each user to the nearest one. Root servers and CDNs use it.
Red flags interviewers notice
- Thinking a DNS change reaches everyone at once.
- Using DNS alone for failover, with long TTLs.
- Not knowing the difference between a resolver and an authoritative server.
Practice
Section titled “Practice”1. Which record? (easy)
Section titled “1. Which record? (easy)”You need these: example.com opens your site at 203.0.113.10; www.example.com opens the same site; email to you@example.com reaches your mail provider; a search tool asks you to prove you own the domain.
Hint 1 of 3
An address for a name is one record type; a name for a name is another.
Hint 2 of 3
Email has its own record type.
Hint 3 of 3
Proving you own a domain usually means adding a line of text.
Show the answers
An A record for everything
The idea: Point example.com and www.example.com at 203.0.113.10 with two A records.
The site works. Email to the domain has nowhere to go.
- Covers
the website- Weak spot
mail and proof are missing; two places to change- Why
- A records only hold addresses, and a move now means changing two records.
A, CNAME and MX
The idea: An A record for example.com, a CNAME from www to example.com, and an MX record for mail.
MX records have a priority number, so you can list a main and a backup mail server.
- Covers
site and mail- Weak spot
ownership not proved yet- Why
- The CNAME follows the A record, so a move changes one record.
A, CNAME, MX and TXT
The idea: Add the TXT record the tool gives you, and an AAAA record if the server has an IPv6 address.
Mail providers also ask for TXT records (SPF and DKIM) so other servers trust your email.
- Covers
everything asked- Weak spot
none- Why
- TXT records hold the proof text; AAAA lets IPv6 users connect directly.
2. Two regions, one name (medium)
Section titled “2. Two regions, one name (medium)”Your app runs in Europe and in the US. Users should reach the nearer region, and if one region fails, everyone should go to the other.
Hint 1 of 3
Can DNS give a different answer depending on who asks?
Hint 2 of 3
If a region fails, how fast do resolvers stop sending users there?
Hint 3 of 3
Which part checks whether a region is healthy?
Show the answers
Two A records
The idea: Return both regions’ addresses and let clients pick one.
Half your European users cross the ocean, and when a region fails half of all requests time out.
- Gives
spreads users roughly evenly- Weak spot
no sense of near, no failover- Why
- Clients pick by order or at random, not by distance, and a dead address stays in the list.
Answers by location
The idea: The DNS host answers with the nearer region’s load balancer, judged by where the resolver is.
The resolver’s location is close to the user’s in most cases. Some resolvers pass part of the user’s address to help (EDNS Client Subnet).
- Gives
users go to the near region- Weak spot
a failed region keeps getting users- Why
- Geo answers fix distance, but nothing removes a failed region.
Location plus health checks, short TTL
The idea: The DNS host checks each region every few seconds and stops handing out a failed one. Keep the TTL at about 60 seconds.
For failover in seconds, use anycast addresses, where the network itself sends users around a failed region.
- Gives
near region, failover in a few minutes- Weak spot
users of a failed region wait out the TTL- Why
- A short TTL plus health checks makes failover take about a minute or two.
3. Your DNS provider goes down (hard)
Section titled “3. Your DNS provider goes down (hard)”In 2016, an attack on one big DNS provider took many large sites offline for hours. Their servers were fine; nobody could find them. How do you make your site survive a DNS provider outage?
Hint 1 of 3
A domain can list more than one set of authoritative servers.
Hint 2 of 3
Resolvers that already hold your answer don’t need your provider until the TTL runs out.
Hint 3 of 3
Two providers must always give the same answers.
Show the answers
Wait it out
The idea: Trust the provider and wait for them to fix it.
Users whose resolvers still hold your answer can reach you, until their copies run out.
- Downtime
as long as their outage- Weak spot
you are down too- Why
- Every new lookup fails until the provider is back.
Longer TTLs on stable records
The idea: Records that rarely change keep a TTL of hours, so cached answers carry users through a short outage.
This conflicts with quick moves, so keep short TTLs only where you need them.
- Downtime
cached users keep working- Weak spot
new users still fail; moves are slower- Why
- The TTL works as a buffer: an outage shorter than the TTL is hidden from users who looked you up already.
Two DNS providers
The idea: List authoritative servers from two independent providers in your NS records, and keep their records the same with a tool.
Some features, like answers by location, work differently at each provider, so keep the records simple.
- Downtime
near zero- Weak spot
two providers to pay for and keep in sync- Why
- Resolvers try another listed server when one does not answer.
Check yourself
Section titled “Check yourself”Cheat sheet
Section titled “Cheat sheet”- Browser → resolver → root → TLD (.com) → the site’s authoritative server. The resolver keeps the answer for its TTL.
- A long TTL means fewer lookups and slow changes. A short TTL means quick changes and more lookups.
- Before a move: lower the TTL one old TTL ahead, move, keep the old server up, then raise it.
- A for IPv4, AAAA for IPv6, CNAME for a name, MX for mail, TXT for text, NS for who answers.
- Point DNS at something stable, like a load balancer, and change servers behind it.
Go deeper
Section titled “Go deeper”- Cloudflare Learning: What is DNS?
- RFC 1034: Domain names, concepts and facilities
- Before this: Client–server, HTTP and HTTPS.
Can you do these now?
If you’re not sure of any of these, scroll back to that part before moving on.
Status: Not started