Transport Layer Protocols
TCP connections, UDP datagrams, QUIC, ports, sockets, congestion control, and reliability mechanisms
Chapter 5: Transport Layer Protocols
The Protocol That Changed Streaming Forever
In 2012, Google engineers faced a problem: despite decades of optimization, web page loads still felt sluggish. The culprit wasn’t bandwidth—it was TCP’s fundamental design. Every new connection required a three-way handshake (one round trip). TLS added another two round trips. A single lost packet could stall the entire connection due to head-of-line blocking.
Their solution was radical: abandon TCP entirely. Build a new protocol from scratch, optimizing for modern internet conditions. They called it QUIC (Quick UDP Internet Connections).
By 2023, QUIC powered over 25% of all internet traffic. Google, Facebook, and Cloudflare use it extensively. HTTP/3 is built on QUIC. It reduces connection establishment from 2-3 round trips to zero (with session resumption) and eliminates head-of-line blocking entirely.
This chapter explores the Transport layer—TCP, UDP, and the revolutionary QUIC protocol that’s reshaping the internet.
End-to-End Communication
The Network layer (IP) handles getting packets from one host to another across the internet. But a host typically runs many applications simultaneously—your web browser, email client, music streaming service, and more. How does incoming data reach the right application? And how do we ensure data arrives reliably and in order?
The Transport layer answers these questions. It provides end-to-end communication between applications, not just between hosts. Through port numbers, it multiplexes multiple application conversations over a single IP connection. Through mechanisms like acknowledgments and retransmission, it can provide reliability over an inherently unreliable network.
Three protocols dominate this layer:
- TCP - Reliable, ordered delivery
- UDP - Fast, connectionless communication
- QUIC - Modern encrypted transport (TCP replacement for HTTP/3)
Ports and Sockets
Port Numbers
A port number is a 16-bit integer (0-65535) that identifies a specific application or service on a host. When data arrives at a host, the Transport layer uses the destination port to determine which application should receive it.
Think of IP addresses as street addresses and port numbers as apartment numbers. The IP gets the packet to the building; the port gets it to the right unit.
Transport Layer Addressing
Transport Layer Addressing:
═══════════════════════════════════════════════════════════════════
Your Computer Web Server
192.168.1.100 93.184.216.34
┌─────────────────────┐ ┌─────────────────────┐
│ Browser :52431 │────────────►│ Web Server :443 │
├─────────────────────┤ ├─────────────────────┤
│ Email Client :52432 │────────────►│ Mail Server :25 │
├─────────────────────┤ ├─────────────────────┤
│ SSH Client :52433 │────────────►│ SSH Server :22 │
└─────────────────────┘ └─────────────────────┘
Client uses ephemeral ports Server uses well-known ports
(49152-65535, random) (0-1023, fixed)Port ranges:
| Range | Name | Description |
|---|---|---|
| 0-1023 | Well-Known | Reserved for standard services (requires root/admin) |
| 1024-49151 | Registered | Assigned by IANA for specific applications |
| 49152-65535 | Dynamic/Ephemeral | Used for client-side connections |
Common well-known ports (memorize these):
| Port | Protocol | Service | Security Notes |
|---|---|---|---|
| 20, 21 | TCP | FTP | Unencrypted, use SFTP instead |
| 22 | TCP | SSH | Secure remote access |
| 23 | TCP | Telnet | Unencrypted, deprecated |
| 25 | TCP | SMTP | Email transfer (often filtered) |
| 53 | TCP/UDP | DNS | Critical infrastructure target |
| 67, 68 | UDP | DHCP | Rogue server attacks |
| 80 | TCP | HTTP | Unencrypted web |
| 110 | TCP | POP3 | Legacy email retrieval |
| 143 | TCP | IMAP | Email access |
| 443 | TCP/UDP | HTTPS/QUIC | Encrypted web |
| 445 | TCP | SMB | Windows file sharing (common attack target) |
| 3306 | TCP | MySQL | Should never be exposed |
| 3389 | TCP | RDP | Remote desktop (brute force target) |
| 5432 | TCP | PostgreSQL | Database |
| 8080 | TCP | HTTP Alt | Common for proxies/alt web |
Sockets
A socket is the combination of an IP address and port number, uniquely identifying an endpoint in a network conversation. A connection between two applications is identified by four values: source IP, source port, destination IP, and destination port.
Socket and Connection Identification
Socket and Connection Identification:
═══════════════════════════════════════════════════════════════════
Socket = IP Address : Port Number
Client Socket: 192.168.1.100:52431
Server Socket: 93.184.216.34:443
Connection (5-tuple with protocol):
(TCP, 192.168.1.100, 52431, 93.184.216.34, 443)
Multiple connections to same server use different source ports:
Browser Tab 1: (TCP, 192.168.1.100, 52431, 93.184.216.34, 443)
Browser Tab 2: (TCP, 192.168.1.100, 52432, 93.184.216.34, 443)
Browser Tab 3: (TCP, 192.168.1.100, 52433, 93.184.216.34, 443)
Each is a unique connection despite same destination!Security Note: Port scanning is a fundamental reconnaissance technique. By probing ports, attackers discover which services are running on a target. Open ports represent potential entry points. We’ll explore scanning techniques in Part II, Chapters 4 and 9.
TCP: Reliable Transport
Why Reliability Matters
IP is an unreliable protocol—packets can be lost, duplicated, corrupted, or arrive out of order. For many applications, this is unacceptable. When you download a file or load a web page, you need all the data, in the correct order, without errors.
TCP (Transmission Control Protocol) provides:
- Reliable delivery: Lost packets are detected and retransmitted
- Ordered delivery: Data arrives in the sequence it was sent
- Error detection: Corrupt data is detected and rejected
- Flow control: Prevents overwhelming slow receivers
- Congestion control: Prevents overwhelming the network
The TCP Three-Way Handshake
TCP is connection-oriented—before sending data, client and server establish a connection through the three-way handshake. This synchronizes sequence numbers and confirms both parties are ready.
TCP ThreeWay Handshake
TCP Three-Way Handshake:
═══════════════════════════════════════════════════════════════════
Client Server
│ │
│ 1. SYN │
│ Seq=100, SYN flag set │
│ "I want to connect, my ISN is 100" │
├────────────────────────────────────────────────►│
│ │
│ 2. SYN-ACK │
│ Seq=300, Ack=101, SYN+ACK flags │
│ "OK, my ISN is 300, expecting your 101" │
│◄────────────────────────────────────────────────┤
│ │
│ 3. ACK │
│ Seq=101, Ack=301, ACK flag │
│ "Got it, expecting your 301" │
├────────────────────────────────────────────────►│
│ │
│ Connection Established! │
│◄═══════════════════════════════════════════════►│
│ │
Why this matters:
- Synchronizes initial sequence numbers (ISN)
- Confirms both sides can send and receive
- ISNs should be random (security against prediction attacks)What each step does:
-
SYN (Synchronize): Client sends a segment with the SYN flag set and its Initial Sequence Number (ISN). This ISN should be random to prevent sequence number prediction attacks.
-
SYN-ACK: Server responds with SYN and ACK flags, its own ISN, and an acknowledgment of the client’s ISN+1.
-
ACK: Client acknowledges the server’s ISN+1. Connection is now established.
Security Note: The three-way handshake is vulnerable to SYN flood attacks. An attacker sends many SYN packets with spoofed source addresses. The server allocates resources for each half-open connection and waits for the final ACK that never comes, eventually exhausting memory. See Part II, Chapter 4.
TCP Header Structure
TCP Header Format (20 bytes minimum, up to 60 with options)
TCP Header Format (20 bytes minimum, up to 60 with options):
═══════════════════════════════════════════════════════════════════
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤
│ Source Port │ Destination Port │
├───────────────────────────────┼───────────────────────────────┤
│ Sequence Number │
├───────────────────────────────────────────────────────────────┤
│ Acknowledgment Number │
├───────┬───────┬─┬─┬─┬─┬─┬─┬─┬─┬───────────────────────────────┤
│Data │ │C│E│U│A│P│R│S│F│ │
│Offset │ Res │W│C│R│C│S│S│Y│I│ Window │
│4 bits │ 4 bits│R│E│G│K│H│T│N│N│ 16 bits │
├───────┴───────┴─┴─┴─┴─┴─┴─┴─┴─┼───────────────────────────────┤
│ Checksum │ Urgent Pointer │
├───────────────────────────────┴───────────────────────────────┤
│ Options (if Data Offset > 5) │
├───────────────────────────────────────────────────────────────┤
│ Data │
└───────────────────────────────────────────────────────────────┘Key fields:
| Field | Size | Description |
|---|---|---|
| Source Port | 16 bits | Sending application’s port |
| Destination Port | 16 bits | Receiving application’s port |
| Sequence Number | 32 bits | Position of first data byte in this segment |
| Acknowledgment Number | 32 bits | Next expected byte from sender |
| Data Offset | 4 bits | Header length in 32-bit words |
| Flags | 9 bits | Control bits (see below) |
| Window | 16 bits | Receive buffer space available |
| Checksum | 16 bits | Error detection |
| Urgent Pointer | 16 bits | Urgent data offset (rarely used) |
TCP Flags and their meanings:
| Flag | Name | Purpose | Security Relevance |
|---|---|---|---|
| SYN | Synchronize | Initiate connection | SYN floods, scanning |
| ACK | Acknowledge | Acknowledgment field valid | Session hijacking |
| FIN | Finish | Sender finished sending | Connection teardown |
| RST | Reset | Abort connection | RST injection attacks |
| PSH | Push | Deliver data immediately | - |
| URG | Urgent | Urgent pointer valid | Rarely used, some evasion |
| ECE | ECN Echo | Congestion notification | - |
| CWR | Congestion Window Reduced | Sender slowing down | - |
Sequence Numbers and Acknowledgments
TCP tracks data using byte sequence numbers. Each byte of data has a sequence number, and acknowledgments indicate the next expected byte.
TCP Data Transfer with Sequence Numbers
TCP Data Transfer with Sequence Numbers:
═══════════════════════════════════════════════════════════════════
Client sends 1000 bytes:
┌─────────┬──────────────────────────────────────────────────────┐
│ SEQ=100 │ Data: 1000 bytes │
│ │ (Bytes 100-1099) │
└─────────┴──────────────────────────────────────────────────────┘
│
▼
Server acknowledges:
┌──────────┬─────────────────────────────────────────────────────┐
│ ACK=1100 │ "I received up to byte 1099" │
│ │ "Send me byte 1100 next" │
└──────────┴─────────────────────────────────────────────────────┘
│
▼
Client sends more:
┌──────────┬─────────────────────────────────────────────────────┐
│ SEQ=1100 │ Data: 500 bytes │
│ │ (Bytes 1100-1599) │
└──────────┴─────────────────────────────────────────────────────┘
│
▼
Server acknowledges:
┌──────────┬─────────────────────────────────────────────────────┐
│ ACK=1600 │ "Got it, send byte 1600 next" │
└──────────┴─────────────────────────────────────────────────────┘This mechanism enables:
- Loss detection: If an ACK doesn’t arrive in time, retransmit
- Duplicate detection: Receiving same sequence number twice indicates duplicate
- Ordering: Reassemble out-of-order segments using sequence numbers
Flow Control: Sliding Window
TCP’s sliding window mechanism prevents a fast sender from overwhelming a slow receiver. The receiver advertises its available buffer space in the Window field of each ACK.
TCP Sliding Window
TCP Sliding Window:
═══════════════════════════════════════════════════════════════════
Sender's view of sequence number space:
◄─── Sent & ACKed ──►◄─── Sent, not ACKed ──►◄── Can send ──►
┌────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┐
│ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │ 9 │ 10 │ 11 │ 12 │
└────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┘
▲ ▲
│ │
Window start Window end
(Last ACK received) (Start + Window size)
Window shrinks when receiver buffer fills:
Receiver: "Window = 0" (Stop sending, I'm full!)
Window grows when receiver processes data:
Receiver: "Window = 8000" (OK, you can send more)Congestion Control
Beyond flow control (protecting the receiver), TCP implements congestion control to protect the network. The internet would collapse without it—every sender would blast traffic until packets dropped everywhere.
TCP Congestion Control Phases
TCP Congestion Control Phases:
═══════════════════════════════════════════════════════════════════
cwnd Packet loss!
(KB) cwnd cut in half
│ │
64├ ●──────────┘
│ ╱│╲
32├ ─ ─ ─ ─ ─ ─ ─ ─ ─●─ ─ ─ ─ ╱─ │ ╲ ─ ─ ─ ─ ─ ssthresh
│ ╱│ ╱ │ ╲
16├ ╱ │ ╱ ▼ ╲
│ ╱ │ ╱ ╲
8├ ╱ │ ╱ Slow ╲
│ ╱ │ ╱ Start ╲ ╱
4├ ╱ │ ╱ (again) ●──────╱
│ ╱ │ ╱ ╱
2├ ╱ │╱ ╱
│ ╱ ╱ ╱
1├────● ╱ ╱
│ │ ╱ ╱
└────┴───────────┴───────────────────────┴────────────► Time
│ │ │
│ │ │
Slow Start Congestion Congestion
(exponential) Avoidance Avoidance
1→2→4→8→16 (linear +1/RTT) (resumes)
Phase 1: SLOW START - cwnd doubles each RTT (exponential)
Phase 2: CONGESTION AVOID - cwnd += 1 MSS per RTT (linear)
Phase 3: LOSS DETECTED - cwnd cut to half (or to 1 for timeout)
Phase 4: RECOVERY - Fast recovery or slow start again
Algorithms:
- Slow Start: cwnd doubles every RTT (exponential growth)
- Congestion Avoidance: cwnd increases by 1 MSS per RTT (linear)
- Fast Retransmit: Retransmit on 3 duplicate ACKs
- Fast Recovery: Don't start over after fast retransmitModern congestion control algorithms:
| Algorithm | Description | Use Case |
|---|---|---|
| Reno | Classic, conservative | Legacy |
| CUBIC | Default Linux, aggressive | General internet |
| BBR | Google, bandwidth-based | High bandwidth, lossy |
| DCTCP | Data center optimized | Low latency DC |
THINK ABOUT IT
Why is congestion control a “tragedy of the commons” problem? What would happen if some senders ignored congestion control while others implemented it faithfully?
TCP Connection Termination
Connections are terminated gracefully using a four-way process:
TCP Connection Termination (4way)
TCP Connection Termination (4-way):
═══════════════════════════════════════════════════════════════════
Client Server
│ │
│ 1. FIN │
│ "I'm done sending" │
├────────────────────────────────────────────────►│
│ │
│ 2. ACK │
│ "OK, got your FIN" │
│◄────────────────────────────────────────────────┤
│ │
│ (Server may still send data) │
│ │
│ 3. FIN │
│ "I'm done sending too" │
│◄────────────────────────────────────────────────┤
│ │
│ 4. ACK │
│ "OK, goodbye" │
├────────────────────────────────────────────────►│
│ │
│ TIME_WAIT (2*MSL) │
│ Client waits before fully closing │
│ (handles delayed/duplicate packets) │
│ │
/ /
Or abrupt termination:
│ RST │
│ "Connection killed immediately" │
├────────────────────────────────────────────────►│Security Note: RST packets can be injected by attackers who can guess sequence numbers, causing connection disruption. This is the basis of TCP Reset attacks. See Part II, Chapter 4.
UDP: Speed Over Reliability
When Reliability Isn’t Needed
Not all applications need TCP’s guarantees. Video streaming can tolerate occasional lost frames—retransmitting would cause delays worse than the loss. DNS queries are simple request-response pairs that can be easily retried. Real-time games need low latency more than perfect reliability.
UDP (User Datagram Protocol) provides minimal transport services:
- Port-based multiplexing
- Optional checksum
- That’s it.
No connections, no acknowledgments, no retransmission, no ordering, no flow control. Applications that use UDP must handle these concerns themselves if needed.
UDP Header Structure
The UDP header is remarkably simple—just 8 bytes:
UDP Header Format (8 bytes)
UDP Header Format (8 bytes):
═══════════════════════════════════════════════════════════════════
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤
│ Source Port │ Destination Port │
├───────────────────────────────┼───────────────────────────────┤
│ Length │ Checksum │
├───────────────────────────────┴───────────────────────────────┤
│ Data │
└───────────────────────────────────────────────────────────────┘
Compare to TCP's 20+ bytes header!
UDP prioritizes speed over features.| Field | Size | Description |
|---|---|---|
| Source Port | 16 bits | Sending application’s port |
| Destination Port | 16 bits | Receiving application’s port |
| Length | 16 bits | Total datagram size (header + data) |
| Checksum | 16 bits | Optional in IPv4, mandatory in IPv6 |
UDP Use Cases
| Application | Why UDP? |
|---|---|
| DNS | Simple queries; easy to retry if lost |
| DHCP | Bootstrap protocol; TCP handshake impractical |
| VoIP/Video calling | Real-time; latency more important than perfection |
| Video Streaming | Loss tolerable; retransmission would cause buffering |
| Online Gaming | Low latency critical; application handles reliability |
| NTP | Time synchronization; simple request-response |
| QUIC | Builds own reliability on top of UDP |
| IoT/Sensors | Resource-constrained devices |
UDP Security Considerations
UDP’s connectionless nature creates security challenges:
Source Address Spoofing: UDP has no handshake to verify the source address. Attackers can easily send UDP packets with forged source IPs.
Amplification Attacks: Some UDP services respond with much more data than the request. Attackers send small requests with spoofed source IPs; the large responses flood the victim.
UDP Amplification Attack
UDP Amplification Attack:
═══════════════════════════════════════════════════════════════════
Attacker
│
1. Small UDP request │ Source IP: Victim (spoofed!)
to vulnerable server │ "Get me all your DNS records"
(e.g., DNS query) ▼
┌───────────────┐
│ DNS Server │
│ (Reflector) │
└───────┬───────┘
│
2. Large UDP response │ "Here's 50x more data than
to spoofed source │ you asked for!"
(the victim) ▼
┌───────────────┐
│ Victim │ ← Flooded with responses
└───────────────┘
Amplification factors:
DNS: 28-54x
NTP: 556x (monlist)
Memcached: 51,000x (!!)
SSDP: 30xSecurity Note: UDP-based DDoS attacks are a major threat. DNS amplification, NTP amplification, and memcached amplification have powered some of the largest attacks in history (over 2 Tbps). See Part II, Chapter 7.
QUIC: The Modern Transport
Why QUIC Was Created
TCP served the internet well for decades, but its design assumptions don’t match modern reality:
| TCP Limitation | Impact | QUIC Solution |
|---|---|---|
| Handshake latency | 1-3 RTT to start | 0-1 RTT |
| Head-of-line blocking | One lost packet stalls all data | Independent streams |
| Ossification | Middleboxes interfere with changes | Encrypted transport |
| No built-in encryption | TLS added separately | TLS 1.3 integrated |
| Connection tied to IP | Mobile networks break connections | Connection IDs |
QUIC Architecture
QUIC vs Traditional Stack
QUIC vs Traditional Stack:
═══════════════════════════════════════════════════════════════════
Traditional HTTPS: QUIC/HTTP3:
────────────────── ────────────
┌─────────────────┐ ┌─────────────────┐
│ HTTP/2 │ │ HTTP/3 │
├─────────────────┤ ├─────────────────┤
│ TLS │ │ │
├─────────────────┤ │ QUIC │
│ TCP │ │ │
├─────────────────┤ ├─────────────────┤
│ IP │ │ UDP │
└─────────────────┘ ├─────────────────┤
│ IP │
└─────────────────┘
QUIC combines:
- Transport (reliability, flow control)
- Encryption (TLS 1.3)
- Multiplexing (streams)
- Connection migration
All encrypted, no middlebox interference!QUIC Key Features
1. Faster Connection Establishment
TCP TLS 1.3 QUIC
TCP + TLS 1.3: QUIC:
══════════════ ═════
Client Server Client Server
│ SYN │ │ │
├──────────────►│ │ Initial │
│ SYN-ACK │ │ (crypto) │
│◄──────────────┤ ├──────────────►│
│ ACK │ │ Handshake │
├──────────────►│ │◄──────────────┤
│ │ │ ACK+Data │
│ ClientHello │ ├──────────────►│
├──────────────►│ │ │
│ ServerHello │
│ + cert + etc │ 1 RTT (or 0 with resumption!)
│◄──────────────┤
│ Finished │
├──────────────►│
│ │
3 RTT before data2. Stream Multiplexing Without Head-of-Line Blocking
HTTP/2 over TCP (Headofline blocking)
HTTP/2 over TCP (Head-of-line blocking):
════════════════════════════════════════
Stream 1: ████░░░░░░ (waiting for lost packet)
Stream 2: ████░░░░░░ (blocked even though its data arrived!)
Stream 3: ████░░░░░░ (blocked even though its data arrived!)
│
└── One lost packet blocks ALL streams
QUIC (Independent streams):
═══════════════════════════
Stream 1: ████░░░░░░ (waiting for lost packet)
Stream 2: ████████ (continues normally!)
Stream 3: ██████████ (continues normally!)
│
└── Loss only affects the specific stream3. Connection Migration
Mobile scenario
Mobile scenario:
════════════════
With TCP:
Phone on WiFi ────────────► Server
(IP: 192.168.1.50) │
│
Phone moves to cellular │
(IP: 10.0.0.50) │
│
Connection BROKEN! ✗ │
New 3-way handshake needed │
With QUIC:
Phone on WiFi ────────────► Server
(Connection ID: ABC123) │
│
Phone moves to cellular │
(Same Connection ID) │
│
Connection CONTINUES! │
Zero interruption │QUIC Security Implications
Advantages:
- All packets encrypted (except first byte)
- No cleartext metadata for middleboxes to inspect
- TLS 1.3 integration = modern cryptography
- Connection IDs prevent tracking by IP
Challenges:
- Encrypted traffic harder to inspect for security
- Traditional firewalls/IDS less effective
- Network troubleshooting more difficult
- New protocol = new potential vulnerabilities
PRO TIP
QUIC uses UDP port 443. If you’re blocking QUIC, block UDP 443. However, browsers will fall back to TCP, so this may not achieve your goal. Better to use application-layer controls.
TCP vs UDP vs QUIC Comparison
| Feature | TCP | UDP | QUIC |
|---|---|---|---|
| Connection | Yes (3-way) | No | Yes (0-1 RTT) |
| Reliability | Guaranteed | Best effort | Guaranteed |
| Ordering | Preserved | Not guaranteed | Per-stream |
| Flow Control | Yes | No | Yes |
| Congestion Control | Yes | No | Yes |
| Encryption | Optional (TLS) | No | Built-in (TLS 1.3) |
| Header Size | 20+ bytes | 8 bytes | Variable (encrypted) |
| Head-of-line Blocking | Yes | N/A | No |
| Connection Migration | No | N/A | Yes |
| Use Cases | Web, email, file transfer | Streaming, DNS, gaming | HTTP/3, modern apps |
Port Scanning: Reconnaissance Primer
Port scanning probes a target to discover which services are running. Different scan types use TCP and UDP behavior in clever ways.
TCP Connect Scan
The simplest scan—complete the three-way handshake. If it succeeds, the port is open.
TCP Connect Scan
TCP Connect Scan:
═══════════════════════════════════════════════════════════════════
Port OPEN: Port CLOSED:
────────── ────────────
Scanner Target Scanner Target
│ SYN │ │ SYN │
├──────────────►│ ├──────────────►│
│ SYN-ACK │ │ RST │
│◄──────────────┤ OPEN │◄──────────────┤ ✗ CLOSED
│ ACK │ │ │
├──────────────►│
│ RST │
├──────────────►│
(Close connection)Pros: Reliable, works without special privileges Cons: Easily logged (complete connections are suspicious)
TCP SYN Scan (Half-Open)
Send SYN, but don’t complete the handshake. Less likely to be logged.
TCP SYN Scan (Stealthy)
TCP SYN Scan (Stealthy):
═══════════════════════════════════════════════════════════════════
Port OPEN: Port CLOSED:
────────── ────────────
Scanner Target Scanner Target
│ SYN │ │ SYN │
├──────────────►│ ├──────────────►│
│ SYN-ACK │ │ RST │
│◄──────────────┤ OPEN │◄──────────────┤ ✗ CLOSED
│ RST │ │ │
├──────────────►│
(Never complete handshake)
No full connection = less likely to be logged
(But modern IDS will still detect this)Pros: Stealthier, faster Cons: Requires raw socket privileges (root/admin)
Other TCP Scan Types
| Scan Type | Packet Sent | Open Port Response | Closed Port Response |
|---|---|---|---|
| FIN | FIN flag only | No response | RST |
| XMAS | FIN+PSH+URG | No response | RST |
| NULL | No flags | No response | RST |
| ACK | ACK flag only | RST | RST (but different) |
| Window | ACK flag | Different window values | - |
These “stealth” scans exploit RFC-defined behavior: closed ports should respond to invalid packets with RST, while open ports should silently ignore them. Modern firewalls often detect these.
UDP Scanning
UDP scanning is trickier—there’s no handshake to detect open ports:
UDP Scan
UDP Scan:
═══════════════════════════════════════════════════════════════════
Port OPEN: Port CLOSED:
────────── ────────────
Scanner Target Scanner Target
│ UDP packet │ │ UDP packet │
├──────────────►│ ├──────────────►│
│ (silence) │ │ ICMP Port │
│ │ ? OPEN/FILTERED │ Unreachable │
│ │ │◄──────────────┤ ✗ CLOSED
No response = open OR filtered (can't tell!)
ICMP response = definitely closed
UDP scans are SLOW (ICMP rate limiting)Security Note: Port scanning is covered extensively in Part II, Chapter 9. Tools like Nmap automate these techniques and add OS fingerprinting, service version detection, and more.
Practical Commands
# View listening ports and connections
ss -tuln # Linux (modern, faster)
netstat -tuln # Linux (legacy)
netstat -an # macOS/Windows
# View established connections
ss -t state established # Linux
netstat -an | grep ESTABLISHED
# Check specific port
nc -zv host port # Test TCP port
nc -zuv host port # Test UDP port
# Capture TCP traffic
tcpdump -i eth0 tcp port 80
sudo tcpdump -i any 'tcp[tcpflags] & (tcp-syn) != 0' # SYN packets only
# Watch TCP connection states
watch -n 1 'ss -t'
# View per-connection statistics
ss -ti # Detailed TCP info with RTT, cwnd, etc.
# Check for QUIC traffic
tcpdump -i eth0 'udp port 443'
# Nmap port scanning (covered in Part II)
nmap -sT target # TCP connect scan
nmap -sS target # SYN scan (requires root)
nmap -sU target # UDP scan (slow)
nmap -sV target # Service version detectionTRY IT YOURSELF
Compare TCP states during a web request:
# Terminal 1: Watch connections watch -n 0.5 'ss -t state all | grep :443' # Terminal 2: Make a request curl https://example.comYou’ll see the connection go through SYN-SENT, ESTABLISHED, and TIME-WAIT states.
Key Takeaways
-
Ports identify applications; combined with IPs, sockets identify connection endpoints
-
TCP provides reliable, ordered, connection-oriented communication through handshakes, acknowledgments, and retransmission—but has latency costs
-
UDP provides fast, connectionless communication with minimal overhead—but no reliability guarantees
-
QUIC combines the best of both: reliability like TCP, speed improvements, built-in encryption, and stream multiplexing without head-of-line blocking
-
Congestion control prevents network collapse; different algorithms optimize for different scenarios
-
Port scanning exploits TCP/UDP behavior to discover services—understanding the protocols helps understand the scans
Self-Assessment
-
Comprehension: Why does TCP need a three-way handshake while UDP doesn’t need any handshake?
-
Application: A video streaming service experiences buffering when packets are lost. Should they use TCP, UDP, or QUIC? Why?
-
What if: If an attacker can observe (but not modify) your TCP traffic, what information can they learn? What if the traffic is QUIC?
Review Questions
- How does TCP’s three-way handshake work, and why is it necessary?
- What advantages does UDP offer over TCP, and when would you choose UDP?
- How do sequence numbers and acknowledgments enable reliability in TCP?
- Why are SYN floods effective, and what resource do they exhaust?
- What problems does QUIC solve that TCP couldn’t?
- How does a TCP SYN scan differ from a TCP connect scan?
Key RFCs
- RFC 9293 - Transmission Control Protocol (TCP) - Latest consolidation
- RFC 768 - User Datagram Protocol (UDP)
- RFC 9000 - QUIC: A UDP-Based Multiplexed and Secure Transport
- RFC 9001 - Using TLS to Secure QUIC
- RFC 9114 - HTTP/3
- RFC 6335 - IANA Procedures for Port Assignment