TCP, UDP & the Handshake

Every connection your browser makes, every reverse shell that calls home, every packet nmap sends - all of it runs on either TCP or UDP. Understanding exactly what happens at the transport layer is what separates someone who runs tools from someone who understands why they work.

TCP vs UDP: The Core Distinction

The Transport Layer (layer 4 of OSI, layer 2 of TCP/IP) handles end-to-end delivery between applications. Two protocols dominate: TCP and UDP. They make fundamentally different promises.

The mental model: TCP is a phone call - you establish a connection, both sides confirm they're ready, data flows reliably, and you hang up properly. UDP is shouting into a room - you broadcast your message and hope the right people hear it. Nobody confirms receipt.

Ports and Sockets

A port is a 16-bit number (0-65535) that identifies a specific process or service on a host. The IP address gets you to the machine; the port gets you to the right application.

• Well-known ports: 0-1023 - assigned by IANA, require root to bind on Linux (e.g., port 80, 443, 22)
• Registered ports: 1024-49151 - registered use (e.g., MySQL on 3306, RDP on 3389)
• Dynamic/ephemeral ports: 49152-65535 - OS assigns these as source ports for outbound connections

A socket is the combination of IP address + protocol + port that uniquely identifies one end of a connection:

Socket = Protocol + IP Address + Port
Example: TCP 192.168.1.100:54322  ↔  TCP 93.184.216.34:443

A connection is a pair of sockets. The server listens on a fixed port (e.g., 443). Your OS assigns an ephemeral source port (e.g., 54322). The tuple (src IP, src port, dst IP, dst port) uniquely identifies every active connection.

The TCP Three-Way Handshake

Before any data flows over TCP, the two endpoints perform a three-way handshake to establish a connection. It synchronizes sequence numbers so both sides can track and reorder segments.

Client                          Server
  |                               |
  |──── SYN (seq=x) ─────────────>|   "I want to connect; my starting seq is x"
  |                               |
  |<─── SYN-ACK (seq=y, ack=x+1)─|   "OK; my starting seq is y; I got your x"
  |                               |
  |──── ACK (ack=y+1) ───────────>|   "Got it; your seq acknowledged"
  |                               |
  |   [Connection established]    |
  |<══ Data flows both ways ════>|

The flags in TCP headers that drive this:

• SYN (Synchronize): initiate a connection, share starting sequence number
• ACK (Acknowledge): acknowledge receipt of a segment
• FIN (Finish): gracefully close a connection
• RST (Reset): immediately terminate a connection (also: "port is closed")
• PSH (Push): send data immediately rather than buffering
• URG (Urgent): urgent pointer is valid

Connection teardown is a four-way process (FIN, ACK, FIN, ACK) - but that detail matters more to you as a developer than as an attacker. What matters offensively is what happens when the handshake is interrupted.

How nmap's SYN Scan Exploits the Handshake

The nmap SYN scan (nmap -sS) is the default scan for root-level nmap runs. It's sometimes called a "half-open" scan because it never completes the three-way handshake. Here's exactly how it works:

Step 1: nmap sends a SYN

nmap crafts a raw TCP packet with the SYN flag set, targeting the port being probed. It doesn't use the OS's TCP stack - it constructs the packet directly (requires raw socket privileges, hence needing root/sudo).

Step 2: Read the response

The server's response reveals the port state:

Open port:

nmap → SYN ──────────────> target:80
nmap ← SYN-ACK ──────────< target:80   (the service is listening, accepted the SYN)
nmap → RST ──────────────> target:80   (nmap kills the half-open conn - never sends ACK)

Closed port:

nmap → SYN ──────────────> target:81
nmap ← RST-ACK ──────────< target:81   (port is closed; OS responds with RST)

Filtered port:

nmap → SYN ──────────────> target:82
                            [no response / ICMP unreachable]

A firewall drops the packet silently (filtered) or sends an ICMP "port unreachable" message.

Why SYN scan is preferred

1. Stealth: The connection is never completed, so many older services never log it (no full connection = no application-level log entry). Modern IDS/SIEM do detect half-open scans, but legacy syslog-based monitoring often doesn't.
2. Speed: No teardown handshake needed - nmap fires SYNs and reads responses without holding state. This enables the massive parallelism that makes nmap fast.
3. Accuracy: SYN-ACK vs RST vs silence gives clean open/closed/filtered signal.

The connect scan (-sT)

When nmap runs without root, it falls back to the connect scan (-sT), which uses the OS's connect() system call to attempt a full three-way handshake. This is more detectable (full connection established, application logs it) but works without raw socket privileges.

Filtered vs Closed: The Firewall Fingerprint

The difference between filtered and closed is operationally important:

• Closed (RST reply): the OS is reachable; that specific port has nothing listening. Good news for the attacker: the host is alive and responsive.
• Filtered (no reply or ICMP unreachable): a firewall or ACL is blocking the packet before it even reaches the host's TCP stack. Silence is the signature of a stateful firewall drop rule.

Key Takeaways

• TCP is connection-oriented and reliable; UDP is connectionless and fast. Know which protocol a service uses - it determines how you enumerate it.
• A socket is the IP + port + protocol triple that identifies one end of a communication channel.
• The three-way handshake (SYN → SYN-ACK → ACK) establishes TCP connections and synchronizes sequence numbers.
• nmap's SYN scan deliberately never completes the handshake - send SYN, read the reply (SYN-ACK = open, RST = closed, silence = filtered), then reset.
• Open means a service is listening. Closed means the host is up but nothing is there. Filtered means a firewall is blocking the probe - a firewall, not absence.
• SYN scan requires root/sudo because it constructs raw packets directly; connect scan (-sT) uses the OS TCP stack and needs no special privileges.