This is a multi-part blog post about port scanning. In this post we're going to cover port scan theory. In the next post in this series we'll look at the practical parts of port scanning. It might seem a little dry in places, but stick with it as it's knowledge you're almost certainly going to be tested on in any professional exam. If it's not in your exam, you're in the wrong class.

Port scanning is the basic foundation of service identification within a TCP/IP network, but is generally associated with network mapping. The most popular tool covered in every single pen testing book you’ll ever read is Nmap. It’s pretty much the de facto standard port scanner. It’s not the only one out there and it’s important to understand port scanning theory so you can remain comfortable in situations where Nmap or an equivalent isn’t available, or at least is misbehaving.

If [port scanning theory] isn’t in your exam, you’re in the wrong class

Occasionally in your career you may find yourself in a situation where you don’t have access to your favourite tools, or you might be blinded by the choice between Nmap, UnicornScan, Zmap and so on. 99.9% of the time you’ll use Nmap to perform port scanning, but there are situations where you might use other tools or indeed none at all. The two most common port scan types you’ll encounter as a pentester are TCP SYN (sometimes called half-open) scans and UDP scans. Before we dive into the world of TCP scans, let’s take a look at how a TCP connection is formed.

TCP Three way handshake

The three way handshake is a term used to describe how a TCP connection is established. Imagine two hosts, Alice and Bob. Alice wants to open a TCP connection to Bob on TCP port 79 to access Bob’s finger server and see who’s online.

Alice sends a SYN to Bob

To initiate the connection, Alice sends a TCP packet with the SYN flag set to Bob and a random sequence number as shown below:

Bob Responds to Alice

Bob receives the request. If a process running on Bob is listening on TCP port 79, providing there’s no filtering in the way, it responds with a TCP packet sent back with the SYN and ACK flags set. An acknowledgement number is also included, normally set to the sequence number plus one, and a random sequence number is set.

Alice acknowledges receipt of Bob’s response

Alice receives the SYN/ACK flagged packet, checks the acknowledgement number is what was expected and responds with a new packet. This packet, with just the ACK flag is set. The sequence number is incremented and used as the acknowledgement number and a new sequence number is randomly generated. At this point both Alice and Bob have acknowledged each other’s intention to connect, and synchronized TCP sequence numbers.

From this point on until the moment the connection is torn down the two endpoints will only send TCP packets with the ACK flag set unless there’s a need to resynchronize the connection (such as when the sequence numbers fall out of synchronization).

If the port is closed

If there was no process running on Bob that listens to TCP port 79 at the point the initial SYN packet was received, the TCP/IP stack on Bob’s Operating System should respond with a TCP packet with the RST and ACK flags set as shown below.

As there’s no further expected packets in the sequence, the sequence number is normally set to zero. The acknowledgement should still be one higher than the original sequence number sent by Alice.

Other port scan responses

There are also other possible responses to the original SYN packet. If there’s a firewall between you and the target, you might find that the SYN is dropped by the firewall and never reaches the target.

Alternatively the SYN packet may reach the target and the response is firewalled, although this is a lot rarer than the former as most filtering in the real world tends to look to stop things going in rather than going out (ingress rather than egress).

Different Operating Systems also exhibit different response patterns. Just as you might expect to see differences in IP TTL values between Windows and Unix-based TCP/IP stacks, there are differences at the TCP level for initial packets.

If you want to model individual packet communication sequences, then you’ll get more out of the process by using tools such as Scapy or Hping3 in conjunction with sniffers like Wireshark or Tcpdump.

TCP Port scanning techniques

On a normal penetration test, the only TCP port scanning technique you need to consider is the SYN scan. There are circumstances where other scans may be needed, but most modern firewalls understand the different types of port scan, and many of the techniques supported by Nmap were developed primarily as temporary methods used to bypass firewall defences.

TCP SYN scan

The TCP SYN scan is the most common form of TCP port scan in use. It uses raw sockets to send a TCP packet with the SYN flag set. If a SYN/ACK is received, the port is open. If a TCP packet with the RST/ACK flags set is received, the port is closed. If nothing is received, the scanner will typically try again a certain number of times before marking the port as filtered and moving on. Because it uses raw sockets, this form of scan typically needs root privileges on Unix-based Operating Systems.

TCP SYN Port scans are like sending 65,536 slightly different knock-knock jokes to your target and not waiting for the answer.

Sockets are the programmatic method that computers use to communicate over TCP/IP networks. The easiest way to think about them is to treat them the same way as files on a computer. Most programming languages on most Operating Systems open a socket, connect the socket to a host and port and then read from or write to the socket as though it were a file descriptor or handle before closing it when done. This way the program doesn’t need to worry about maintaining state, dealing with congestion and so on.

The term raw means that the program needs more control over the socket or certain sensitive OS features (such as being able to listen on a low TCP port) than is normally available.

However, because that level of access is considered sensitive by some Operating Systems (most notably Unix-based OSes) the general convention is to restrict the ability to open and use raw sockets to processes running within the root or superuser user context.

TCP Connect scan

A TCP Connect scan creates (and tears down) a full TCP connection to each port for each successful connection. In this instance port scanners typically use standard Operating System calls to create connections rather than just send SYN packets and see what responds. This makes for a much slower scan that uses more resources compared to the more familiar SYN scan.

about TCP jokes The best thing is that out of order they can arrive - W. Yoda Stephens, Illustrate TCP/IP you must

TCP Connect scans can also lead to Denial-of-Service situations under certain conditions. Sometimes a poorly written service will try to read from or write to it’s socket descriptor after the connection has closed, which will fail as there’s nothing to access. If the underlying application can’t handle the resulting error condition it may crash, taking the associated service down with it.

TCP ACK scan

This form of scan is typically used to identify the state tracking capabilities of a firewall. A TCP ACK scan sends a TCP packet with only the ACK flag set to each target address and port.

When an ACK without a SYN is received, a system will typically assume the originating host is attempting to continue a current connection that doesn’t exist and will send back a TCP packet with the RST flag set.

If an intermediary device isn’t tracking the state of existing TCP connections, then the RST flag is passed through to the system that sent the ACK packet and the host, port and protocol combination can be considered unfiltered.

If an intermediary device is tracking the state of TCP connections there’ll be no matching TCP session in it’s internal state table and will drop the packet, optionally sending an ICMP type 3 (Destination unreachable) message to the originating system.

The corresponding code in the ICMP message sent to the originating system will vary based on systems and configuration, but typically will be one of the following:

• 1 - Host Unreachable
• 2 - Protocol Unreachable
• 3 - Port Unreachable
• 9 - Communication with destination network prohibited
• 10 - Communication with destination host prohibited
• 13 - Communication administratively prohibited

If the intermediary device returns codes 1,2 or 3 then it means that either the host, port or service can’t be reached from the firewall (and may be down or otherwise blocked further on). Codes 9,10 and 13 are basically the Firewall’s way of telling you to very politely leave it alone.

Of course if the intermediary device is configured correctly then you won’t receive anything at all regardless of what you send as the unsolicited ACK will be dropped at the firewall.

TCP Flag combination scans

As well as the ACK and SYN scans there are other scans using different combinations of flags. Rather than list them out separately, I thought I’d discuss the more common ones here. When I say common, I mean supported by Nmap rather than likely to be used in anger against modern firewalls.

Scans that abuse RFC793 like a politician with a Panamanian bank account

RFC793 has a particular loophole that allows us to determine whether or not a port is open or closed. When a packet is sent to a closed port with no RST flag set, RFC793 requires that the receiving host responds with an RST packet. This means we now know that an RFC793 compliant system will send an RST back if the port is closed regardless of what we send, if anything as long as we don’t set an RST in our outbound packet. RFC793 then goes on to say that packets sent to open ports without SYN, ACK or RST flags set should just be dropped. This means that if RFC793 compliant system has an open port there’ll be no response. This can be abused to determine whether a port is open or closed even if SYN packets are being blocked (for example by a host based firewall).

NULL Scans

A NULL scan has no flags set. For packets sent to an open port on a compliant system the packet should be dropped and for a closed port we should receive an RST packet.

FIN Scans

A FIN scan uses probes with the FIN flag set. This should solicit the same responses as a NULL scan on compliant systems, but it wasn’t always unusual for TCP/IP stack implementations to forget to implement code to catch a packet with no RFC793 flags set, meaning that packets with no flags would be handled in unexpected ways. When code was implemented to catch NULL scans, a FIN flag would often retrigger the same unintended condition as a NULL scan did previously.

Christmas tree scans

A christmas tree scan uses the FIN, PSH and URG flags and should solicit the same results as NULL and FIN scans on compliant systems.

Maimon scan

A Maimon scan (developed by Uriel Maimon) uses a FIN/ACK packet. In all cases an RST should be received, but some operating systems based on the BSD TCP/IP stack will respond only when a port is closed. Because the scan types above rely on RST packets issued by the target system, an intermediary firewall will sometimes treat these responses in the same way as the ACK scan by sending ICMP type 3 messages or dropping the packet. This can be useful for mapping out firewalls in theory, but sadly mostly limited to those built and configured in the 1990s in practice. Most modern firewalls will detect these scan types, but Operating Systems will handle these sorts of probes differently to each other depending on how they implement RFC793.

TCP Window scan

The TCP Window scan is a modified version of the ACK scan that uses the TCP Window field to determine whether or not a port is open. The TCP Window field is used as part of TCP’s flow control to manage data transmission volumes.

On some systems the response to an open port will specify a positive TCP Window size as this value is initialized within the Operating System’s TCP/IP stack even though a RST packet is being sent. On these systems a TCP Window size of 0 means that the port is closed as no data is to be transmitted.

This implementation of TCP Window management isn’t particularly common and when not supported most port scanners will mark all ports as closed, although sometimes the inverse applies.

It’s a very unreliable scan type and not one I’d expect to see someone use unless they really understood what they were doing with it, but may pop up on some final exams, usually on a written, rather than practical paper.

UDP Scanning

UDP is a stateless protocol. Unlike TCP there’s no control flow, so there are no SYN or RST flags, or state of any kind. So how do we port scan it? It’s not as hard as you think, and realistically just needs us to abuse the way IP, ICMP and UDP were designed.

I could tell you a joke about UDP, but you might not get it

In a normal UDP session a packet is sent from Alice to Bob and entirely dependent on the individual application protocol, Bob may or may not respond to Alice. There needs to be a way for Alice to know if Bob isn’t listening and that way is determined by ICMP, as shown below.

An ICMP Port Unreachable message (Type and code 3) is sent back if the port is closed. Other forms of ICMP destination unreachable message may be sent back as per the ACK scan results depending on whether the host is down, whether there’s an intermediary device blocking access and so on. But how can we tell if the port is open?

When a UDP datagram is sent to an open port it’s passed from the Operating System to the application listening on that port. The application then processes it accordingly. If the application is happy to respond to the content sent (such as with echo on UDP port 7) then it’ll do so.

Unfortunately most UDP protocols will only respond to UDP datagrams that are formatted for their use, dropping anything that doesn’t match. This is helpful because we can determine first whether a port is closed, then if a port is open or possibly filtered from the non-responses and look to send application specific requests to services that we know might respond if appropriately queried.

There is however, a catch. When a port scanner sends a datagram it will wait for a certain amount of time. After this point it will either retry the port a certain number of times or will move on to the next port.

Some Operating Systems (most notably Linux and Solaris) implement a rate limit on the number of ICMP error messages that can be sent to a specific host. This means that a port scanner can send a bunch of UDP messages to a target, wait, send another bunch and miss the ICMP responses slowly trickling back.

At best this slows down scan time considerably. In my experience it’s taken upwards of 9-18 hours to fully scan Solaris and Linux systems in some cases even on the local network.

Some port scanners like Nmap use fairly sophisticated methods to detect ICMP error rate limiting and throttle things back to try and keep under the radar. In practice there are several things you can do, which we may look at in the next post in this series.

Protocol scanning

Although this post is specifically focused on port scanning TCP and UDP services on an IPv4 network, we should look at other IP protocols too. In an IP packet the IP protocol is determined by an integer set in the packet’s Protocol field (in IPv6 this is in the next header field).

The best thing about proprietary protocol jokes is [REDACTED]

We can send combinations of blank and correctly formatted datagrams on each IP protocol number to a target host to see what protocols are supported. In some cases we can also use this to bypass incorrectly configured firewalls. For any responses received it’s worth investigating further, but the exact steps required will vary by OS and network. Nmap can perform IP protocol scanning, but if you’re going to look into it, be prepared to break out Wireshark to find ways of getting better results.

Conclusions

Port scanning seems straightforward, but under the hood a huge amount of things happen to determine whether a port is open or closed. If you’re serious about penetration testing you really need to understand TCP/IP at an extremely low level in order to abuse it more effectively. Port scanning theory crops up in every exam worth taking, and rightly so, so it’s important to understand at least the basic SYN and UDP techniques.

In this post we looked at:

• The TCP three-way handshake
• TCP SYN, Connect, ACK and RST scan variants
• UDP Port scanning
• A little bit about IP protocol scanning

In part two I’ll cover TCP port scans. If you’ve enjoyed the post, or have any thoughts, comments or suggestions, please contact me as @stevelord on twitter. If you’d like to get notice of new posts and useful security articles I find, subscribe to my mailing list below.