OSDev.org

I've looked at some Ethernet card (NE2000 specifically) documentation on the Wiki, and I understand the concept of sending and receiving packets. But what is it exactly that I put in/read from the packets? Are they just IP packets or are they encapsulated as Ethernet frames? I'm assuming it is Ethernet frames as that would make sense, but I do not understand how I would "connect" with the device - i.e. get the IP address etc so that I can then talk to the Internet. I tried googling "ethernet protocol", "ethernet get ip", etc but I did not seem to find any information about this. Can someone give me some pointers? I'm planning to implement a TCP/IP stack soon.

You'll need ARP and DHCP

Its' packets all the way down.

TCP packets run inside IPv[46] packets which might run inside Ethernet packets, or they might run inside an 802.2 LLC header on another network (e.g. Token Ring), etc, etc.

If you're dealing with IPv4, then you also need the Ethernet ARP (Address Resolution Protocol) in order to do, well, address resolution (that is, you use ARP to ask and receive the answer to the question "What is the MAC address of 80.34.2.17"). For IPv6, the NDP (Neighbor Discovery Protocol) is used. This is a subprotocol of ICMPv6 (The IPv6 Control Message Protocol).

There are three general ways a machine gets an address:

• Manual configuration. Enough said.
• Link local addresses. For IPv6, every lower layer port has its' own link local address; for IPv4, they're the fallback when the machine isnt' statically configured and doesn't find a DHCP server. Machines basically allocate these by asking "Who is fe80::32", then after a timeout declaring "I am fe80::32." The IPv6 traditional method of picking an address is to use the network card's 64-bit EUI number (expanded from a 48-bit MAC for Ethernet) for the last 64 bits. (The first 20 bits are the link local prefix, followed by 44 zero bits)
• Route Advertisements. This is the preferred mechanism in IPv6
• The Dynamic Host Configuration Protocol. This is the common mechanism in IPv4 and also used in IPv6 where route advertisements aren't good enough.

Route Advertisements
These are ICMPv6 messages. There are two important ones:

• Route solicitation. Link local multicast. "Hello, is there a router out there?"
• Route advertisement. "Hi, I'm fe80::64 delegating prefix 2001:DB8::0000"

You send a route solicitation, hope for a route advertisement, and when you get one you claim your IP the same way as for link local addresses.

DHCP
DHCP is largely the same on IPv6 and IPv4, with subtle differences: on IPv4, there are no link local addresses, so every packet is broadcast. For IPv6, just the initial discovery packet is link local mutlcast; after that, everything is done using link local addresses. DHCP runs on UDP.

DHCP normally proceeds in 4 stages:

• DHCPDISCOVER. "Hey, is there a DHCP server out there?"
• DHCPOFFER. "Hi, I'm 2001:DB8::0001 and I offer you 2001:DB8::0002/64. Your default gateway is 2001:DB8::0003 and your DNS servers are 2001:DB8::0004 and 2001:DB8::0005."
• DHCPREQUEST. "Hi, I'd like to claim 2001:DB8::0002"
• DHCPACK "OK, you're 2001:DB8::0002 for the next 86400 seconds. Your default gateway is 2001:DB8::0003 and your DNS servers are 2001:DB8::0004 and 2001:DB8::0005" (I have no idea why it bothered including all of that lot in the DHCPOFFER...)

Miscellany
There are ways to speed up or slow down the IP acquisition process. Traditionally Linux dhclient tries your last DHCP server for 30 seconds before it gives up and does a DHCPDISCOVER, which as you might imagine sucks if you take your laptop places often.

Modern devices have a very quick process for getting an IPv4 address:

• Send out a router solicitation packet. Simultaneously ARP/NDP the last 5 used DHCP servers, pause a moment
• If one of them IPs exists on this network, try and DHCPREQUEST for our last IP. We will hopefully get an IP shortly
• After a short timeout (~200ms) broadcast DHCPDISCOVER and go through the motions

This is what I'm currently working on as we speak. Here is my recommended path...

1. Get your Ethernet address from the network card.
2. Get an IP address from a DHCP server.
3. Get the Gateway IP address from the same DHCP server.
4. Get the Ethernet address for the Gateway from ARP.
5. Get the IP address for a DNS server from the DHCP server.
6. Get the IP address for a web server from the DNS server.
7. Send the proper TCP packets to the web server to start a connection and request a web page.

I would recommend ignoring IPv6, for now. You can also combine steps 2, 3, and 5 into a single request, if you want.

Each of these steps took me a several days to get working, and I'm still finding issues here and there.

Also, I would recommend getting real comfortable with the various packet formats that you will be using.

Ethernet
IPv4
UDP
TCP
ARP
DHCP
DNS

All of these can be found on wikipedia, with field positions and descriptions. Just remember that all multi-byte fields need to be stored in reverse order. (Except Ethernet and IP address fields. Those can be left as-is.)

I haven't found any good network debugging tools, so I just wrote my own C# app to listen for packets on my local machine, and ran my OS in VirtualBox, with Port Forwarding turned on and configured.

Let us know if you run into any problems, and we'll help out where we can.

Good luck!

Edit: Probably should have mentioned... All Ethernet packets sent to machines on your local network (DHCP, ARP) need to be addressed to them, directly.(i.e. Put their Ethernet address in the Destination field in the Ethernet packet.), or broadcast to the entire network (ff:ff:ff:ff:ff:ff).

For machines outside of your network (DNS, Web Servers, etc.), send all of your Ethernet packets to the Gateway's Ethernet address. (This is why you need to do step 4 above.)

The Gateway will use the Destination IP address from the IPv4 packet to deliver the packet to its destination.

I've found that Wireshark actually works very well for this kind of thing -- you get QEMU or whatever emulator you use to write to a packet dump file and open it with Wireshark. Is there something that it doesn't do? Just curious.


Otherwise, @Owen you did a really nice job of explaining how everything works,

At the moment, I'm sort of stuck using VirtualBox, because the only network hardware that I currently have "drivers" for is VirtIO-Net devices. As far as I know, VirtualBox is the only free, standalone VM that supports VirtIO-Net devices, and as far as I know, VirtualBox will not log network packets. (But I could be wrong...)

Welp. Maybe you should try exploring the documentation for VBox online? There are a plethora of options that are only accessible via the command line, like the debugger (which is IMHO quite shity). Who knows?