OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 12:41 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 15 posts ] 
Author Message
 Post subject: [Solved] Passing network traffic from macOS to QEMU
PostPosted: Sat Jan 16, 2021 10:07 am 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
Hello!

Apologies for any errors in terminology or missing relevant info, as I'm quite new to networking in general.

I'm writing an RTL8139 driver in my OS, and am having a truly hard time getting the TAP device working correctly on macOS on WiFi with QEMU v5.2.0. In brief: I can transmit packets and they are visible on the "tap0" interface, but do not receive any packets.

I've set up a bridge with my main network interface, en0, and the TAP device, tap0. "bridge0" is used for Thunderbolt connectivity, so I've created "bridge2" for my purposes:

QEMU config:
Code:
-nic tap,model=rtl8139,script=./qemu-ifup.sh,downscript=./qemu-ifdown.sh,id=u1 -object filter-dump,id=f1,netdev=u1,file=dump.dat


qemu-ifup.sh:
Code:
#!/bin/bash
# https://gist.github.com/EmbeddedAndroid/6572715
# https://gist.github.com/artembeloglazov/db8c16efc91443955fca
sysctl -w net.link.ether.inet.proxyall=1
sysctl -w net.inet.ip.forwarding=1
sysctl -w net.inet.ip.fw.enable=1
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.link.tap.user_open=1
sysctl -w net.link.tap.up__on__open=1
ifconfig bridge2 create
ifconfig bridge2 inet 192.168.13.1/24
ifconfig bridge2 addm en0 addm $1
ifconfig bridge2 up
ifconfig en0 up
ifconfig $1 up


I am in fact able to transmit packets within my OS, and these packets are visible in Wireshark when snooping the tap0 interface (though the packet only contains 0x41 repeated at the moment!). The issue is that QEMU is not receiving any packets, even though a wide breadth of network traffic appears in Wireshark or tcpdump. I have recompiled QEMU with the debug flag for this driver enabled, and I don't get any extra output except while initializing the driver or when I transmit a packet.

I've scoured a lot of info online about setting this up and have yet to resolve it. I saw someone imply that the WiFi card in MacBooks simply doesn't support this, so wanted to check-in here to see if it is, in fact, possible.

Does anyone have any relevant experience they can throw in? Thanks very much!

_________________
www.github.com/codyd51/axle.git


Last edited by codyd51 on Sun Jan 31, 2021 7:44 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Sat Jan 16, 2021 4:05 pm 
Offline
Member
Member

Joined: Mon Dec 07, 2020 8:09 am
Posts: 212
The tap setup with qemu is a bit tricky, but it is easy to verify by answering this simple question:

Does your tap setup work if you run a "known good guest"?

like, run some liveCD of Linux and try to ssh into it from the host through your tap.


Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Sun Jan 17, 2021 1:59 pm 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
xeyes wrote:
The tap setup with qemu is a bit tricky, but it is easy to verify by answering this simple question:

Does your tap setup work if you run a "known good guest"?


Good idea! Thank you. I tried it and was unable to get it to work, but this test exists at the intersection of two things I'm unfamiliar with (network routing and Linux), so I don't take that as conclusive!

The plot thickens (from https://support.apple.com/en-au/guide/m ... h43557/mac):

Quote:
Note: You can’t use bridging if the physical network device used for bridging is a wireless device.


I also see that the TAP device is assigned a random MAC when it's opened by QEMU (https://sourceforge.net/p/tuntaposx/cod ... ap.cc#l150), and that the emulated NIC created by QEMU has a different MAC. Just as a test, I'm now re-assigning the MAC of the TAP in the qemu-up script to match the QEMU NIC.

For now, I've purchased a USB WiFi adapter that arrives tomorrow, and will try doing all this with an ethernet cable plugged between my router and Macbook.

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Sun Jan 17, 2021 2:23 pm 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
No dice when using an ethernet cable either.

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Sun Jan 17, 2021 10:49 pm 
Offline
Member
Member

Joined: Mon Dec 07, 2020 8:09 am
Posts: 212
codyd51 wrote:
xeyes wrote:
The tap setup with qemu is a bit tricky, but it is easy to verify by answering this simple question:

Does your tap setup work if you run a "known good guest"?


Good idea! Thank you. I tried it and was unable to get it to work, but this test exists at the intersection of two things I'm unfamiliar with (network routing and Linux), so I don't take that as conclusive!

The plot thickens (from https://support.apple.com/en-au/guide/m ... h43557/mac):


No worries, I'm also not familiar with this but I can recommend this guide to you, which is the one that enabled me to connect to a guest (Linux, SSH or other servers).

While it is not exactly what you want (I don't think it bridges the real interface to the guest, just between host and guest), it might still be of some help to you.

https://alberand.com/host-only-networki ... visor.html

Good luck!


Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Mon Jan 25, 2021 8:33 am 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
I’ve simplified the failing case some more. If I write my own application to consume packets from the tap, I am able to successfully receive packets:

Code:
# In `sudo python3`
f = open("/dev/tap0", "r")
while True:
    r = select([f.fileno()], [], [])[0][0]
    packet = os.read(f.fileno(), 4000)
    print(packet)

# In `/bin/bash`
sudo ifconfig tap1 10.12.0.2 10.12.0.1
ping 10.12.0.1

# The Python program prints packets from the tap


However, if I spawn QEMU instead of my python3 script and run the same bash commands, QEMU doesn’t output anything about having received a packet. I am being led further to think that there is some QEMU issue receiving packets on a tap with macOS, of which I’m running Big Sur v11.0.1 and 11.1. I am going to dig into the QEMU source a bit and add some logs where possible.

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Mon Jan 25, 2021 12:11 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
You ifdown script is using "sysctl" and "ifconfig". Both are Linux specific commands, and since neither your host nor your guest is Linux, I doubt they will work correctly.
(MacOS might actually have an incompatible "ifconfig" since it's a BSD (probably with different switches and flags), but I'm 100% sure it doesn't have the same "sysctl" flags as Linux. The man page does not list "net.link.tap.user_open" for example) So are you sure those are the correct flags?

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Mon Jan 25, 2021 6:40 pm 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
bzt wrote:
You ifdown script is using "sysctl" and "ifconfig". Both are Linux specific commands, and since neither your host nor your guest is Linux, I doubt they will work correctly.
(MacOS might actually have an incompatible "ifconfig" since it's a BSD (probably with different switches and flags), but I'm 100% sure it doesn't have the same "sysctl" flags as Linux. The man page does not list "net.link.tap.user_open" for example) So are you sure those are the correct flags?


Indeed, macOS ships a clearly-different ifconfig from the one referenced by lots of discourse online: mine has different switches and different output from the (plural?) Linux one.

Similarly, as you correctly guessed, the available sysctl's are different. Some of the ones in my post, cobbled together from info online, are invalid on macOS, but sysctl simply complains about it and no-op's when an invalid name is provided. I fiddled with a few more switches to no avail after inspecting the output of "sysctl -a | grep -i net".

Currently, I'm knee-deep in qemu. I've learned that it uses glib for its event loop, and has an internal system called "aio" used to dispatch events in respond to glib wakeups. The tap's file descriptor is certainly provided to glib to poll on, but QEMU does not seem to react when new data appears on the tap. I'm still digging...

Thanks for your input :)

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
 Post subject: Re: Passing network traffic from macOS to QEMU
PostPosted: Mon Jan 25, 2021 6:45 pm 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
As another "quick test", I wrote an equivalent C program to the Python script above that opens the tap and select()'s on it. When I configure the tap via ifconfig, data appears and select() unblocks. No reaction within QEMU, but I've yet to dig deep enough into it to isolate where the wakeup should happen -- there are lots of internal systems even just in the small bit I'm touching! (notify, timers, bottomhalf, aio, iohandler... it's enough to drive anyone to sticky notes)

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
 Post subject: Re: [Solved] Passing network traffic from macOS to QEMU
PostPosted: Sun Jan 31, 2021 7:51 am 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
I've finally gotten to the bottom of this, after hacking around in QEMU for a while and speaking with a couple of very helpful QEMU contributors in their IRC.

As mentioned above, select() works fine on the http://tuntaposx.sourceforge.net driver tap device. However, QEMU (via glib) doesn't use select() to wait on file-descriptors: it uses poll(). And, as it turns out, macOS does not support poll() on devices https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/poll.2.html:

Quote:
BUGS
The poll() system call currently does not support devices.


A couple of years ago, macOS introduced a framework intended to provide support for virtualization software to pass packets to and from the host: vmnet.framework https://developer.apple.com/documentation/vmnet?language=objc.

I am now working on a QEMU patch implementing a new "netdev" backed by the native vmnet.framework. Receiving packets works well, and I'm currently adding support for fun options (like providing a MAC address instead of letting the kernel allocate one).

A bit disappointing, but I'm glad to finally understand what was going on here and what the path forward is. I've marked this post as [Solved].

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
 Post subject: Re: [Solved] Passing network traffic from macOS to QEMU
PostPosted: Sun Feb 14, 2021 8:59 pm 
Offline

Joined: Sun Feb 14, 2021 2:41 am
Posts: 1
You would’ve needed to route from the bridge to your real network interface. Pf would help with that. Don’t put the wireless interface into the bridge.


Top
 Profile  
 
 Post subject: Re: [Solved] Passing network traffic from macOS to QEMU
PostPosted: Wed Feb 17, 2021 7:43 am 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
Update: My QEMU patch to implement a macOS/vmnet network backend has been in review for a while, and I've been using it to continue building my network stack: https://lists.gnu.org/archive/html/qemu ... 04637.html

So far, I've got ARP, IP, UDP and DNS, and am about to break ground on TCP! It's so gratifying to be discovering devices on my local network.

If all goes well, this feature will be included in QEMU 6.0. I hope this patch helps someone else on macOS who was in a similar situation to me!

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
 Post subject: Re: [Solved] Passing network traffic from macOS to QEMU
PostPosted: Wed Feb 17, 2021 9:33 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
codyd51 wrote:
Update: My QEMU patch to implement a macOS/vmnet network backend has been in review for a while
Well done! That's the power of Open Source! Don't be alarmed, I'm a qemu contributor myself, I had some patches for which I had to wait months... On the other hand, others (like my ARM QA7 timer patch) was reviewed within a few weeks. No worries, they will get there eventually!

Again, well done!

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: [Solved] Passing network traffic from macOS to QEMU
PostPosted: Wed Jan 19, 2022 6:22 pm 
Offline

Joined: Mon Jul 05, 2021 7:05 am
Posts: 8
that help me!thanks a lot!


Top
 Profile  
 
 Post subject: Re: [Solved] Passing network traffic from macOS to QEMU
PostPosted: Wed Jan 19, 2022 8:30 pm 
Offline

Joined: Mon Jul 05, 2021 7:05 am
Posts: 8
Code:
If all goes well, this feature will be included in QEMU 6.0. I hope this patch helps someone else on macOS who was in a similar situation to me!


my qemu-system-i386 is
Code:
QEMU emulator version 6.2.0
, still has this problem, it seems the qemu does not fix this bug in 6.x version qemu!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 15 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 80 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group