OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 9:27 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 14 posts ] 
Author Message
 Post subject: USB Help
PostPosted: Sat Apr 15, 2017 9:33 am 
Offline

Joined: Wed Feb 08, 2017 7:51 pm
Posts: 22
Hi. I have gotten to a stage in my os where I think I want to actually write to disk space and create a file architecture. The only problem is that I boot of a USB thumb stick for convinence (instead of a CD or something, or worse re-imaging your HDD every time you make changes). USB is (as I have discovered) a extremely complicated interface, and all of you on this site have shouted in the loudest possible way that you don't want to teach ANYTHING, so I was wondering if you have any good links or tutorials that explain the USB in a very specific way (not too general). At a minimum, could you provide a link to the manual? Thanks.


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sat Apr 15, 2017 10:44 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4591
Location: Chichester, UK
Ben Lunt's book, mentioned in this thread viewtopic.php?f=1&t=30636&hilit=Usb+book is a good starting point. Don't expect it to be easy. But it can be done.


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sat Apr 15, 2017 1:03 pm 
Offline
Member
Member
User avatar

Joined: Sat Nov 22, 2014 6:33 pm
Posts: 934
Location: USA
interruption wrote:
Hi. I have gotten to a stage in my os where I think I want to actually write to disk space and create a file architecture. The only problem is that I boot of a USB thumb stick for convinence (instead of a CD or something, or worse re-imaging your HDD every time you make changes). USB is (as I have discovered) a extremely complicated interface, and all of you on this site have shouted in the loudest possible way that you don't want to teach ANYTHING, so I was wondering if you have any good links or tutorials that explain the USB in a very specific way (not too general). At a minimum, could you provide a link to the manual? Thanks.

USB is quite complicated until you understand what is going on. As iansjack stated, I have written a book that explains this in detail, from a beginners point of view. It details each controller and a few common devices and how to communicate with them.

As for the "don't want to teach anything" comment. I have to disagree. There are many people on this forum that are more than eager to help, including myself. I think what happens is that a beginner, or even a not so beginner, posts a vague question, with very little details, expecting a very detailed answer.

For example, if I posted a question of, "how do I fix the fourth wheel on the second row of my Turing Machine made from Legos", what would your response be? Then if I was not pleased with your answer, I would probably get a little upset. That seems to be the theme around here.

We are more than eager to help if you are willing to put in the effort to explain your needs. For example,

Quote:
Hi guys,
I have a Turing Machine made out of Legos, <here> is a picture. The third wheel on the second row fell off. <here> is a picture of the absent piece
with a blow up of the mounting shaft. Does any one have any familiar knowledge of this and if so, could someone please have a look and see what
I might be able to do to fix the shaft? Please let me know if you need more details.

Get the idea? Many people are here to help, including myself, as long as the person asking the questions is willing to put in the effort to help us help you.

I am not directing these comments toward you alone, please don't take it as so. I am directing these comments to those that have visited this group and have in some way had their feelings hurt because we didn't help them enough.

Back to the mentioned book. I strongly recommend the printed version, print is so much easier to see and there is space in the margins and bottom of each page to write notes. Once you have received the book, follow the directions within it to get the CDROM which includes source code to the mentioned utilities and examples. Once you have tried some code yourself and can't quite get it, be sure to post here, with details, and we will try to help.

Please note that we will not write code for you. Please do not post "I need code to put that wheel back on my Turing Machine!". You may post your code and we will look at it and see if we can find where you have made a mistake, but we won't write it for you.

Thank you,
Ben
http://www.fysnet.net/the_universal_serial_bus.htm


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sat Apr 15, 2017 1:03 pm 
Offline
Member
Member
User avatar

Joined: Sun Sep 19, 2010 10:05 pm
Posts: 1074
If you aren't using a virtual machine, like VirtualBox, QEMU, BOCHS, or VMWare, you really should be. It will speed things up considerably.

Once you have everything working in your VM, then making it work on physical hardware is pretty much just finding bugs and fixing them.

I would start with creating a bootable floppy image if your OS is smaller than 1.4Mb.

You can even create a USB hard drive image, boot it up in a VM, and test your USB code at the same time.

Let us know if you run into any problems.

_________________
Project: OZone
Source: GitHub
Current Task: LIB/OBJ file support
"The more they overthink the plumbing, the easier it is to stop up the drain." - Montgomery Scott


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sat Apr 15, 2017 8:38 pm 
Offline

Joined: Wed Feb 08, 2017 7:51 pm
Posts: 22
I appreciate the help, only Ben Lunt's (your) book cost money. I would gladly pay 50 dollars or so to buy it, but you see, I don't have a credit card...

If anyone knows a good free source, I would much appreciate it. If that is really the only one, then I guess I will see what I can find online and/or wait for my parents to get it...
Also, does anyone have a link to a USB software development manual (as there are many USB documents of all sorts)? I would appreciate that.

Also another thing: I pretty much just want USB capability to interface with my USB drive (it would be nice in the long run to have it identify other things, but for now I can cut it there). Does anyone know is there is another way to accomplish this that is less complicated than setting up a full-blown USB driver? It has to be with a USB disk, but since I only am supporting one type of device as of now, is there any way I could get away with just focusing on flash drives (Btw, I often shift flash drives frequently, and they vary in age and capacity. I could just use one type from now on, though).

BenLunt wrote:
As for the "don't want to teach anything" comment. I have to disagree. There are many people on this forum that are more than eager to help, including myself. I think what happens is that a beginner, or even a not so beginner, posts a vague question, with very little details, expecting a very detailed answer.


I understand that completely. It would frustrate me as well not having a specific question to answer. That is actually not what I was referring to, just I got a little offended when I was having trouble creating my IDT and some people were getting mad because I made some extremely stupid mistakes. I was perfectly specific about my problem, just that I made very stupid errors in my code, and people got pissed off that I was wasting their time or something. That was (still is) my first real project in assembly, and I have been pretty much learning as I go along. That being said, I learn very quickly, but I was very tired and too inexperienced at the time. Just an initial bad association with this site, I will try not to be so naive in the future.

Thanks again for all the help, I appreciate it.


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sat Apr 15, 2017 10:14 pm 
Offline
Member
Member

Joined: Mon Jan 03, 2011 6:58 pm
Posts: 283
If you really want to deal with USB, you can check out the Wiki page, though I can not vouch for its accuracy, http://wiki.osdev.org/USB

If you want your next step to just be file systems and hard drives, I suggest you install an emulator and work with disk images. This is much faster, and safer while you are developing file system code!

- Monk


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sun Apr 16, 2017 2:14 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

interruption wrote:
Hi. I have gotten to a stage in my os where I think I want to actually write to disk space and create a file architecture. The only problem is that I boot of a USB thumb stick for convinence (instead of a CD or something, or worse re-imaging your HDD every time you make changes). USB is (as I have discovered) a extremely complicated interface, and all of you on this site have shouted in the loudest possible way that you don't want to teach ANYTHING, so I was wondering if you have any good links or tutorials that explain the USB in a very specific way (not too general). At a minimum, could you provide a link to the manual? Thanks.


I have reached the stage where I think I want to run the marathon at the Olympics. Of course just because I think I want to do this doesn't mean I've done any preparation or any training, and if I actually tried running the marathon I'd probably end up in hospital before I reach half way.

Start with something simpler like the ancient floppy disk controller, then (using the skills/knowledge you gained from that) try something a little harder like (parallel) ATA controller, then (using the skills/knowledge you gained from that) try something a little harder like a SATA/AHCI controller, then...

Don't forget that even a driver for a simple device will depend on a lot of things (physical and virtual memory management, IRQ handling, scheduler and timing, IPC, "block storage device interface", PCI bus enumeration, etc); and until you have these things you shouldn't even consider starting a simple device driver.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sun Apr 16, 2017 9:59 pm 
Offline

Joined: Wed Feb 08, 2017 7:51 pm
Posts: 22
Thanks for all of the input. I will consider the following options.

But for one...

Brendan wrote:
I have reached the stage where I think I want to run the marathon at the Olympics. Of course just because I think I want to do this doesn't mean I've done any preparation or any training, and if I actually tried running the marathon I'd probably end up in hospital before I reach half way.


Yeah, yeah, I know. But can we chill with the sarcasm? You've been doing this for years (decades?) I imagine, and you are a expert on the subject. I may be an expert on another subject that you are not, but that doesn't mean I have to rub my experience in everyone's face. That being said, I (being a hypothetical expert) might be annoyed if someone tries to pretend like they're an expert, or make broad assumptions on topics they don't begin to understand, but I don't think I am doing such. Or if I am, then I do want correction on my wrongs, but I do so humbly without intention of being superior or running marathons without training or such.

And for the next statement...

Brendan wrote:
Start with something simpler like the ancient floppy disk controller, then (using the skills/knowledge you gained from that) try something a little harder like (parallel) ATA controller, then (using the skills/knowledge you gained from that) try something a little harder like a SATA/AHCI controller, then... [maybe you could even begin to fathom writing a USB driver]


#####PREPARE FOR A SET OF OBTUSE GENERALIZATIONS#####

I basically (in my limited understanding) see writing a device driver as voo-doo.

If I were to summarize the process:
-You have a device.
-You have hardware controller(s) for that device, which are basically chips on the motherboard.
-These hardware controller(s) interface with the motherboard (and then with the processor, since the motherboard is sort of the bridge that holds everything together).
-This interface is given by one or more ports on the motherboard (perhaps another interface?), and you would need to know seemingly random port numbers to be able to connect to the
controller(s) (our first bit of voo-doo).
-When you connect with the interface, you have to interpret a bunch of voo-doo that the controller spits out.
-You also have to send a bunch of voo-doo commands to the controller which correlate with instructions or processes that the controller executes according to the command issued (and
commands produce outputs, more voo-doo).
-And there is, of course, the context of your problem, which may need algorithms to convert stuff or something else.

In my view, writing a floppy disk driver is not easier than something else (like USB), it's just much less tedious (and contains a simpler form of voo-doo).
The voo-doo for writing a floppy disk driver is completely different and unrelated to that for USB, or SATA for that matter.

Remember that I am just in this to learn, that's all (perhaps...). Please correct me on my mistakes, being the more experienced ones.

And about the USB specifications, I did find them and downloaded a giant zip file with at least perhaps thirty different specifications. Could I at least have a pointer as to which one to read?

Thank you for your input.


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Sun Apr 16, 2017 11:53 pm 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4591
Location: Chichester, UK
I'm afraid that your understanding, that a floppy disk driver is no easier than a USB driver, is contrary to most people's experience. A floppy disk driver requires you to know how a floppy disk controller works. A USB driver requires you to understand how to scan the PCI bus to locate a particular controller, how to determine which of several varieties that controller is. Then you have to understand how to set up the complicated data structures required to communicate with it, and how to create the information packets that you need to send to a USB device. Also you need to know how to walk your way through the tree of devices to enumerate which devices are present. At this point you have reached the equivalent of sending data to a port on the floppy controller. So now you have to do more work, on a similar level to communicating with the floppy controller, to actually communicate with the device.

I've probably explained this poorly, but take it from people who have written drivers for disk controllers, network cards, USB devices, etc. that the latter is an order of magnitude harder than the former. Until you have actually tried to write some drivers you are not in a position to assert that one is as easy as another.


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Mon Apr 17, 2017 10:32 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1150
Location: Scotland
I started out by writing a floppy disk driver, but it really doesn't prepare you for writing anything more complex - all it does is provide you with an quick reward in that you have a device working in very little time. In your case though, if you did that work you would get little reward from it as you don't greatly care about floppy disk access, so all it would do is delay the thing that would serve as a reward for you, and that is to have working USB drivers - it would just make it all even more of a grind having to start with an extra, pointless exercise.

Later on, I looked at the USB spec and decided it was going to take too much time and effort to work out how to do anything with it - you need someone to guide you through it to make it manageable unless you're really determined and are prepared to spend a year of your life repeating tons of work that other people have already done just to try to make sense of it. The only sane approach is to pay a guide to show you the path of least resistance through the mountains of information in the USB spec, and that means you should find a way to buy Ben's book - you will never regret that purchase, and if you're confused about anything in it, you'll find that he's willing to help get you over any remaining obstacles (provided that you're a competent programmer who's clearly put in the work rather than someone lazy who can't be bothered to read carefully and to do the hard thinking - your aim should be to get through the thing without bothering him though). Pay someone to buy it for you if you have to, and get the paper copy rather than the Kindle one if you can afford to (because you will spend a lot of time jumping back and forth between many bookmarks and you'll want to be able to write notes directly onto the pages too - Kindle software makes it really slow to work on this kind of thing, although it is just about practical if you have the book open at different pages on three different devices, which is how I've been working with it).

This is a case where you need to buy a book - it will save you a large chunk of your life.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Mon Apr 17, 2017 11:10 am 
Offline

Joined: Wed Feb 08, 2017 7:51 pm
Posts: 22
So I guess I have misunderstood...

Sorry for being sort of arrogant.

However, could you skip a step and assume that there is only one device on the "tree of devices" part? I have never used USB hubs to expand the capacity of the ports present on my computer. Unless the ports on the computer sort of act as one giant 'hub', and you need to select different devices from them...

So yes, it will be a great challenge. But after all, I could've chosen to create a website with HTML and make pretty pictures move in circles with a few lines of javascript (plus sine and cosine), but I chose to do this, and I chose this because
1: I want to know how computers work at a very deep level (massive computer nerd),
2: it's a challenge, and
3: if colleges see that this is what I do with my free time, then they might be more apt to take me seriously (primarily the former two).

So I appreciate you guys' warnings! I really do! But does anyone have a good starting point on which to start on?
You have mentioned so far:
Ben Lunt's book - Great resource, but it will take time to acquire.
This site - I have looked at the page, and I got scared (don't laugh). I will give it another much more thorough look, however; it seemed to talk a lot about generally how USB
works, and the structure of the packets which I haven't gotten to yet. I need specifics!
Virtual Machines - If this fails miserably, I will consider setting one of these up. However, I have tried in the past and they have given me an awful amount of trouble, plus its just more
exciting to develop on real hardware. I have heard things about them giving you things (like a cleared memory) that don't exist in real life, and I have also heard people have trouble
switching to real hardware after using these.

But does ANYONE have a pointer as to which USB manual I should read? In the end, I can generally understand how anything works, but you need the sacred voo-doo magic to actually do stuff! If that involves interfacing with different types of controllers on a PCI bus by means of voo-doo data packets, then so be it!

Thanks for your input. I am really appreciating it and heeding your warnings, even though it may seem like I am disregarding them.

@DavidCooper - I got your response as I was finishing writing mine. I am sure that Ben Lunt's book will help me greatly, but I just want a chance to interpret the USB manuals for myself. Can I get the titles of the manuals that you started to look at (if you remember)?

And I completely agree with you on the floppy controller, just a different type of voo-doo. That is why I said what I did.


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Mon Apr 17, 2017 11:32 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 999
Here are a few pointers what you need to do in order to get USB working.

  • Prerequisites: You need a kernel that supports memory allocation, in particular allocation of contiguous physical memory. You need working IRQ handlers. You need to perform PCI enumeration to be able to find USB host controllers.
  • Read about the USB protocol layer basics. For example the "USB in a NutShell" tutorial might help with that. Be sure to also get a copy of the real USB specification (freely available) as tutorials won't be detailed enough to get things right. Make sure you understand what the USB terms "device", "configuration", "interface", "endpoint", "control/interrupt/bulk/isochronous transaction", "SETUP/DATA0/DATA1/ACK/NAK/STALL token", "setup packet", "device/configuration/interface/endpoint descriptor" mean.
  • Grab a copy of the specification of your favorite USB host controller (also freely available). For USB 1 there is UHCI and OHCI. For USB 2 there is EHCI and XHCI supports USB 1,2 and 3 at the same time. In term of difficulty UHCI < EHCI < XHCI; I cannot comment on OHCI as I don't have a driver for it yet. Some parts of the "older" specs are a bit stupid: For example UHCI does not generate an IRQ when you plug in a device so you have to poll for that. UHCI also does not support DMA to virtual memory (i.e. scatter/gather type DMA).
  • Implement a driver for your host controller.
  • Implement a driver for the USB bulk-only storage class (freely available). This will allow you to read/write flash drives.
  • Optionally implement a driver for USB hub class. For USB devices that are plugged directly into root ports no hub driver is necessary. Note however that many devices and some mainboards might have integrated hubs.

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Mon Apr 17, 2017 12:01 pm 
Offline
Member
Member
User avatar

Joined: Sun Sep 19, 2010 10:05 pm
Posts: 1074
interruption wrote:
But does ANYONE have a pointer as to which USB manual I should read?

Use Windows or Linux to figure out what physical USB controllers your computer has installed. From that list, pick the easiest one. In order from easiest to hardest is OHCI, UCHI, EHCI, xHCI. (In my opinion).

Then read the wiki page, and find the latest spec sheet for that controller.

Spec sheets for all of these can be found at http://www.usb.org/developers/resources/. I'll also try to add links to them on the wiki if I get a second.

_________________
Project: OZone
Source: GitHub
Current Task: LIB/OBJ file support
"The more they overthink the plumbing, the easier it is to stop up the drain." - Montgomery Scott


Top
 Profile  
 
 Post subject: Re: USB Help
PostPosted: Wed Apr 19, 2017 7:30 am 
Offline
Member
Member

Joined: Thu Jul 03, 2014 5:18 am
Posts: 84
Location: The Netherlands
Why focus on writing to the USB stick? Why don't you simply only use the USB stick to boot and then use the physical hard drive(s) as read/write storage?

_________________
My blog: http://www.rivencove.com/


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 46 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group