How about "Operating System Concepts 7th edition" by Silberschatz Galvin and "Understanding the Linux Kernel"...

This book explains, from the beginning, how to write code to communicate with the USB, to be able to write a driver for the
four major types of USB controllers; the UHCI, OHCI, EHCI, and the new xHCI. It explains how to send requests to the
devices attached, to return the device's descriptors to be able to configure and use the devices. Some examples include drives,
mice, keyboards, printers, and external hubs. This is all done without any outside help. In other words, you are communicating
directly with the hardware. You are not relying on an existing operating system's function calls. This book also contains example
descriptions of why your driver may not be working and then gives details on how to fix it. If you are an operating system
development enthusiast, a beginner or a more advanced developer, this is a book for your desk.

The author fully supports this book and upon request and receipt of purchase, will send you an .ISO image with example source
code to retrieve a device's Device Descriptor, plugged in to one of the four controller types, along with other source code
described within the book.

This book is for those who are building an operating system, or want to start building one. This book takes you from the
first step to the last step to create a minimal multi-threading protected mode OS with a round-robin style task scheduler,
memory allocator, etc., as well as retrieving all the information you will need before the switch to protected mode.
This book is for those who want to start but have no idea where or how to do so. Full source code is included as well as
many utilities, all with source, to build your system.
It is designed so that once you have this minimal system up and running, you can then start adding all of the items you need
to make it a fully functional operating system. Part 2 contains an example of a more detailed memory allocator, just an
example on how to add to this operating system.
Chapter 11 gives an example of creating the link between the kernel and the devices and file systems attached.
The remaining books in this series explains in more detail these devices.

This book is Volume 2 of the series, FYSOS: Operating System Design, and will describe the process needed to connect a file
system to your kernel.
It explains the steps to create a file system independent link between a file system residing on the media to your kernel code.
Once you have done this, a simple call to fopen("C:\filename.txt", OPEN) is all you need to open a file.
It then goes into detail, giving examples of what a file system should be, how to create your own, and what to include in that
file system. It also explains the process of a simple journaling file system.
Part 2 of this book details four file systems, two commonly used and well known, and two hobby file systems.
The companion CD-ROM contains source code to utilities to check your work, display the contents of a file system, dump a file
system, and repair a file system.
This book, and its continued series of books, does not expect you to build the next great wonder of the computer world. It
simply will help you with your interest in controlling the computer's hardware, from the point the BIOS releases execution to
your boot code to the point of a fully working Graphical User Interface.
It is not required that you know much about operating system design to use this book and no specific hardware or software
is required.

This book is Volume 3 of the series, FYSOS: Operating System Design, and will show the reader how to detect, initialize, and communicate
with three of the most common media hardware devices, the Floppy Disk Controller, the IDE Hard Disk Controller, and the SATA (AHCI)
Hard Disk Controller. The reader will learn how to detect the controller, what type of controller it is, initialize it to default values, detect
attached devices, and then communicate with those devices, such as reading and writing to the attached media.
All of this is done without any outside help, such as operating system calls or the help of the BIOS. The reader will learn how to communicate
with the hardware directly, reading and writing to the system bus to achieve these tasks.
The companion CD-ROM contains complete source code of each example within the book, showing how to accomplish these tasks.
This book, and its companion series of books, does not expect you to build the next great wonder of the computer world. It simply will help
you with your interest in controlling the computer's hardware, from the point the BIOS releases execution to your boot code to the point of a
fully working Graphical User Interface.
It is not required that you know much about operating system design, though a good knowledge of C Programming Language and a moderate
knowledge of an Intel/AMD x86 computer's hardware is expected to use this book.

I think the best book about CPU and memory virtualization (cpu scheduling, paging, segmentation) and file systems is "Three Easy Pieces" from Remzi and Andrea H. Arpaci-Dusseau at the University of Wisconsin. The book is very accessible, it's extremely very well written, very pedagogical and the authors manage to explain difficult concepts in a very clear manner. Someone was talking about it here and they were saying it has received good reviews. You can also see this which was talking about the same thing. Overall, people seem to agree with this point, and you can also check the posts below to find more information.There is a lot of info out there and it's just a matter of filtering the good from the bad. People of the older generation generally don't understand this because back then they did not have all this information available to them.
I've read many others OS books, but this one is IMHO the best of all, by a good margin and the best part is that one can download the different chapters for free here.
I'm actually pretty surprised nobody mentionned it here...

This is a great resource. I just skimmed through the chapters for a while and I can say it is really good. Easy to follow. Do you know if all the chapters of the book is also available here? If you have similar resources please share them too. Thanks!

Try "Understanding Linux Kernel" and "Understanding the Linux Virtual Memory Manager". These will help you much in design consideration.