OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Apr 16, 2024 12:04 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Fri Apr 15, 2016 9:53 am 
Offline
User avatar

Joined: Mon Dec 28, 2015 7:36 pm
Posts: 9
Hi, I found this paper online and I thought that some of you might find it interesting.

http://www.ece.ubc.ca/~sasha/papers/eur ... inal29.pdf

Quote:
13% higher latency for kernel make, and a 14- 23% decrease in TPC-H throughput for a widely used commercial database


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Mon Apr 18, 2016 6:09 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7614
Location: Germany
The paper completely aside, it's the Torvalds quote that cracked me up.

Linux Torvalds wrote:
And you have to realize that there are not very many things that have aged as well as the scheduler. Which is just another proof that scheduling is easy.

-- Linus Torvalds, 2001


That guy is such a smug {bleep} that I really don't understand why certain people revere him so much. He made things happen, granted. Bill Gates did that as well... Linus might (might!) even be a competent developer. But that's where I draw the line regarding that bloke...

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Mon Apr 18, 2016 11:00 am 
Offline
User avatar

Joined: Mon Dec 28, 2015 7:36 pm
Posts: 9
Solar wrote:
The paper completely aside, it's the Torvalds quote that cracked me up.

Linux Torvalds wrote:
And you have to realize that there are not very many things that have aged as well as the scheduler. Which is just another proof that scheduling is easy.

-- Linus Torvalds, 2001


That guy is such a smug {bleep} that I really don't understand why certain people revere him so much. He made things happen, granted. Bill Gates did that as well... Linus might (might!) even be a competent developer. But that's where I draw the line regarding that bloke...


I think he's being sarcastic in that quote because the Linux scheduler is changed all the time.

Also there's this quote too:
Linus Torvalds wrote:
I'm an egotistical bastard, and I name all my projects after myself. First 'Linux', now 'Git'


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Mon Apr 18, 2016 11:56 am 
Offline
Member
Member
User avatar

Joined: Fri Oct 27, 2006 9:42 am
Posts: 1925
Location: Athens, GA, USA
mallardest wrote:
Linus Torvalds wrote:
I'm an egotistical bastard, and I name all my projects after myself. First 'Linux', now 'Git'


Unfortunately, that joke doesn't work for Merkins unless they know a bunch of Brit humor; over here, 'git' is a regional pronunciation of 'get', not a synonym for a-hole.

_________________
Rev. First Speaker Schol-R-LEA;2 LCF ELF JAM POEE KoR KCO PPWMTF
Ordo OS Project
Lisp programmers tend to seem very odd to outsiders, just like anyone else who has had a religious experience they can't quite explain to others.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Mon Apr 18, 2016 2:32 pm 
Offline
Member
Member
User avatar

Joined: Wed May 21, 2008 4:33 am
Posts: 294
Location: Mars MTC +6:00
Solar wrote:
That guy is such a smug {bleep} that I really don't understand why certain people revere him so much. He made things happen, granted. Bill Gates did that as well... Linus might (might!) even be a competent developer. But that's where I draw the line regarding that bloke...


I think he has a personality disorder, something like narcissism, which isn't something people choose to be or even recognise themselves that they have a disorder. I'm not a fan of him either but I'll always watch a video interview with him as he made a successful software project and there's always something to be learnt from others. I don't admire Steve Jobs or Bill Gates either but I watch their interviews too.

The guys I really admire tend to hide behind the scenes and focus only on the software that they need for their own (or clients) use. Unix was invented in house without financial incentive, similar with CP/M, the Xerox UI and the computer mouse.

As for what he's done, he cloned some software and did it better than the other guys. Games do the same thing, angry birds == crush the castle and candy crush == bejewelled.

_________________
"God! Not Unix" - Richard Stallman

Website: venom Dev
OS project: venom OS
Hexadecimal Editor: hexed


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Mon Apr 18, 2016 3:47 pm 
Offline
Member
Member
User avatar

Joined: Mon Apr 18, 2016 9:50 am
Posts: 138
Location: New York New York
I've always gotten the impression, after watching many an interview and what-have-you, that people take the guy way too seriously. He's obviously a relatively humble and jokey person who, like the rest of us (and look at this particular board as a rather egregious example at most times), has some strong opinions about certain things.

But if there's one constant I've noticed about him, it's that he's always witty and self-deprecating and almost constantly jokes about how often his own strong opinions change.

I think the real thing here is that this is pretty much the way most of us technical nerds are when it comes to our opinions, and most of our opinions are silly and religious and we hold onto them because that's what we do. And we like picking on Torvalds because he's an easy icon who represents everything we can project onto a person about our own place in our weird little culture, and it's nice to have a lightning rod.

I mean, the guy has made some bad decisions re: linux in the past, sure, maybe, I don't really know. But come on, guys, we're on a forum that's full of people working on always unfinished pipedream projects with a respectable contingent of relative crackpots in its ranks. And I count myself among that group, and am semi-proud to identify as such. But cast not the first stone and maybe chill a bit.

_________________
The OS is P5. Don't expect it to build.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Mon Apr 18, 2016 6:08 pm 
Offline
Member
Member
User avatar

Joined: Wed May 21, 2008 4:33 am
Posts: 294
Location: Mars MTC +6:00
jojo wrote:
I've always gotten the impression, after watching many an interview and what-have-you, that people take the guy way too seriously. He's obviously a relatively humble and jokey person who, like the rest of us (and look at this particular board as a rather egregious example at most times), has some strong opinions about certain things.

But if there's one constant I've noticed about him, it's that he's always witty and self-deprecating and almost constantly jokes about how often his own strong opinions change.


There is a difference between strong opinions and plain insult: https://en.wikiquote.org/wiki/Linus_Torvalds#External_links.

People on this board do have strong opinions that differ but I rarely see straight up insult.

Quote:
I'm a bastard. I have absolutely no clue why people can ever think otherwise. Yet they do. People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work, if it just results in what I consider to be a better system. And I'm not just saying that. I'm really not a very nice person. I can say "I don't care" with a straight face, and really mean it.

Torvalds, Linus (2000-09-06). Message to linux-kernel mailing list. Retrieved on 2007-05-28.


And I believe him.

If he was working for a commercial company there's a good chance he would have been fired like Steve Jobs was years ago. I don't think Kernighan, Richie and Wozniak would have been able to work this way and still be part of the business.

_________________
"God! Not Unix" - Richard Stallman

Website: venom Dev
OS project: venom OS
Hexadecimal Editor: hexed


Top
 Profile  
 
 Post subject: You read it? What is the paper actually about?
PostPosted: Tue Apr 19, 2016 2:29 am 
Offline

Joined: Thu Apr 24, 2008 6:14 am
Posts: 23
Nerd showing a paper about Linux Scheduler ... and inevitably Nerd-Folks slide into ranting about Linus? Yeah! [-X

I did a (brief) overview about this paper... so lets see...

Spoiler (TL;DR)

Quote:
“Nobody actually creates perfect code the first time
around, except me. But there’s only one of me.”
Linus Torvalds, 2007 [44] (from page 5 of this paper)


I actually LIKE this insane and sarcastic troll of a guy 8) (and am quite glad that I do not work near him)

Now what is the paper really about... new hardware had turned the once already solved scheduler problem
into a complex nightmare and its implementations full of bugs...
So they have crafted new tools to detect invariant violations and proposed bugfixes

Well, AND they could not resist to spice it all by citing Linus endless reservoire of sayings #-o

################################

Quote:
scheduler must maintain the following, simple, invariant:
make sure that ready threads are scheduled on available
cores.


Quote:
Our recent experience with the Linux scheduler revealed
that the pressure to work around the challenging properties
of modern hardware, such as non-uniform memory access
latencies (NUMA), high costs of cache coherency and synchronization,
and diverging CPU and memory latencies, resulted in a scheduler with an incredibly complex implementation.


Quote:
The bugs we described resulted from developers want-
ing to put more and more optimizations into the scheduler,
whose purpose was mostly to cater to complexity of modern
hardware.


Quote:
This occurs even though the scheduler is not explicitly configured to save
power by purposefully leaving cores unused so they can be brought into a
low-power state.


Quote:
We provide fixes to these bugs, and observe substantial performance improvements.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Tue Apr 19, 2016 2:34 am 
Offline
Member
Member

Joined: Sat Mar 01, 2014 2:59 pm
Posts: 1146
b.zaar wrote:
If he was working for a commercial company there's a good chance he would have been fired like Steve Jobs was years ago. I don't think Kernighan, Richie and Wozniak would have been able to work this way and still be part of the business.
...which just proves why open source software can be such a successful development model. You can have many people pooling together their development knowledge, skills, and experience without having to get along with each other on a personal level.

_________________
When you start writing an OS you do the minimum possible to get the x86 processor in a usable state, then you try to get as far away from it as possible.

Syntax checkup:
Wrong: OS's, IRQ's, zero'ing
Right: OSes, IRQs, zeroing


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Tue Apr 19, 2016 2:49 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7614
Location: Germany
onlyonemac wrote:
...which just proves why open source software can be such a successful development model. You can have many people pooling together their development knowledge, skills, and experience without having to get along with each other on a personal level.


Err... do you have actual experience being employed in software engineering?

Teams aren't put together "because you get along". You aren't hired because you "get along" with other team members, and unless you are also lacking in technical skills, you aren't fired for "not getting along" with others either (extreme cases notwithstanding).

But you tend to "get along" in the corporate field because you are spending 8+ hours in the same office every working day, and don't want to make your own life hell. So you swallow your pride from time to time, and compromise.

Enter Open Source development, where communication is usually done via the internet and without a clear hierachy of superiors to appeal to if things go sour... yeah. Doesn't take much imagination to picture how great this will work if you don't "get along".

----

That being said, I remember the first day at a company, being presented my new colleagues. I don't remember the exact words anymore, but the general message was, "meet Mr. D.; he's from an external company. Your first assignment is to make him redundant as soon as possible so we can terminate his contract."

Oh joy...

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Tue Apr 19, 2016 8:47 am 
Offline
Member
Member

Joined: Sat Mar 01, 2014 2:59 pm
Posts: 1146
Solar wrote:
onlyonemac wrote:
...which just proves why open source software can be such a successful development model. You can have many people pooling together their development knowledge, skills, and experience without having to get along with each other on a personal level.


Err... do you have actual experience being employed in software engineering?

Teams aren't put together "because you get along". You aren't hired because you "get along" with other team members, and unless you are also lacking in technical skills, you aren't fired for "not getting along" with others either (extreme cases notwithstanding).

But you tend to "get along" in the corporate field because you are spending 8+ hours in the same office every working day, and don't want to make your own life hell. So you swallow your pride from time to time, and compromise.

Enter Open Source development, where communication is usually done via the internet and without a clear hierachy of superiors to appeal to if things go sour... yeah. Doesn't take much imagination to picture how great this will work if you don't "get along".

----

That being said, I remember the first day at a company, being presented my new colleagues. I don't remember the exact words anymore, but the general message was, "meet Mr. D.; he's from an external company. Your first assignment is to make him redundant as soon as possible so we can terminate his contract."

Oh joy...
Perhaps you should direct your post at b.zaar, because he's the one that suggested that people would get fired if they didn't get along with the rest of the team; I was merely building on what he had said.

_________________
When you start writing an OS you do the minimum possible to get the x86 processor in a usable state, then you try to get as far away from it as possible.

Syntax checkup:
Wrong: OS's, IRQ's, zero'ing
Right: OSes, IRQs, zeroing


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Tue Apr 19, 2016 8:56 pm 
Offline

Joined: Tue Mar 15, 2016 8:46 am
Posts: 6
I think that's a pretty common viewpoint among younger workers. Remember, those of us under 35 did not benefit from our governments eviserating their economies with inflation just to get us employed. When one comes of age in a slack labor market one learns to care about teamwork, as otherwise one is liable to be replaced.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Thu Apr 21, 2016 2:45 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7614
Location: Germany
onlyonemac wrote:
Perhaps you should direct your post at b.zaar...


Yea... I rambled a bit all over the place, didn't I?

Old age.

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Fri Apr 22, 2016 4:50 pm 
Offline
Member
Member

Joined: Wed Oct 01, 2008 1:55 pm
Posts: 3192
At least the paper shows the complexity of it. Also, it's not a problem that can be solved with a huge amount of complexity really, because at least the low-level scheduler doing the real-time scheduling should be as fast as possible, and thus simple.


Top
 Profile  
 
 Post subject: Re: "The Linux Scheduler: a Decade of Wasted Cores"
PostPosted: Mon Apr 25, 2016 12:20 am 
Offline
Member
Member

Joined: Sat Mar 15, 2014 3:49 pm
Posts: 96
We presume that all the other OS have decades of wasted cores too.

This paper shows a way to find out and quantify it. I hope other OSes get tested and fixed too.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot] and 20 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group