OSDev.org

The Place to Start for Operating System Developers
It is currently Sun Nov 18, 2018 5:29 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 14 posts ] 
Author Message
 Post subject: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 1:54 pm 
Offline
Member
Member
User avatar

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 606
My network stack has been around for some time but recently I discovered a problem. When trying to download a relatively large file (1.4M) from a server via HTTP, it is unacceptably slow. So I ran wireshark to analyse the traffic. Obviously it's a big mess, but what I think I'm seeing is the following:

1. Server sends some data.
2. Server sends more data.
3. Server sends even more data.
4. Glidix ACKs #1
5. Server re-sends #2
6. Server re-sends #3
7. Server sends #4
8. Glidix ACKs #2
9. Server re-sends #3
10. etc etc etc

The conclusion that I'm getting from this is that the server isn't expecting every single message to be ACKed, and treats the fast ACKs as indication that data has been dropped. I don't see this mentioned anywhere in RFCs or on wikipedia (i might be blind).

So my question is, with TCP, am I supposed to be waiting some time before sending an ACK, and if more data arrives, am I supposed to "reset" the timer, and effectively only send an ACK if I have not received more data for some time ???

_________________
Glidix: An x86_64 POSIX-compliant operating system, aiming to be as optimized as possible, especially in graphics.
https://github.com/madd-games/glidix


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 2:59 pm 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3304
Location: Chichester, UK
TCP is stream oriented. It doesn't consist of messages. Why doesn't your stack immediately acknowledge all the data it has received so far?


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 3:12 pm 
Offline
Member
Member
User avatar

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 606
iansjack wrote:
TCP is stream oriented. It doesn't consist of messages. Why doesn't your stack immediately acknowledge all the data it has received so far?


It DOES axknowledge all the data it has received so far. But it takes some tiny bit of time for the ACK to be delivered to the server (obviously). And it appears that in that time, the server sends MORE data, before it even receoved the ACK. The server runs Ubuntu.

And by messages, i meant the packets. I know TCP is stream oriented, but data over IP is still sent using packets.

_________________
Glidix: An x86_64 POSIX-compliant operating system, aiming to be as optimized as possible, especially in graphics.
https://github.com/madd-games/glidix


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 3:30 pm 
Offline
Member
Member

Joined: Mon Feb 02, 2015 7:11 pm
Posts: 256
mariuszp wrote:
But it takes some tiny bit of time for the ACK to be delivered to the server (obviously). And it appears that in that time, the server sends MORE data, before it even receoved the ACK. The server runs Ubuntu.


This is perfectly normal for TCP. If it was waiting for your ACK before sending more data, performances would be very slow. There is a sliding windows used to limit how much data can be sent without an ack. IIRC that window size can change dynamically.

_________________
https://github.com/kiznit/rainbow-os


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 3:52 pm 
Offline
Member
Member
User avatar

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 606
kzinti wrote:
mariuszp wrote:
But it takes some tiny bit of time for the ACK to be delivered to the server (obviously). And it appears that in that time, the server sends MORE data, before it even receoved the ACK. The server runs Ubuntu.


This is perfectly normal for TCP. If it was waiting for your ACK before sending more data, performances would be very slow. There is a sliding windows used to limit how much data can be sent without an ack. IIRC that window size can change dynamically.


So, if I understand correctly:

1. My OS receives data over TCP.
2. It should wait some small amount of time (50ms??), or until window_size bytes are received, before it sends an ACK.
3. This way, if the server sends more data, the client ACKs the entire thing instead of acknowledging each individual packet.

Is that what the process is supposed to be like?

_________________
Glidix: An x86_64 POSIX-compliant operating system, aiming to be as optimized as possible, especially in graphics.
https://github.com/madd-games/glidix


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 4:10 pm 
Offline
Member
Member

Joined: Tue Mar 04, 2014 5:27 am
Posts: 911
I haven’t looked at TCP for a while, so I don’t remember much from my short experience with it. But you should pay attention to the fact that packets may be coming out of order (there are TCP implementations that mess up the order before the packets even hit the wire!) making it look like you have lost some. And if you start complaining to the sender about the loss too soon, you’re going to have bad performance. In reality you should keep a number of out of order packets, so you don’t end up rerequesting or receiving them again. I can’t direct you to the right place in the right RFC, but look around SACK.


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 10:06 pm 
Offline
Member
Member

Joined: Mon Feb 02, 2015 7:11 pm
Posts: 256
mariuszp wrote:
1. My OS receives data over TCP.
2. It should wait some small amount of time (50ms??), or until window_size bytes are received, before it sends an ACK.
3. This way, if the server sends more data, the client ACKs the entire thing instead of acknowledging each individual packet.

Is that what the process is supposed to be like?


I am not sure why you want to wait at all... Just ACK each time you receive a packet of data. Waiting won't help with anything: there will always be another packet of data coming in. If you wait for window_size bytes, you defeat the point of having a window in the first place.

_________________
https://github.com/kiznit/rainbow-os


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 10:53 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

mariuszp wrote:
My network stack has been around for some time but recently I discovered a problem. When trying to download a relatively large file (1.4M) from a server via HTTP, it is unacceptably slow. So I ran wireshark to analyse the traffic. Obviously it's a big mess, but what I think I'm seeing is the following:

1. Server sends some data.
2. Server sends more data.
3. Server sends even more data.
4. Glidix ACKs #1
5. Server re-sends #2
6. Server re-sends #3
7. Server sends #4
8. Glidix ACKs #2
9. Server re-sends #3
10. etc etc etc


Can you provide more detail?

This could be interpreted as:

1. Server sends some data.
2. Server sends more data.
3. Server sends even more data.
4. Glidix ACKs #1
5. Server didn't get an ACK for #2 in time and assumes packet was lost, so server re-sends #2
6. Server didn't get an ACK for #3 in time and assumes packet was lost, so server re-sends #3
7. Server sends #4
8. Glidix ACKs #2
9. Server didn't get an ACK for #3 in time and assumes packet was lost, so server re-sends #3
10. etc etc etc

If that's the case, then the server is working correctly and the client has major performance problems (not sending ACKs fast enough).

Alternatively it could be like this:

1. Server sends some data.
2. Server sends more data.
3. Server sends even more data.
4. Glidix ACKs #1
5. Server re-sends #2 even though receiver hasn't had time to receive the first copy
6. Server re-sends #3 even though receiver hasn't had time to receive the first copy
7. Server sends #4
8. Glidix ACKs #2
9. Server re-sends #3 even though receiver hasn't had time to receive the previous copy
10. etc etc etc

In that case there's a problem with the server (bad/misconfigured or missing time-outs and/or inappropriate window size) and the client is working correctly.

Note: I am assuming that this is on a nice "quiet" LAN, and network congestion is not involved.

mariuszp wrote:
The conclusion that I'm getting from this is that the server isn't expecting every single message to be ACKed, and treats the fast ACKs as indication that data has been dropped. I don't see this mentioned anywhere in RFCs or on wikipedia (i might be blind).


As far as I know; the client can coalesce ACKs (e.g. instead of sending "ACK #1", "ACK #2", "ACK #3" it can just send "ACK #3" to tell server it received everything up to #3); but (even though it's a good idea - e.g. improve performance/reduce bandwidth consumption by reducing the number of packets sent by client) this is a purely optional optimisation, and server can not/must not assume that a packet won't be received simply because it wasn't included in the latest ACK from client.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Thu Aug 09, 2018 11:43 pm 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3304
Location: Chichester, UK
You don't acknowledge that you have received a packet, you acknowledge that you have received all bytes up to a certain point. So if you really have received all bytes up to the end of packet 3 then you can acknowledge them all at once (assuming there are no lost packets). But why are your acknowledgements so slow? The last thing you need to do is to deliberately wait before acknowding received data; that's just going to throttle the transfer. The only reason not to immediately acknowding received bytes if some preceding bytes in the stream have not yet arrived.

You might find this site useful: http://www.tcpipguide.com/free/index.htm


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Fri Aug 10, 2018 4:51 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3304
Location: Chichester, UK
In the case that a packet has been dropped, the acknowledgement mechanism offers a significant optimization. If a dozen packets have been received since the dropped packet then (when that packet is eventually received) you can miss out a dozen acks. That might well be enough to prevent some of the received packets from being retransmitted.

(Note that you can't acknowledge the bytes in those later packets untill the dropped one is received. Acknowledgement means "I have received every byte up to [the one specified in the ack packet]".)


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Fri Aug 10, 2018 6:02 am 
Offline
Member
Member
User avatar

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 606
I send an ACK every time a packet arrives. My code simply receives a TCP segment, and ACKs all the bytes it received so far, including the newly-arrived segment. In case I misdiagnosed the problem, a attached a Wireshark capture showing what is going on if anyone wants to take a look.


Attachments:
File comment: Wireshark capture
tcp-problems.zip.zip [25.86 KiB]
Downloaded 4 times

_________________
Glidix: An x86_64 POSIX-compliant operating system, aiming to be as optimized as possible, especially in graphics.
https://github.com/madd-games/glidix
Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Fri Aug 10, 2018 6:56 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3304
Location: Chichester, UK
Well, you seem to have a lot of erroneous packets there. I'd start at the beginning, with packets 22 and 23. Why is a duplicate SYN packet sent?

Then packets 25 and 26 again seem to be duplicates. What's going on here?

Packet 27 is again duplicated almost immediately.

Later on you are getting a host of duplicate ACK packets, so I have to think there is something wrong with your acknowledgement mechanism. I get the impression that every packet you send is sent twice, which is causing problems.

I would try first of all to get rid of that duplicate SYN packet, then work your way through the other duplicated packets to see what is causing the problem. What happens if you just do a very simple connection and request, that involves just a single packet of data. Do you still get the duplicates?


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Fri Aug 10, 2018 8:38 am 
Offline
Member
Member
User avatar

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 606
wireshark is showing me all packets duplicated when i sniff on eth0 with VirtualBox brisged to it. I do not know if 2 packets are really sent or if its a sniffing problem caused by the bridging...

If i try 2 different drivers in my OS, and thdn debug each one to see what packets are sent, it tells me that only one of each is sent as expected. So it seems to be wireshark getting confused by sniffing on a bridges network.

EDIT: It now appears that if I sniff on a "host-only" network, all packets are also duplicated... perhaps it is in fact some obscure problem at the OS level. I'll have to try a few more things...

EDIT 2: If I run Ubuntu in a VM, it ALSO shows all packets duplicated... so it must be wireshark/virtualbox doing something

_________________
Glidix: An x86_64 POSIX-compliant operating system, aiming to be as optimized as possible, especially in graphics.
https://github.com/madd-games/glidix


Top
 Profile  
 
 Post subject: Re: are TCP ACKs supposed to be limited?
PostPosted: Sat Aug 11, 2018 6:02 pm 
Offline
Member
Member
User avatar

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 606
Well, when i disable monitoring mode on the VM (and so the guest was receiving less unnecessary packets) the download now goes up to 14% almost instantly (previously took a few minutes for 3%) but is also less reliable and sometimes fails to make connections... that's interesting

_________________
Glidix: An x86_64 POSIX-compliant operating system, aiming to be as optimized as possible, especially in graphics.
https://github.com/madd-games/glidix


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Google [Bot] and 20 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group