OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 8:29 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 16 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: TSS requiered?
PostPosted: Sun Aug 25, 2019 8:05 pm 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
I always read, tss are not required, Linux and co are using software context switches. But is it still required(!) to setup a TSS, when just only want entering ring3? (without any fancy features).

In my case, i was able successfully get Threading working, even with other CS/DS (placing the CS on the newly created Thread-Stack before IRETD), but only if they are ring0 too. In the moment i change The "user-level"-Selectors to Ring3, i always get a GP (error code=User-CS-Segment), regardless of Confirming-Bit 0 or 1 (of kernel CS-Code-Segment).

The Context of the initial Thread-Setup is from within a Timer-Interrupt (Interrupts are fully working).

Operating in 32-Bit mode, but with enabled PAE, No Long Mode.


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Sun Aug 25, 2019 8:23 pm 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 671
You need a TSS if you intend on having interrupts enabled while in a ring other than ring 0 (like ring 3). The TSS is used to determine the stack SS:ESP to use when an interrupt occurs while in a ring > 0 (Like ring 3). Although Linux (and most other 32-bit OSes) have a TSS, it isn't used for hardware task switching. Two things the TSS are good for are for setting the stack address to use when an interrupt occurs when it happens while in ring > 0, and the TSS contains the Port IO bitmap that can be used to allow port in/out instructions to occur without causing a fault (when Current Privilege Level > IOPL).


Last edited by MichaelPetch on Sun Aug 25, 2019 8:30 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Sun Aug 25, 2019 8:28 pm 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
Ok, of course i have Interrupts enabled. The last commands in the asm file are:

Code:
sti ;enable interrupts
iretd ;return interrupt. Load EIP, CS and EFLAGS. Because we where called initially from Ring0, we have SS and ESP on the stack, too.


So, as i understood, because Interrupts are enabled, i definitively need a TSS Segment Descriptor?


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Sun Aug 25, 2019 8:40 pm 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 671
When you did STI and IRETD I assume that was to transition to Ring 3. How did you push flags onto the stack. The STI will enable interrupts (sets IF=1) but then IRETD will change the flags register based on what is popped off the stack during the IRETD (if the flags on the stack have IF=0 the interrupts will be turned off before entering ring 3). But yes, you NEED a TSS if you intend to have interrupts occur while in ring 3.


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Sun Aug 25, 2019 9:19 pm 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
Code:
─── Assembly ───────────────────────────────────────────────────────────────────────────────────────────────────────────────
0x00000000c01020c7 InterruptReturn+33 popad 
0x00000000c01020c8 InterruptReturn+34 add    esp,0x8
0x00000000c01020cb InterruptReturn+37 sti   
0x00000000c01020cc InterruptReturn+38 iret   
─── Registers ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
   rax 0x0000000000000000         rbx 0x0000000000000000         rcx 0x0000000000000000         rdx 0x0000000000000000     
   rsi 0x0000000000000000         rdi 0x0000000000000000         rbp 0x00000000413c4ff8         rsp 0x00000000413c4fec     
    r8 0x0000000000000000          r9 0x0000000000000000         r10 0x0000000000000000         r11 0x0000000000000000     
   r12 0x0000000000000000         r13 0x0000000000000000         r14 0x0000000000000000         r15 0x0000000000000000     
   rip 0x00000000c01020cc      eflags [ IF ]                      cs 0x00000008                  ss 0x00000010             
    ds 0x00000020                  es 0x00000020                  fs 0x00000020                  gs 0x00000020             
─── Stack ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[0] from 0x00000000c01020cc in InterruptReturn+38 at external/MOSA-Project/Source/Mosa.Runtime/Internal.cs:209
(no arguments)
[1] from 0x413c4fc400000202
(no arguments)
[+]
─── Threads ────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[1] id 1 from 0x00000000c01020cc in InterruptReturn+38 at external/MOSA-Project/Source/Mosa.Runtime/Internal.cs:209

(don't be confused, its still all x86 mode).
Memory of $ESP:
Code:
>>> x/20x $esp
0x413c4fec:   0xc01047f2   0x00000018   0x00000202   0x413c4fc4
0x413c4ffc:   0x00000020   0x00000000   0x00000000   0x00000000
0x413c500c:   0x00000000   0x00000000   0x00000000   0x00000000
0x413c501c:   0x00000000   0x00000000   0x00000000   0x00000000
0x413c502c:   0x00000000   0x00000000   0x00000000   0x00000000

0x18: User-Code, 0x20: User-Data/User-Stack.
0xc01047f2: Entry-Point in User-Thread. 0x413c4fc4: ESP
GS, FS, ...is already set. SS not, because IRETD will do this for us. As you See i placed 0x202 for EFLAGS the Stack, so interrupts keep enabled.

GDT Memory:
Code:
>>> x/10x 0x04300006
0x4300006:   0x00000000   0x00000000   0x0000ffff   0x00cf9a00
0x4300016:   0x0000ffff   0x00cf9300   0x0000ffff   0x00cffa00
0x4300026:   0x0000ffff   0x00cff300


Current Context: Timer-Interrupt in Ring0, Goal: Create Thread. SS and ESP are not on the stack automatically "yet" (like documented in intel documentation), so i write them manually on the stack via assembler (that's more a hack).

If I move one step forward, I get immediately a GP, Error 0x18.

So, as I understood your explanation: Because interrupts are enabled (via STI and/or popping the specified EFLAGS from stack) AND then trying switching the Ring to 3 while not having a TSS, i get a GP. Did I got the point?

Greetings


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 6:30 am 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
I got the Reason: It was because i used 0x20 instead of 0x20|0x03.

Task Switching works now, so long all Threads are using the same Ring (all Ring 3 or all Ring 0). I get trouble when switching from R3->R0 or R0->R3. That's because of the iretd, if Level Changed, SS:ESP is placed on the stack, too. It seems to be very complicated now and I need to write a separate Switch-Method for every Combination. But as i read here and everywhere it should be simpler.

As I understood, when using a TSS, the values are never pushed on the stack, they are placed on the TSS, and then stack is changed. But cant get "Tasks" working. TaskSegmentDescritor is set up, and I can successful invoke LTR (without GP). But the "tasks" is never used. Qemu Monitor shows me all the time "TSS-AVAIL", so its never busy.

As I read, you can "Execute that Task" via a IRETD, too. But what values and where I need to place on the stack for that?


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 9:41 am 
Offline
Member
Member

Joined: Tue May 13, 2014 3:02 am
Posts: 280
Location: Private, UK
MichaelPetch wrote:
Two things the TSS are good for are for setting the stack address to use when an interrupt occurs when it happens while in ring > 0, and the TSS contains the Port IO bitmap that can be used to allow port in/out instructions to occur without causing a fault (when Current Privilege Level > IOPL).


The other use for a TSS (and an actual hardware task switch) is a double-fault handler. Since a double-fault often/usually means a problem with the kernel-space stack, using a TSS to switch to a different stack to at least have a "graceful" kernel-panic is better than a CPU reset.

_________________
Image


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 9:56 am 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 671
You say you have problem using IRETD to get from R0 to R3? I thought you said you got that working.
You can't use IRETD to change from a less privileged ring to a more privileged ring. So you can't change from privilege level 3 back to privilege level 0 (or 1 or 2 as well). You can change to a more privileged through a call gate, interrupt gate or trap gate. That can allow the system to switch into privilege level 0 that can then shut down the task. For instance you could create an interrupt gate for software interrupt 0x80 and do an end task. You could in theory create a GPF interrupt handler that looks to see if a HLT instruction caused the fault and exit the task. How you do it is up to you.

The first question you need to ask yourself. Do you intend to use hardware task switching or do you intend to software task switching? Most (not necessarily all) modern OSes use software task switching and do not rely on the 386+ processors hardware switching support. Software task switching is generally more portable between different types of processors. The hardware task switching doesn't have nearly the performance of software task switching, and hardware task switching doesn't save all the processor state automatically.


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 10:30 am 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
Hi Michael - definitively software switching, but with the use of tss for kernel stack.

Switching really works - of course only with the help of timer: CPU is interrupted from whatever ring, jump into ISR and from there I can choose the next task and ring. But as it seems, my TSS doesn't have any effect: Stack at [tss.esp] is always empty. Depending on of what Ring was interrupted, I have SS:ESP on stack (or not). So, it seems, TSS has zero effect. But I was able to use LTR, and I see the value 0x2b all the time (it never goes away - which is a good sign). After Setting LTR I see in Qemu "TSS32-BUSY". But after some instructions, I see only "TSS-AVL" (for ever).

I set up TSS After IDT. My Flow:

Bootstrap code set GDT and PageTable for Upper-Memory-Kernel.
After real Kernel EntryPoint and pimping PageTable, i set up PIC, than IDT. Send EOI. In that moment, the Clock-Interrupt will fire, but do nothing, because the Scheduler is not setup.
Then i setup User Segemnts (User-CS, User-DS, TSS Descriptor and struct, and assign kernel stack). Updating GDT (LGDT) and Loading LTR. Then i Setup the Scheduler (Idlethread and Worker-Threads (in RING0 and RING3, depending on of the worker type).

On Clock-Interrupt i switch the tasks. But as told, tss seems to have no effect. At the moment, I need to track from which ring to which other ring I go to which new ring and pushes/popping the specified SS:ESP, depending of the privilege-Direction.

For better debuggin and undertanding: Will which situation will be the TSS Busy and when Available? Always busy? Only when not in ISR? Or Only when in ISR?


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 11:04 am 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 671
The TSS isn't consulted if the interrupt (timer) occurs while the processor is already in Ring 0 at the time the interrupt occurs. So you will have a ring 0 kernel stack for the times interrupts occur while already in Ring 0 and you will have a a stack for your ring 3 processes. In fact if you are doing software task switching each process (and thread) should have its own stack. You change the SS0:ESP0 in the TSS when you switch between processes/thread. Each Process/thread has its own kernel SS:ESP, thus the process/thread's that is being switched to has its kernel SS:ESP copied into the TSS's SS0:ESP0 on each task switch by the scheduler. The kernel stack of each process/thread is separate from the user space stack that the process may be using.

You generally only have one TSS per processor. If you are working with a single CPU everything can be done with one TSS. Note it may be that all your processes use the same SS for kernel stacks so it doesn't need to be updated (if it is always the same).

In 32-bit code using task switching the only fields of interest really are SS0:ESP0 (if you are only using Ring0 and Ring3), and the IOBitmap. The rest of the fields in the TSS are used with hardware task switching and pretty much become irrelevant although the fields can always be repurposed for other things.


Last edited by MichaelPetch on Tue Aug 27, 2019 2:00 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 12:26 pm 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
Hi Michael, thank for your replies. They are really helpful to understand the details. But i still do not get the point. I show you in pseudo-code my code the way I understood/I'm currently doing it.

Some names:
ClockISR=Clock-Routine,
MAIN the default entry point "thread", that should "die" when Scheduling I set up.
K1 and K2: 2 Kernel Threads (Ring 0)
U1 and U2: 2 User Threads (Ring 3).

[all done outside of an ISR, all done in MAIN]
setup GDT, TSS, TaskSegmetnDescritor
Use "only" TSS.ESP0 and TSS.SS0.
CPU still operates in normal mode.
Let's say, create 2 ring0 and 2 ring3 threads (K1-2, U1-2). Every Thread gets its own stack-space.

Now the big Question/Part: How to "awake" the Scheduler, how to switch to a Thread K1 or U1 while we are still in MAIN (which can/should "die" directly after the first Thread in online).
My way of understanding:
MAIN (while not in an ISR) sets global (only one CPU) tss.esp0 and tss.esp0 for a specific "first" thead in the queue, lets say K1 OR U1.
Loading LTR. "TSS32-AVL" is now visible in qemu ("info registers"), and the given TR-selector.
So now....are we in U1 or K1? No!
Can we use IRETD? No, because NT-Flag is not set, so SS:ESP from Stack would be used instead (i read the intel docs). And by the way, we are not in an ISR.
If we can really use IRETD outside of ISR: Can i simply put SS:ESP on the stack like yet? Should i set the NT-Flag manually?

Or use JMP TS:0x00000000? If I do jmp 0x2B:0x00000000, I get a GP, bochs is telling me "jump_protected: gate type 11 unsupported".

Ok, lets say we are now "in Task".

As i understood, when "in task" (the normal operating mode of the CPU, after Scheduling is set up), it will interrupt the task, will NOT push additional SS:ESP on the stack, regardless of currently running K1 or U1, and not popping when doing a IRETR.

So, ClockISR sets tss.esp0 and tss.ss0 to a stack of the next task. It should not matter if from U1->U2, R1->R2, U1->K1 or K1->U1 (all should be possible).
Now do a IRETR and the CPU will apply tss.esp0 and tss.ss0. But IRETR will do this only, if NT-Flag is set. How ensure NT-Flag is set?

Questions over questions. As you see, i understood mostly. But i miss a "small piece" the complete "the big picture".


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 12:54 pm 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 671
I have limited time right now to look at this, but what I will say is that if you aren't using hardware switching then you want to ensure that the NT flag is clear (not set). Having NT set causes IRETD to perform operations to return to previous tasks (that are done by hardware task switching) using the back link field in TSS. In most cases you won't want to be using that feature unless you know what you are doing. Since you are using software task switching make sure that every task you create has the NT bit of EFLAGS cleared upon entry. Then you don't concern yourself with that case and IRETD will never do hardware task switching using the NT flag.

Since you are learning I would simplify things and make sure you can get it working with all Ring3 tasks first. Then deal with a combination of Ring0 and ring3 tasks.


Last edited by MichaelPetch on Tue Aug 27, 2019 2:15 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 2:14 pm 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
Now I'm finally perfect confused. Ok, no NT-Flag.

Regarding that flow of ( https://www.felixcloutier.com/x86/iret:iretd ):

Code:
PROTECTED-MODE:
    IF NT = 1
        THEN GOTO TASK-RETURN; (* PE = 1, VM = 0, NT = 1 *)
    FI;
    IF OperandSize = 32
        THEN
                EIP ← Pop();
                CS ← Pop(); (* 32-bit pop, high-order 16 bits discarded *)
                tempEFLAGS ← Pop();
        ELSE (* OperandSize = 16 *)
                EIP ← Pop(); (* 16-bit pop; clear upper bits *)
                CS ← Pop(); (* 16-bit pop *)
                tempEFLAGS ← Pop(); (* 16-bit pop; clear upper bits *)
    FI;
    IF tempEFLAGS(VM) = 1 and CPL = 0
        THEN GOTO RETURN-TO-VIRTUAL-8086-MODE;
        ELSE GOTO PROTECTED-MODE-RETURN;
    FI;
TASK-RETURN: (* PE = 1, VM = 0, NT = 1 *)
    SWITCH-TASKS (without nesting) to TSS specified in link field of current TSS;
    Mark the task just abandoned as NOT BUSY;
    IF EIP is not within CS limit
        THEN #GP(0); FI;
END;
...


This means: If NT=0, it will never take tss into account, so tss.esp0 and tss.ss0 will not applied.
If NT=0, it will additionally use SS:ESP from STACK, but only if privilege are upgraded (ISR is always R0, so, switch to a R3 will always take these values). If Level keeps the same ore lower, the values will not taken from stack.

So my big question mark: What mechanism triggers now, that SS0:ESP0 will be consumed? Which instruction "takes" that values and put them finally on the SS:ESP Registers = Task switch finished?


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 3:02 pm 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 671
I think you need to take a step back and forget about scheduling. It is first important to understand what happens when an interrupt occurs. Assume a simplified kernel that uses just Ring0 and Ring3. Assume that you have an appropriate IDT set up to handle external interrupts. Let us say for the sake of argument there is only one interrupt enabled and that is the timer.

You have to carefully follow through the pseudo-code for the INT instruction (which applies to external interrupts, but there is never an error code pushed for an external interrupt).

A simplification is that when you are in ring 3 and that external interrupt comes in SS0:ESP0 are read from the TSS, and SS:ESP are set to those new values. The original SS:ESP that at the point of interruption are then pushed on the stack the processor just changed us to as determined by the TSS. So the old SS is pushed, the old ESP is pushed, EFLAGS is pushed, CS is pushed and EIP is pushed. Then control is transferred to the CS:IP in the interrupt gate of the IDT. At this point we are now executing our interrupt handler in Ring0. Let us say the timer interrupt does nothing and simply does an IRETD.

Assuming NT=0 The IRETD will do:
Code:
    IF OperandSize = 32
        THEN
                EIP ← Pop();
                CS ← Pop(); (* 32-bit pop, high-order 16 bits discarded *)
                tempEFLAGS ← Pop();


So now EIP, CS, and EFLAGS have been popped off the stack. Then it continues down and reaches
Code:
    IF tempEFLAGS(VM) = 1 and CPL = 0
        THEN GOTO RETURN-TO-VIRTUAL-8086-MODE;
        ELSE GOTO PROTECTED-MODE-RETURN;
    FI;
We are not using VM tasks so VM=0 so it starts to do PROTECTED-MODE-RETURN. That is:
Code:
PROTECTED-MODE-RETURN: (* PE = 1 *)
    IF CS(RPL) > CPL
        THEN GOTO RETURN-TO-OUTER-PRIVILEGE-LEVEL;
        ELSE GOTO RETURN-TO-SAME-PRIVILEGE-LEVEL; FI;
END;
The RPL of CS on the stack is privilege level 3 (that was what we interrupted as to begin with). CPL is currently 0 so it starts doing RETURN-TO-OUTER-PRIVILEGE-LEVEL. It then does:
Code:
RETURN-TO-OUTER-PRIVILEGE-LEVEL:
    IF OperandSize = 32
        THEN
                ESP ← Pop();
                SS ← Pop(); (* 32-bit pop, high-order 16 bits discarded *)
At this point SS:ESP are popped off the stack which are the User space SS:ESP of the ring3 code that was originally interrupted. It then does some EFLAGS processing but when done transfer will be sent back to CS:EIP of the ring3 code we originally interrupted.

In the case of an interrupt occurring while already running in Ring0, the TSS isn't consulted. You can review the Int instruction pseudo-code for this case. The current ring0 kernel stack (SS:ESP) is used by the interrupt directly. The CPU pushes EFLAGS, CS, EIP onto it. When the IRETD is encountered it does the RETURN-TO-SAME-PRIVILEGE-LEVEL path which doesn't involve popping SS:ESP from the stack (they were never pushed). EFLAGS is popped (and processed) and EIP and CS are popped, no privilege level change occurs and code continues executing at CS:EIP popped off the stack by IRETD.


Top
 Profile  
 
 Post subject: Re: TSS requiered?
PostPosted: Tue Aug 27, 2019 4:13 pm 
Offline

Joined: Sun Aug 25, 2019 7:56 pm
Posts: 12
Hello Michael,
first of all, thank you for the greate explaination. This is really amazing.

The Pseudo-Code of the IRETD i read very often. But i never saw "the big picture".

The most important sentences i will remind:
Quote:
when you are in ring 3 and that external interrupt comes in SS0:ESP0 are read from the TSS, and SS:ESP are set to those new values

Quote:
In the case of an interrupt occurring while already running in Ring0, the TSS isn't consulted


So, now will test, if I understood it correctly: We are using the tss only to ensure, that when we are interrupting a Ring3-Task, that the ISR get a separate Kernel Stack? And if a Ring0 Task is interrupted, we do NOT get a Stack from tss (so we stay on the current kernel stack).

So right, so far?

Ok, now the tricky test, where i will fail :-P

Tss in considered when a Ring3 Task is interrupted. To change to another Ring3-Task, I push SS:ESP on the stack (like I'm doing it now) - stop....because they are already on the stack (because Ring 3 was interrupted), I need to pop them off before, correct? (I think, this is very tricky, because they are not at the top of the stack. Do I'm seeing something wrong?).

When switching to another Ring0 Channel, nothing needs to be considered. Just push CS:EIP and EFLAGS on the current Stack, with the EIP

Once SS0:ESP0 is initially set, will we ever need to change that on task switch?

By the way, here is my (real) code of the switching code. The code is working then all rings are 0, or all are ring 3. If mixed, it will not work. But I understand the matters.

Code:
private static void SwitchToThread(uint threadID, uint oldThreadID)
{
    var thread = Threads[threadID];
    var oldThread = Threads[oldThreadID];

    //Assert.True(thread != null, "invalid thread id");

    if (KConfig.TraceThreadSwitch)
        KernelMessage.WriteLine("Switching to Thread {0}. ThreadStack: {1:X8}", threadID, (uint)thread.StackStatePointer);

    var oldThreadWasUserMode = oldThread.User;

    thread.Ticks++;

    SetThreadID(threadID);

    PIC.SendEndOfInterrupt(ClockIRQ);

    if (thread.Status == ThreadStatus.Creating)
    {
        thread.Status = ThreadStatus.Running;
    }

    // K=Kernel, U=User, 1=Extra SS:ESP on Stack
    // K->K: 0->0: InterruptReturnKernelToKernel
    // U->U: 1->1: InterruptReturnUserToUser
    // U->K: 1->0: InterruptReturnUserToKernel
    // K->U: 0->1: InterruptReturnKernelToUser

    if (thread.User && oldThreadWasUserMode)
    {
        Asm.InterruptReturnUser(thread.StackStatePointer);
    }
    else
    {
        Asm.InterruptReturn(thread.StackStatePointer);
    }
}

ASM-Code (it's all the same for every version (yet)):
Code:
[BITS 32]
push ebp
mov ebp, esp
mov esi, [ebp+0x8]   ;get stack pointer argument
mov esp, esi      ;set stack pointer

mov eax, dword 0x23
mov gs,eax
mov ds,eax
mov fs,eax
mov es,eax

;mov [esp+13*4],esi
;mov [esp+14*4],dword 0x20
popad
add esp, 0x8

;mov [esp], dword 0xBBAABBAA
;mov [esp+4], dword 0xCCDDCCDD

sti
iretd


The Matrix in the Comment is the place, where I thought too complicated ;-)


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 16 posts ]  Go to page 1, 2  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot], SemrushBot [Bot] and 73 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group