1.Does malloc need to be aware of paging? If so does that mean that I have to map every allocation with mapPhysicalToVirtual?
It may be very loosely aware of page translation and virtual memory. For example, you may have a mechanism to automatically map new pages into the address space when they're accessed. In this case malloc() needs not do anything (or much). If you make this mechanism work in selected regions of the address space, malloc() will only need to enable it once for the region that is reserved for the heap.
A somewhat similar approach is when you implement your heap on top of (s)brk. The brk system call may perform the mapping and unmapping and malloc() will only need to call this system call to extend the heap size. ((s)brk works like allocating stack storage by moving the stack pointer, look it up)
Otherwise malloc() will need to do something every time it needs a new page of memory (or can release a page to the system).
2.Do I need to map my linear frame buffer using malloc so it gets sort of protected from overwriting? (I am already mapping it using physicalToVirtual)
malloc() is intended to be used to allocate arbitrary memory for use (see any C reference), which implies address space allocation, page frame allocation and mapping of physical pages into the address space. The frame buffer already is memory and it has a very specific use unlike RAM. It is not supposed to be a store of pages for arbitrary use. It just needs to be mapped into the address space.
3.Does back buffer for double buffer system need to be mapped using physicalToVirtual as well?
Any kind of memory that software needs to access must be mapped into the address space, whether it's RAM or a memory-mapped device.
4.How do I test malloc and see if it is all okay?
Use your imagination. That's what OS devers are supposed to do.
My advice is to test separately: page frame (de)allocation, (un)mapping, address space (de)allocation, malloc()/free(). Don't forget to test the underlying linked lists and sorted trees. Actually, do that first. When it works in a single thread, extend the test for multiple threads concurrently running malloc() and free().