For people (like me) who came to the Mac from Windows, there's one big difference with many (but not all) programs when you close them. In Windows, closing the program's window usually (99.9% of the time) ends the program. It's removed from system memory (RAM), freeing up that space. On the Mac, however, that's not necessarily true. Sometimes you're just sending the window away, but the program is still running in the background. (Some programs do go away when you close the last window, but not the majority of them).
This can have a big effect on your available memory. For example, I just took a look at the memory on my system in the Activity Monitor (which you'll find in the Utilities folder if you're on 10.5 or 10.6 – I can't remember when they changed the name, but it was called something else before that). One program had been up for about a week – NetNewsWire, my RSS reader (and a pretty good one it is, too, if you are looking for an RSS reader). It was using up 500 MB of RAM – half a gig! I sent it away (with Command-Q, which does completely end a program, just like going to the program's menu and selecting Quit) and started it over – and it came up using 75 MB.
Why the huge difference? As an RSS reader, it goes out to a lot of web sites (I have 130 feeds that I follow) and as I go through the new listings, many have graphics in them. Also, there's quite a few of them that I load up in the internal browser each day. NetNewsWire is caching the graphics (and probably the texts, too) so that if I go back to them, they are instantly available. Safari does this too, caching graphics and web pages (which is why if a page changes you don't always see the change without forcing Safari to refresh the page).
All that caching has to go somewhere. Some caching is done to disk, and some to memory. This is the decision of the programmer, so unless there's something in the preferences that lets you fine-tune what's cached, there's not much you can do about it.
Now, you won't run out of memory (unless you leave a lot of stuff running) because OS X will use swap space, or disk, to move out chunks of memory you haven't used recently (like that photo you were editing but haven't touched in a few minutes) for another program if there's no free memory. Of course, you could run out of free disk, and then you will be out of memory. But unless your disk started out with little free space, that's usually not an issue.
But swapping is slow. You hard drive is many many many times slower than the RAM used by active programs. So it's a good idea to fully Quit a program every so often to free up the memory. I know at first for me this was an odd concept – I've closed all the windows, why is the program still showing in the Dock with the little "running" pip? Well, some programs (like NetNewsWire) can be doing a lot of work behind the scenes, even if there are no windows open.
Leaving something running in the background, like Pages, means if you go back to it a few minutes (or hours) later, it will spring to life immediately (since it is still running and doesn't have to go through all the startup work). If you've got a lot of memory, this can be a good way to easily speed up you machine (and one reason Terry White and I agree that getting as much RAM as you can will be the easiest and often cheapest way to speed up your computer).
But if you are RAM constrained – especially on an older system, where you might be limited to 1 or 2 GB of RAM total – you need to keep a close eye on the Dock, and send away programs you aren't using. Or if it's something like NetNewsWire, send it away every so often to free up memory.