I'll put my two cents in, as I've drilled a few holes...
For most shop work, you can use simple hand tools to do layout. This means scratching fine layout lines on things to locate a hole. I use various tools to do layout. If applicable, I use my dial calipers, as they can be set to the desired dimension and then scribe a line with respect to an edge. If I can't use those for some reason, then I'll try to use either hermaphrodite calipers or machinist dividers. For most work, however, a machinist rule is adequate. Draw lines or arcs to locate the centers of the holes you want. I keep a blue Sharpie marker in my shop apron to mark where the scribed lines will go to make them easier to see.
Then you want to put a center punch mark at the intersection of those lines. I've got a number of automatic center punches (bought 30-40 years ago), but I virtually never use them. Instead, I use a Starrett toolmaker's hammer and a prick punch. The toolmaker's hammer has a good magnifying lens built in, so I can see that the tip of the center punch is at the intersection of the lines. Then I just barely tap the prick punch with the hammer. I inspect the mark made with the magnifier and move it if necessary. Once it's where I want it, I give it a deeper mark with a center punch.
Then you use your drill and drill bit to start drilling. But the secret is to start drilling and just make a mark with the drill bit's tip. Inspect the mark with respect to the layout lines and, if it's not centered, you can correct it with a diamond or cape chisel (see any basic shop book written in the last 100 years about how to do this). Once it's where you want it, finish drilling the hole.
For fussy work, I like to start a hole using a center drill (as someone else mentioned). If you don't have one of those, you can also get a small (say, 1/4" diameter) screw machine drill that can work as a substitute. These drills are more rigid than the typical jobber's length drills.
Of course, as others have stated, it helps quite a bit to do this drilling on the drill press rather than free-hand. If you've properly center-punched the work, the spinning drill bit will pull the work to be centered under the drill bit when you lower it into the center-punched hole. If the material is hard to hold, you'll want to center it under the drill bit without the spindle turning, then clamp things to the table. You'll only get your hands mashed up once or twice before you get pretty good at knowing what needs to be secured and what doesn't.
Buy good HSS drills. Forget cobalt drills or titanium nitride coated, etc. Regular HSS drills will be fine for most shop work and, with care, should outlast you. My drill sets are near or over 50 years old and they'll be passed onto my descendants. If I had more discretionary cash, I'd buy a set of screw-machine length drills, as I find them useful for lots of my drilling tasks (these are basically shorter styles of drills used on automated machines).
You can learn to hand-sharpen drills (any machinist can show you how to do it in short order), but unless you do it regularly, you're probably better off buying something like a Drill Doctor. General Hardware made a simple die-cast drill sharpener that also works pretty well (I bought mine in the 60's for about $10). Starrett makes a drill-grinding gauge, although the industrial suppliers often give such things away.
You'll also need to countersink the holes you drill. This can be done with larger center drills (as I do often on my lathe), but a better tool is the Weldon-style countersink. These are countersinks with a hole through them and work the best of all the different countersink styles I've tried (i.e., chatter the least). I often just use a machinist's hand-held deburring tool instead; in fact, I mostly use a machinist's scraper for single holes. You can make such a thing from an old triangular file.
You'll find that careful layout and center-punching will serve for most of your drilling needs. I have a milling machine sitting next to my drill press, but 99% of the holes I drill are done on the drill press, as it's the most convenient tool and is adequate for most tasks.