Headless is can be better, it needs less power so you do away with the extra power unit. Also because you are using SSH it comes up on the screen your using, be sure you change the password and login details. There is alot of information on the web about this, its worth looking at the pi site. I would advise against the full kit, simply because you dont need it all and some of it you can get cheaper shopping around.
Sure a case is nice but ebay does have cheaper ones, same with external psu, setting up dosnt need a external psu but get one if your going to be pulling much off the pi. Also look at what your using it for, if your giving it alot of work then think about buying a heat sink for it, again ebay or spare parts bin is the way to go.
If your running much off the usb ports then please consider a powered USB hub. If you get the pi3 then look into how to boot from an external HDD, honestly this saves massive headaches with sd cards failing, and trust me they do fail. You can also boot a pi 3 from usb stick, you can boot the pi2 from a usb stick as well but booting from external HDD is alot harder on the pi2.
I dont use a case, i simply use PCB stand offs for mine, unless there is a chance of it getting wet etc, then i simply use a plastic tub. It dosnt have to be expensive and depends what you use it for, i tend to not use the IO so much, i use micros and other dev kits, the pi i normally use for data logging and as a way to access the other systems via the web.