First thing I’d do is find a local kayaking cluband join it. You’ll shorten your learning curve significantly by gleaning knowledge from experienced boaters rather than re-inventing the wheel yourself. Plus you’ll be a lot safer joining group outings than trying to learn solo. And, many clubs will likely have access to “group equipment” like boats that you can river test before buying your own.
Ask around for local clubs in your area. Any local kayak dealer will know of one. Also ask how welcoming the clubs are to newcomers, and gravitate to the one with the best rep. Then, go to a meeting and volunteer to do something for the club. Every club needs volunteer help at something, and you’ll instantly make new friends.
As for gear, rent what you can for now (boat-wise) to get a sense of what works best for you. Try different boats to see how the varying fits and hull designs affect the performance of each and to see what suits you best. Helmet, PFD (personal flotation device), and stuff like that you’ll want to buy; REI and L.L. Bean, among other places, have lots of good, affordable gear. You can also find stuff on eBay, of course. And those new friends of yours in the local kayak club are bound to have gear they’re selling. In time, you’ll likely even find a boat through club members who are “upgrading.” So keep your eyes peeled and ears open.