You can walk around around immediately post-op. It's recommended to get up and around as much as you're able to.
Personally if I could go back, I would've taken a whole two weeks off of work. I didn't take painkillers after the first day, and I was up and as active as possible the first week post-op before my drains came out. No matter what, it's exhausting. You may feel like the anesthesia has worn off after a day or so, but it takes a while to fully leave your system. As a procedure, top surgery is not terribly invasive and not as huge a deal as other procedures. But it's still surgery. You will be tired. You will be less mobile than you're used to. You will be uncomfortable.
Not telling you what to do, I just think you're underestimating it and I don't think your surgeon is giving you a realistic picture. I had a very, very easy recovery and I regret only taking a week. I would highly recommend taking the full two weeks off, even if it means you need to postpone surgery to save up more money to handle the extra time off.