The inside of Palazzo Pitti has 5 museums, so you have to decide which ones to see and which ones are less interesting for you.... so if you decide for the Palatine Gallery + Royal Apartments, for example, you could do those in the morning, head out nearby for lunch and return to see the gardens in the afternoon.
If you want to see more of the palace museums, then I would spread it out over two days. The gardens themselves are large and have various sections so you do need at least the entire afternoon to spend 2-3 hours there. If there is sun, enjoy sitting on a bench and the view of Florence (near the Kaffehaus there is a great view).