3 inches off plumb on a 6 foot post is actually about a 2.8 degree lean, which is technically within the 2-3 degree tolerance most inspectors let slide on fence posts. You can try to shim it with some gravel or break the concrete loose around the top couple inches to get it closer. If the concrete has really set hard, you might need to chip out around the post and repour a new collar. Just don't try to muscle it over after the concrete has set, that usually cracks the footing and makes things worse.