Question about my Boxers Fracture?

I recently suffered from Boxers Fracture. After 4 weeks since the injury the swelling has gone down and I have full mobility. Without any pain however my pinkie knuckle where the injury occured is noticably larger than the non injured hand. Is that something to visit my doctor about or is it nothing to stress about?