Hey shooter, I realize it has been a long time since your post but I think that you should go for it.
Only two other suggestions:
1. Don't swap! keep the .44
2. Why not try the most powerful .500 S&W or the fastest factory load .460
One bonus with the .460 is that you could shoot the .454 and .45 Colt.