Well, here's my take on it.
First of all, I have never owned an M5 so not sure what kind of power it makes, but I'm sure it's a good engine. That said, I am assuming your are running a tuned pipe on it.
Typical mod engines with tuned pipes like to run in the 16.5k to 17.5k rpm range. (this is with a dry pipe, if you have a pipe with water going INTO it, then it will be slightly less). To me it still sounds you are at least 1000 rpm down from optimal.
Tuning the carburetor is done according to weather conditions more than load conditions, although tuned pipe settings will affect carb setting slightly. In general gas engine carbs are very forgiving and once you get it close and not running it lean or on the edge of lean, you don't really have to mess with them much. So for you basic testing it's almost a set and forget thing...you can come back to it later to fine tune.
Most good mod Zenoahs I've seen actually make there best torque and HP in the 15k range. We usually prop for 16.5 to 17.5 as stated before. So we are indeed over revving them somewhat, but that's good. Boats don't need all that power up top...we need it most when turning and getting back up after slowing down to avoid.
Find a prop that will put you in that rpm range while keeping a fair load on the engine on the way up there and you will have a faster boat I assure you. Adjusting the pipe is a must in tuning also, so you have to optimize it when you try different props, especially if they seem a bit "heavy" or hard to pull...see if you can fix that with the pipe, if not, prop is too much and step down one notch.
That's how I go about tuning my boats anyway. That said, the Magnum is a big boat and fairly heavy so you can't be expecting 60mph...I'm sure it would do a good 45 though. My fullsize boat run about 35 mph and I would pass it easy with my Magnum. I think the wet pipe system is a major factor limiting the rpm of the engine on that setup...it wasn't designed to go fast, was design for big water fun and user friendliness. I had one and loved it!
But to answer you question some of us do use HP and torque curves to help in prop selection. Mix that up with experience and trial and error!