There is no "answer"! So many variables but I'll make a few comments...
Packs specifically for canoe tripping don't sell in big numbers so generally not a big selection at retail. Packs designed for backpacking or alpine don't work well for canoeing although they are nice when you have a long portage.
In the vinyl dry bag style the 115L is pretty much the standard, I wouldn't want one any bigger, it would weigh a ton, be hard to pack in a boat and a pain to get at anything if you filled it up. There is no requirement to fill them up and you need enough empty space to get a good seal on the roll-top.
Your post mentions "give portaging a try", how about less portaging and adding in a little paddling!!!!!
I'd recommend one 115L drybag per boat, one smaller 60L food barrel/bag, two more (1 per person) for clothing, sleeping bag personal items. Use the 115L as a gear bag, tent, sleeping pads, kitchen etc.
If you are doing routes where the portages are long you would want to rethink that plan, maybe eliminate the 4th pack. Of course you could always go semi-ultralight and do single carries but unless that's really your thing you should probably take a nice selection of "comfort items" (functional or edible), double carry and pick routes that are not too brutal.