Your opinion is noted, however, since I've been designing custom boxes and used to work with several competition level shops doing custom fabs, I'm telling you flat out that you are wrong.
This goes for all speakers not just subs. Ideally the volume of the box creates a dampening effect on the movement of the speaker. If the volume is too small, it will dampen it too much, especially if it is a sealed enclosure. This will cause loss of frequency and introduce distortion. Distortion creates heat and heat sure as ********* hell WILL damage your voice coils and possibly the driver as well.
It will also reduce how fast the speaker can move - this also has many effects on performance including the potential for clipping. Note that if the walls of the enclosure flex - are not rigid - this compromises performance as well as the volume changes dynamically. All speakers benefit from having the proper type and size of enclosure and when an enclosure is significantly too small or too big, or the ports are not the right size, length and diameter, there will definitely AT MINIMUM be distortions or sound issues like lack of volume or muddy sound and at worst may, and in many cases does, damage the speaker in a relatively short amount of time.
It also takes more power to achieve the same amount of volume if the enclosure is too small. The small size of the box makes it harder for the speaker to move. As the speaker moves, the pressure difference inside the box is increased due to the small size of the box. If you are running a small amp, the reduced box size will cause additional strain on the amplifiers as well. Finally, if the speaker is not a high quality one with a very stiff cone, the small box will cause additional distortion due to increased cone distortion from the higher pressures. In summary, the system becomes much less efficient and potentially damaging to both the speaker and the amplifier.
And if the box is ported, and the volume is not accurate for the specific speaker as per the manufacturers suggested dimensions, it will definitely sound like crap.