Weaker amp, stronger speaker --> nothing bad will happen. The worse thing could happen is only that the sound would perhaps not loud enough. You can simply try and see.
Stronger amp, weaker speaker --> bad thing could happen only on higher volume (quite rare to happen). You might wanna be careful a bit. Sound could be also too loud on lower volume.
Your case: weaker amp, stronger speaker. Just try and see

.