When you say "never had any password" do you mean WIFI password or admin login password? If it is WIFI password, then someone could have accessed your network. That is potentially bad because it allows unknown devices behind your router firewall.
Not having an admin password is bad, but someone still has to have access to your network to take advantage of it.