Bad idea. If someone hits someone with a car, should the automaker be held accountable? If I take my computer monitor and throw it at someone, should the maker of the monitor be held accountable?
The reason we don't hold monitor makers, auto makers,gun makers and yes, ISP's accountable is because they cannot control what we do with their product.
Even more so if we give an ISP the right to police its customers access, then aren't we handing over censorship to a private entity? Whats to prevent them from censoring free speech? Think of all the little things an ISP could do, they couldn't outright block everything but they could certainly be sneaky about what you can and cannot see.
Then there is cost. ISP's aren't hurting (contrary to their claims), but to police EVERY internet user will require time, money, and lots of people (which also need time and money by themselves).