At what point does one admit that their product might never be as successful as they had thought?

You come up with an idea. Then you run it by your friends and family. You think you have got something real good on your hands when they approve of it. Now, after spending around 300 hours on building it you realize that it might not be as good. You have done the promotion, you have done good development. You cannot figure out what is missing or what is wrong with product/idea, but you just can not get people to use it. Would you go further with it? If yes, then at what point would you decide to quit if the same goes on.