I don't want to get into any legal problems as there is a contract. I also don't want to upset anyone because I am neighbors with them. I just don't have the time any more to commit to what I told them I would do.
That's the problem when doing business with friends or people that you know; In some sense you are using the trust built-up by the 'friendship' as basis for getting the job.
Either way you are going to disappoint someone; you need to figure out which party it's going to be. If you are legally bound then you risk more than a ruined friendship, have a look at the contract and see if there is a way to get out of it (legally). What josh mentions about giving the money back could be an option in this case.
Turning down a fixed income job is risky; freelancing is good if the money / work is constant; remove either of those from the equation and it's tough to stay afloat. I would take the job, the neighbors should be able to understand (once you give their money back) your dilemma.