Took a job for 3 weeks and now unemployment wants all their money back.?

I assume you didn't report the work you did? If you didn't, then yes you have to pay it back and rightfully so. If you DID report it, then they should of made the correct payments to you and you wouldn't owe them anything