Three brothers say their mother abandoned them as children, leaving them to float from home to home for most of their young lives, while she moved away and started a new family.