Is this sentence correct "They returned back to the house because they forgot the dog"?.

Actually, a part of it is a bit repetitive.  According to
a great source I often use, Purdue OWL, you need to "eliminate words that explain the
obvious or provide excessive detail."  On their site, they say
this:

"Always consider readers while drafting and revising
writing. If passages explain or describe details that would already be obvious to
readers, delete or reword them.

In your sentence, using
"returned" and "back" makes the sentence somewhat repetitive.  The correct form of the
sentence would become:

"They returned to the house because
they forgot the dog."