My first guess is because they were attacked by foreign invaders.
(but at the same time, both empires had split in their own way. The Han empire lost control and broke into different areas. The Roman Empire had become two parts, where the Western part was constantly attacked by Germanic tribes). Both sides were constantly attacked by invaders until there was no more land to take.
