If the person is actively pushing the wall over, knowing full well that the wall is structurally unsound, and that there are innocent people behind the wall...yeah, I'd blame the pusher.
If someone takes out a concrete wall with a five-pound sledge, I'm gonna have harsh words for the guy who build the shoddy wall and the guy who deliberately vandalized it.