Then there were doors that wouldn’t open unless you asked politely, or tickled them in exactly the right place, and doors that weren’t really doors at all, but solid walls just pretending. It was also very hard to remember where anything was, because it all seemed to move around a lot.
(Harry Potter and the Sorcerer’s Stone)
It seems that all is a distributive determinative (Angela Downing) for plural noun or mass one. And so I’m confused about what it refers to. Does it refer to anything? If yes, isn’t there grammatical problem?
It is not exactly a dummy pronoun; perhaps you might call it an “ambient” pronoun here, as in
It sure is hilly here.
How’s it going?
It’s moving too fast for me.
It’s a morgue there after about nine o’clock.
It doesn’t refer to anything in particular but to everything in general, or at least everything to which the predicate might be taken to refer.
Accordingly, it’s very easy for this it to take all as a modifier, since all merely intensifies the underlying sense.
It’s all dark in here.
It’s all hilly in the Ozarks.
How’s it all going?
It’s all moving too fast for me.
It’s all a morgue there after about nine o’clock.
So you could paraphrase it all seemed to move around a lot as “Everything seemed to move around a lot.”