Goals by Lars Stindl and Leon Goretzka, either side of a penalty by Julian Draxler, saw Germany home to leave them second in Group B behind Chile, who they face in Kazan on Thursday.
Celtic midfielder Tommy Rogic had made it 1-1 in the first half and Tomi Juric added Australia's second after the break from a goal that was allowed to stand despite a suggestion of handball leading to a review from the video assistant referee.
The Socceroos then had a string of late chances to equalise, but Germany held out.
"First and foremost, I am happy we picked up the three points," beamed Germany coach Joachim Loew, who punched the air at the final whistle.
"We played well in the first half -- the only bad thing was that we were only 2-1 up at the break.
"We lost our way after 60 minutes and need to work on retaining our stability, but a lot of things worked well."
Australia next face Cameroon, who lost 2-0 to Chile in their opening game, in Saint Petersburg on Thursday with both teams needing a win in their bid to reach the semi-finals.
"It's disappointing, we were barely clinging on in the first half," admitted Socceroos coach Ange Postecoglou.
"The second half was better, we composed ourselves and got back into things.
"It's a loss and the loss falls on me, it's my responsibility."