Venus closed out the 29th career meeting between the two on her second match point as Serena sailed a forehand long to end the third round match in front of a main stadium crowd of about 13,000.
It was the first meeting between the sisters since they clashed in the final of the 2017 Australian Open, which Serena won before taking a 15-month hiatus due to her pregnancy.
Serena said she is still easing her way back into match fitness.
"It wasn't very easy, obviously," she said. "It was good to play and try to get in the rhythm and get into the swing again.
"I can't really replicate the situation no matter how much I do in practice. I make those shots 10 times out of 10 in practice.
"It's just the nerves, the anticipation you feel naturally. It's a little bit of everything that comes in a match that just doesn't normally happen."
The 10th seeded Venus moves on to the round of 16 where she will face Anastasija Sevastova, who defeated 12th seeded Julia Georges 6-3, 6-3.
It is rare for the Williams sisters to play this early in a tournament. It is the earliest they have faced each other since their first encounter at the Australian Open in 1998.
They arrived outside the stadium together on a golf cart then walked through the tunnel with Serena entering the court first, as many in the crowd stood and cheered.
Venus blasted six aces but had eight double faults in the one hour 26-minute main stadium match.
Serena is still shaking off the rust after the long layoff as she hit four aces but had her serve broken four times.
"I just have a long way to go," Serena said.
Despite the loss Serena still leads their career series 17-12.
This was their first encounter with Venus as an aunt and Serena as a new mother after giving birth to her baby daughter, Alexis Olympia, on September 1.