The man, who was in his mid-20s, was pulled from the water at Newcomb Hollow Beach and given emergency first aid, but died in Cape Cod Hospital, police said, adding that the town's beaches would be closed for 24 hours.
According to local media, it was the first fatal shark attack in the waters off the state's coast since 1936.
Division of Marine Fisheries shark researcher Gregory Skomal told the Cape Cod Times the attack had likely been carried out by a great white. "I can't think if any other species that would do this," he said.
He said the shark had likely mistaken the man for a seal.
"I saw an eruption in the water, about 15 feet [4.5 metres] in diameter," witness Joe Booth told the paper, adding that he watched the man's friend go back into the water to help him.
"These two guys with boogie boards and fins were doing all these cool tricks, doing flips, spinning backwards,” another witness, Wendy Rennert, said.
She had just turned away when the attack occurred and the other man ran out of the water shouting "Shark! Shark!"
The attack comes a month after New York neurologist William Lytton was bitten by a shark at nearby Longnook Beach. He suffered severe injuries, but survived.