Dlamini-Zuma, South Africa's home affairs minister and an ex-wife of President Jacob Zuma, on Sunday defeated incumbent Jean Ping of Gabon, who had been at the helm of the Commission, the AU's steering body, since 2008.
Dlamini-Zuma, a 63-year-old who has previously served as minister of health and foreign affairs, had to undergo three voting rounds before Ping, 69, was finally eliminated.
A final confidence vote of 37 in favour gave her the 60 per cent majority she needed to be elected.
"Now we have the African Union chair Madame Zuma, who will preside over the destiny of this institution," Benin's president and current AU chairman Thomas Boni Yayi said.
The contest to head the Commission of the 54-member AU had been deadlocked since last year, with neither Dlamini-Zuma nor Ping winning a two-thirds majority.
It pitted French-speaking states, largely backing Ping, against mostly English-speaking countries, especially in southern Africa, which gave their support to Dlamini-Zuma.
'Means a lot for Africa'
Her former husband, South African President Jacob Zuma, was one of the first to offer his congratulations after the vote.
"It means a lot for Africa ... for the continent, unity and the empowerment of women," Zuma said.
The impasse over the candidates had persisted through a summit of AU heads of state held in Addis Ababa at the weekend.
It prompted Yayi to warn that failure by the continental body to resolve the leadership deadlock would divide it and undermine its credibility in the world
Critics say the AU showed itself hesitant and slow-moving in its response to the conflicts last year in Libya and Ivory Coast, allowing Western governments to take lead roles.
"She's a freedom fighter, not a bureaucrat or a diplomat," said Ugandan President Yoweri Museveni.
Noureddine Mezni, spokesperson for the outgoing chairman, told the AFP news agency that Ping had acknowledged defeat.
He "has accepted the results of the elections and wishes Madame Dlamini-Zuma the very best ... he expressed his readiness to co-operate with her to work together for the unity of the continent," Mezni said.
Some analysts say South Africa has violated an unwritten tradition that continental powerhouses do not run candidates for the post, but leave smaller nations to take the job.
Before the vote however, Dlamini-Zuma played down concerns that the vote could divide the AU.
"I don't think the continent will be polarised," she said.
The winner would "make sure they work with everybody, irrespective of where and who they voted for," she added.
Jakkie Cilliers of the South Africa-based Institute for Security Studies said she will be "an inclusive chair, she won't be divisive ... she was a very competent foreign minister and even better at home affairs."