Berger played piano in Germany when he was ten and worked in his teens at a club in Heidelberg. He learned modern jazz from visiting American musicians, such as Don Ellis and Leo Wright. During the 1960s, he started playing vibraphone and received a doctoral degree in musicology. He worked as a member of Don Cherry's band in Paris. When the band went to New York City to record Symphony for Improvisers, he recorded his debut album as a leader.

With Ornette Coleman and Ingrid Sertso, he founded the Creative Music Studio in Woodstock, New York, in 1972, to encourage students to pursue their own ideas about music. Berger considered Coleman his friend and mentor, and like Coleman he was drawn to avant-garde jazz, free jazz, and free improvisation.

He has worked with Carla Bley, Dave Holland, Lee Konitz, John McLaughlin, Sam Rivers, Pharoah Sanders, Gunther Schuller, Clifford Thornton, the Mingus Epitaph Orchestra, and the Globe Unity Orchestra. He collaborated with Bill Laswell as musical arranger and conductor, thus contributing to albums by Jeff Buckley, Better Than Ezra, Buckethead, Natalie Merchant, Sly & Robbie, Angélique Kidjo, Hōzan Yamamoto, and Shin Terai.