They eventually moved to London in 1952. John performed at Conway Hall in London in 1955, making enough of an impression that the famous guitarist Andrés Segovia invited John to study at his courses at the Accademia Chigiana in Siena, Italy. John accepted and became a student of the pioneering guitar soloist from 1957 to 1959.

Williams made his official debut at London's Wigmore Hall in 1958, and received reviews that noted a strong, clean tone and a polished though undemonstrative technique. However, Williams does not give Segovia or his other official teachers a large share of the credit for his technique. He says that most of these teachers were too "authoritarian" in their approach, not excluding Segovia who, he says, had a tendency to expect his pupils to adopt his interpretive "mannerisms," and would get quite angry when they didn't. The guitarist with whom he formed the closest association is Julian Bream, a fellow student of Segovia. Bream has often appeared in concert and on recordings as a guitar duo with Williams.

Williams has toured throughout the world. He has performed and recorded nearly the entire standard guitar repertory, plus a large quantity of transcriptions. Several of these transcriptions are by his own hand. He was a professor of guitar at the Royal College of Music in London from 1960 to 1973. However, he also has a strong tendency to explore music outside the classical tradition. He does session work on film soundtracks, has arranged Beatles songs, and plays electric guitar in Sky, a classical-rock fusion band. He has also formed his own ensembles, John Williams and Friends and Attacca, to explore other music. On a CD release called The Guitarist, he uses Turkish and Greek rhythms and harmonies to support Medieval music. The 2002 album, The Magic Box, examines African music.