The idea for The Bus Driver came to Segal on a late-evening bus from New York to New Jersey. The driver was grim, sullen, and arrogant, and Segal caught himself thinking, "My God, dare I trust my life to this prig?" Soon afterward he found a derelict bus in a junkyard and hacked out the driver's platform. Incorporated in The Bus Driver, this metal armature pens in a plaster figure-a life cast. (The model was Segal's brother-in-law.)

When they first appeared, in the early 1960s, Segal's plaster molds of people in fragments of real environments were considered Pop art, since they described the everyday life of public places. But where Pop often focused on mass-media images and mass-produced objects, Segal was interested in individuals, their gestures, statures, stances, and also their inner, psychological or spiritual condition. He often left his plaster molds unpainted, valuing their whiteness for "its special connotations of disembodied spirit, inseparable from the fleshy corporeal details of the figure." In the bus driver (who has been likened to Charon, the ferryman of Greek myth who guides dead souls to the underworld), Segal saw "the dignity of helplessness—a massive, strong man, surrounded by machinery, and yet basically a very unheroic man trapped by forces larger than himself that he couldn't control and least of all understand."