Antonia Fraser’s bestselling 2010 memoir “Must You Go?” was a tender, tart chronicle of life with her husband Harold Pinter. In her new memoir, “My History,” Fraser turns the clock back further, to her enchanted yet fraught childhood. Born into British aristocracy and a daughter of a well-known politician and a biographer, Fraser’s early years provide material as vivid and character-rich as her popular histories. 

Granted, it’s hard to go wrong with a youth as glamorous as Fraser’s. Though she favors a lightly self-deprecating tone, alluding to her unpopularity at boarding school and her “undue self-esteem,” she also writes about spending Christmas in 1950 with the Italian prime minister’s family and having Cecil Beaton volunteer to take her (first) wedding photographs. (She divorced conservative MP Sir Hugh Fraser in 1977.) 

The prime minister’s invitation was primarily due to her parents, staunch English socialists — her father held posts in several postwar Labour governments — and Catholic converts who were in every other way totally different from each other. Nonetheless, eccentric, impractical, aristocratic Frank Pakenham and capable, critical, proudly middle-class Elizabeth Longford remained devoted as they argued through their nearly 70-year marriage. Fraser paints nuanced portraits of both, as well as a candid one of her turbulent teenage relationship with her mother, a.k.a. “Mummy Ogress,” which softened as Longford emerged from her prolific childbearing years (Antonia was the eldest of eight). Longford began to write history around the time her daughter gave birth to the first of six children. “Now,” writes Fraser, “we had all the most important things in common.” 

History had been Antonia’s obsession since the precocious 4-year-old read “Our Island Story,” a children’s book, in 1936....