Throughout its history, Brookline has resisted being annexed by Boston, in particular during the Boston–Brookline annexation debate of 1873. The neighboring towns of West Roxbury and Hyde Park connected Brookline to the rest of Norfolk County until they were annexed by Boston in 1874 and 1912, respectively, putting them in Suffolk County. Brookline is now separated from the remainder of Norfolk County.

Brookline has long been regarded as a pleasant and verdant environment. In 1841 edition of the Theory and Practice of Landscape Gardening, Andrew Jackson Downing described the area this way:

“ The whole of this neighborhood of Brookline is a kind of landscape garden, and there is nothing in America of the sort, so inexpressibly charming as the lanes which lead from one cottage, or villa, to another. No animals are allowed to run at large, and the open gates, with tempting vistas and glimpses under the pendent boughs, give it quite an Arcadian air of rural freedom and enjoyment. These lanes are clothed with a profusion of trees and wild shrubbery, often almost to the carriage tracks, and curve and wind about, in a manner quite bewildering to the stranger who attempts to thread them alone; and there are more hints here for the lover of the picturesque in lanes than we ever saw assembled together in so small a compass.”

Coolidge Corner, which is located at the crossing of Beacon Street and Harvard Street, is one of Brookline's two primary retail districts (the other being Brookline Village). It includes a number of historically significant sites, including the S.S. Pierce Building, and the Coolidge Corner Theatre.

the Country Club, an exclusive sporting club in the town, was the first private club in the United States formed exclusively for outdoor activities. It is most famous as a golf club; it was one of the five clubs that formed what is now the United States Golf Association and has hosted the U.S. Open three times and the Ryder Cup Matches once.

Larz Anderson Park is in Brookline on the 64-acre (26 ha) estate once owned by Larz Anderson and Isabel Weld Perkins. The park contains the Larz Anderson Auto Museum, the oldest automobile collection in the country, as well as Putterham School, a one-room schoolhouse from colonial times.

The birthplace of John F. Kennedy stands in Brookline and is listed in the National Register of Historic Places. It is maintained by the National Park Service and is open to the public from May through September.

St. Aidan's Church was where John F. Kennedy was baptized and where the Kennedy family and other prominent Irish-Americans were parishioners. The church was designed by architect Charles Maginnis, who was awarded the American Institute of Architects' gold medal. Though it is on the National Register of Historic Places, St. Aidan's Church has been closed and converted into housing.

The Puppet Showplace Theatre, one of the four oldest puppet theatres in the United States, is located in Brookline Village.

The Dutch House, one of only five surviving buildings from the World's Columbian Exposition of 1893 was relocated to Brookline.

John Goddard House, an historic house at 235 Goddard Avenue, was built in 1767 and added to the National Register of Historic Places in 1985.