The forerunner of A&P was founded in the 1850s as Gilman & Company by George Gilman (1826–1901) to continue his father's leather tanning business; in 1858 the firm's address was 98 Gold Street in Manhattan. Gilman's father died in 1859, leaving the son wealthy. That year, Gilman & Company entered the tea and coffee business from that storefront. One source speculates that Gilman decided to enter a more respectable business in light of his wealth. In May 1861, Gilman turned over the tanning business to his brother Winthrop; George moved his tea business to 129 Front Street. Initially, Gilman & Company was a wholesaler. In early 1863 the firm became a retailer, Great American Tea Company. Quickly, it opened five stores, moving its office and warehouse to 51 Vesey Street.

Gilman proved to be a master at promotion; the business quickly expanded by advertising low prices. The firm was able to offer low prices by acting as both the wholesaler and retailer. Gilman also built a nationwide mail order business. By 1866, the firm was valued at more than $1 million. In 1869, the transcontinental railroad was completed; Gilman changed the firm's name to Great Atlantic & Pacific Tea Company to promote the then-new concept of prepackaged tea under the Thea-Nector brand. The tea company continued to use the Great American name for mail-order purposes. In 1871, A&P introduced another concept when it offered discounts with the purchase of coffee and tea at its stores.

George Huntington Hartford (1833–1917) joined Gilman & Company as a clerk by 1861; he later was promoted to bookkeeper, then cashier in 1866. By 1871 Hartford was in a position of authority and was responsible for expanding A&P to Chicago after the great fire. A&P's first store outside New York City was opened within days after the disaster. The firm rapidly expanded; in 1875 A&P had stores in 16 cities. In 1878, Gilman left the active management of the firm to Hartford. By then, the firm operated 70 lavishly equipped stores and a mail order business with combined annual sales of $1 million.

To raise revenue, Congress significantly raised tariffs on tea and coffee. Profits on these products declined, and in about 1880 A&P started to sell sugar in its stores. The company continued aggressive growth and by 1884 operated stores as far west as Kansas City and as far south as Atlanta. The company also operated wagon routes to serve rural customers. About this time, two of Mr. Hartford's sons, George Ludlum Hartford (1864–1957) and John Augustine Hartford (1872–1951) joined the firm. Company lore is that George convinced his father to expand the product line to include A&P branded baking powder. Over the next decade, the company added other A&P branded products such as condensed milk, spices, and butter. As it expanded its offerings, the tea company was gradually creating the first grocery chain. By the end of the century, the firm had sales of $5 million from 198 stores as well as its mail order and wagon route operations.