The Kampung Kling Mosque's original structure, built by Indian Muslim traders in 1748, was a wooden building and in 1872, it was rebuilt in brick.

The mosque is one of the traditional mosques in Melaka, which still retains its original design. The architectural design of the mosque is a cross between Sumatran, Chinese, Hindu, and the Melaka Malay. The minaret, ablution pool and entrance arch were built at the same time as the main building.

The minaret resembles a pagoda. The mosque also has a blend of English and Portuguese glazed tiles, Corinthian columns with symmetrical arches in the main prayer hall, a Victorian chandelier, a wooden pulpit with Hindu and Chinese-style carvings, and Moorish cast iron lamp-posts in the place of ablution for pre-prayer cleansing.