Hepatocellular carcinoma (HCC) is one of the leading digestive malignancies, with a high metastasis and recurrence. The development and rapid progression of HCC involves numerous complex molecular and cellular events. Therefore, developing effective methods for the prevention and treatment of HCC requires an improved understanding of the biological development of HCC. In our previous analysis of the tissue microarray data, the BTB domain-containing 3 (BTBD3) gene was upregulated in HCC tissues, indicating that it may be a cancer-associated gene and serve a role in the occurrence and development of HCC. In the present study, reverse transcription-quantitative polymerase chain reaction and western blotting were performed to analyze the expression level of BTBD3 in four HCC cell lines; HepG2, Huh7, Bel7404 and Hep3B. The overexpression of BTBD3 in the four cell lines confirmed that BTBD3 was a cancer-associated gene. Subsequently, a short interfering RNA interference technique was performed to investigate the effect of BTBD3 expression on the proliferation and metastasis of Bel7407 cells. MTS assay and flow cytometry were used to evaluate the effect of BTBD3 on the proliferation and cell cycle, and a scratch test and Transwell assay were performed to determine the alterations to the migration and invasion of cancer cells. The results revealed that there was a minimal impact on cell proliferation following silencing of the BTBD3 gene. However, significant inhibition of cell invasion was demonstrated in the scratch test and the Transwell model. Based on these results, it was suggested that BTBD3 gene may be overexpressed in HCC tissues and cell lines, which promotes the invasion and metastasis of cancer cells without affecting cell proliferation.