Tuberculosis (TB) is a major global health problem, and multi-drug-resistant TB (MDR-TB) and extensively drug-resistant TB (XDR-TB) are spreading throughout the world. However, conventional drug susceptibility test (DST) methods, which rely on the detection of the colony formation on a solid medium, require 1-2 months to the result. A rapid and accurate DST is necessary to identify patients with drug-resistant TB and treat them with appropriate drugs. Here, we used microscopic imaging of Mycobacterium tuberculosis (MTB) immobilized in an agarose matrix for a rapid DST. The agarose matrix, which was molded in a microfluidic chip, was inoculated with MTB, and TB drugs in liquid culture medium diffused throughout the agarose to reach the MTB immobilized in the agarose matrix. After the responses of MTB to drugs were tracked with an automated microscopic system, an image-processing program automatically determined the susceptibility and resistance of MTB to specific doses of TB drugs. The automatic DST system was able to assess the drug susceptibility of various drug-resistant clinical TB strains within 9 days with an accuracy comparable to that of conventional method. Our rapid DST method based on microscopic time-lapse imaging greatly reduces the time required for a DST and can be used to rapidly and accurately treat TB patients.