Nanocomposite materials consisting of polymer matrix and inorganic salts in the form of nanocrystals of hydroxyapatite (HA) are regarded as superior candidates for bone treatment. A biomimetic nanocomposite scaffold with HA formation on the electrospun poly(vinyl alcohol) (PVA) nanofibrous structure by employing a Ca-P alternate soaking method was developed in this work. The calcium-containing PVA nanofibers were prepared by adding calcium nitrate to the starting solution prior to electrospinning, and then mineralized by Ca-P treatment in incubation solution. With this rapid and effective procedure, a continuous biomimetic crystalline HA layer could be formed successfully without the need of a prior chemical modification of the substrate surface under very mild reaction conditions. Moreover, the HA formed with a relatively accelerated growth had a carbonated and poor crystalline structure, resembling biological apatite in the bone mineral. The introduction of calcium ions in nanofibers by electrospinning was a favorable approach to induce the deposition of calcium phosphate and improve the distribution, nucleation, and growth of crystalline HA layer on nanofibrous scaffolds. Bioactivity tests revealed that these mineralized PVA/HA composite scaffolds improved the biocompatibility. The porous polymer/HA composite scaffolds produced in the present study might have potential applications in bone tissue engineering.