To investigate the mechanism of structural changes of water and polymer networks with drying and swelling, we measured the Raman spectra of a physically cross-linked poly(vinyl alcohol) (PVA) hydrogel synthesized using the freezing-thawing method. The results show that the vibrational frequencies of the O-H and C-H stretching modes decrease with dehydration. The frequency shifts observed are attributed to reduction of free water inside the polymer network. The C-H bonds elongate as the water density decreases, and the average length of the O-H bonds increases with increasing proportion of bound water to the total amount of water. On the basis of the dependence of the frequency shifts on the PVA concentration of the original solution, it was found that the structure of the polymer network in the reswollen hydrogel becomes inhomogeneous due to shrinkage of the polymer network with drying. Furthermore, to investigate the effects of the cross-linking structure on the drying process, these results were compared with those of a chemically cross-linked PVA hydrogel synthesized using glutaraldehyde as a cross-linker. The result shows that the vibrational frequency of the O-H stretching mode for the chemically cross-linked hydrogel increases with dehydration, whereas that of the C-H stretching mode decreases. The opposite trend observed in the O-H stretching mode between the physically and chemically cross-linked hydrogels is due to the difference in the shrinkage rate of the polymer network. Because the rate of shrinking is slow compared with that of dehydration in the chemically cross-linked hydrogel, water density in the polymer network decreases. For the physically cross-linked hydrogel, the polymer network structure can be easily shrunken, and the average strength of hydrogen bonds increases with dehydration. The results show that the structures of the polymer network and water change with the gel preparation process, cross-linking method, and drying and reswelling processes. The structure of the polymer network and the behavior of water accommodated in the network are important factors governing the chemical and physical properties of gel materials.