The response of a homeotropically aligned nematic liquid crystal layer to oscillatory rectilinear shear (Couette flow) was investigated for frequencies f between 0.01 and 200 Hz, and layer thickness d between 10 and 130 um. Below the onset of instability the cell was laced between crossed polars and light transmission was studied using a parallel light beam. The experimental results for the transmitted light intensity agree quantitatively with numerical solutions of the nematodynamic equations for different cell thicknesses, oscillation frequencies, and amplitudes. For frequencies between 25 and 150 Hz the critical oscillation amplitude for the onset of a spatial pattern, observed in polarized white light, could be reached.The pattern consisted of stationary rolls perpendicular to the direction of the oscillation.The experimentally obtained frequency dependence of the critical shear amplitude for the roll instability for different cell thicknesses is compared with an existing theory and the results of numericalcalculations.