In order to accurately identify the ship's roll model parameters in shallow water, and solve the problems of difficult estimating nonlinear damping coefficients by traditional methods, a novel Nonlinear Least Squares - Support Vector Machine (NLS-SVM) is introduced. To illustrate the validity and applicability of the proposed method, simulation and decay tests data are combined and utilized to estimate unknown parameters and predict the roll motions. Firstly, simulation data is applied in the NLS-SVM model to obtain estimated damping parameters, compared with pre-defined parameters to verify the validity of the proposed method. Subsequently, decay tests data are used in identifying unknown parameters by utilizing traditional models and the new NLS-SVM model, the results illustrate that the intelligent method can improve the accuracy of parametric estimation, and overcome the conventional algorithms' weakness of difficult identification of the nonlinear damping parameter in the roll model. Finally, to show the wide applicability of the proposed model in shallow water, experimental data from various speeds and Under Keel Clearances (UKCs) are applied to identify the damping coefficients. Results reveal the potential of using the NLS-SVM for the problem of the roll motion in shallow water, and the effectiveness and accuracy are verified as well.