Improvements on the kernel estimation in line transect sampling without the shoulder condition

Mack et al. (Comm. Statist. A 28 (1999) 2277) proposed to use the boundary kernel method to estimate the wildlife population density from the line transect data when the detection function does not satisfy the shoulder condition, the first derivative of the detection function at distance zero is 0. It was demonstrated in their paper that the boundary kernel estimator (with the use of the optimal end point kernel developed in Zhang and Karunamuni (J. Statist. Plann. Inference 70 (1998) 301)) performed significantly better than the reflection method (Schuster (Comm. Statist. A 14 (1985) 1123)). However, the boundary kernel method has two main drawbacks: the estimates may be negative and the boundary kernel estimator is always associated with a large variability. These two drawbacks of the boundary kernel method have been noticed in Mack et al. (1999), but no remedies were discussed in their paper. Zhang et al. (J. Amer. Statist. Assoc. 94 (1999) 1231) offered a new method to correct these two drawbacks of the boundary kernel method. It is shown in Zhang et al. (1999) that their proposed method performs better than the boundary kernel method in the sense that it has much smaller mean squared error value than that of the boundary kernel estimator. In this paper, we discuss how to apply their method (it will be called the new method throughout this paper) to the estimation problem in line transect sampling. It is observed that the new method is superior to the boundary kernel method from both asymptotic theoretical results and simulation results.