Segmentation of pancreatic tumors on CT images is essential for the diagnosis and treatment of pancreatic cancer. However, low contrast between the pancreas and the tumor, as well as variable tumor shape and position, makes segmentation challenging. To solve the problem, we propose a Position Prior Attention Network (PPANet) with a pseudo segmentation generation module (PSGM) and a position prior attention module (PPAM). PSGM and PPAM maps pancreatic and tumor pseudo segmentation to latent space to generate position prior attention map and supervises location classification. The proposed method is evaluated on pancreatic patient data collected from local hospital and the experimental results demonstrate that our method can significantly improve the tumor segmentation results by introducing the position information in the training phase.