Pedestrian tracking is highly relevant to the understanding of static and moving scenes in video sequences. The increasing demand for people’s safety and security has resulted in more research on intelligent visual surveillance in a wide range of applications, such as moving human detection and tracking. With the great success of deep learning methods, researchers decided to switch from traditional methods based on hand-crafted feature extractors to recent deep-learning-based techniques in order to detect and track people. In this work, the topic of person detection using a Top-view moving fisheye camera is addressed using a deep learning detector combined with a centroid technique in order to track a selected person. Although the fisheye camera is a useful tool for video monitoring, most object detection techniques use classical perspective cameras, with (or without) deep learning. However, due to the distortions of fisheye images, we expect to have higher requirements and challenges on pedestrian detection using this type of device. In this paper, an end-to-end people detection learning method is proposed; it is based on a YOLOv3 detector that detects people using oriented bounding boxes. The proposed model customizes the traditional YOLOv3 for the detection of oriented bounding boxes, by regressing the angle of each bounding box using a periodic loss function. With rotation bounding box prediction, the new approach is efficient, reaching 98.1% of true detection. This detection model is combined with a centroid tracker in order to track a single person by identifying the trajectory, estimated angle of rotation and target distance. Finally, the proposed method is evaluated on a new available dataset where rotated bounding boxes represent annotations from several fisheye videos.