Scene segmentation and semantic labeling of Synthetic Aperture Radar (SAR) images is one of the key problems in interpreting SAR data. In this paper, a new approach for semantic labeling of SAR imagery is proposed based on hierarchical Markov aspect model (HMAM) with weak supervision. The motivation for this work is to incorporate the multiscale spatial relation between adjacent image patches into supervised semantic labeling of large high resolution SAR image. Firstly, the large SAR image is divided into hundreds of subimages, and the semantic keywords of each training subimage are given by the user. Then, the HMAM is presented by building markov aspect model based on quadtree which can explore multi-scale cues, spatial coherence and thematic coherence simultaneously. Next, we use the trained HMAM model to classify each patch of the unlabeled subimages into a given semantic classes. Finally, we regroup all the labeled subimages into the large SAR scene labeling result. We also elaborately build the ground truth map for a whole scene of TerraSAR-X image to evaluate the labeling results quantitatively. The experimental results on TerraSAR-X dataset show that our labeling method is effective and efficient, and the HMAM can improve labeling performance significantly with only a modest increase in learning and inference complexity than aspect model.