Recently, video sharing services such as YouTube and Daily-motion have become popular and many videos taken with mobile video cameras are uploaded to such a video sharing service. However, such videos can infringe on the privacy right of people in the videos because they may contain privacy sensitive information (PSI) of the people, i.e., their appearances. This strongly motivates us to develop a technique to generate privacy-protected videos. In this paper, we propose a novel system for automatic generation of privacy-protected videos based on background estimation. In most conventional techniques, objects that contain PSI are detected and obscured by, e.g., blurring. Conversely, in our system, background pixels are estimated and then substituted with intended human objects that are essential for the camera person’s capture intention. We quantitatively evaluate our system to demonstrate its potential applicability.