Abstract: Privacy protection of magnetic resonance imagery (MRI) is challenging. Even when the metadata is removed, brain scans are vulnerable to attacks that match renderings of the face to facial image databases. As sharing of data is essential to scientific and medical progress, solutions to de-identify MRI scans that obfuscate or remove parts of the face have been developed. However, we show that these existing solutions either fail to reliably hide the patient's identity or are so aggressive that they impair further analyses. We go on to identify a new class of de-identification techniques that, instead of removing facial features, remodels them. Our solution to this task relies on a conditional multi-scale 3D GAN architecture. It takes a patient's MRI scan as input and generates a 3D volume conditioned on the patient's brain, which is preserved exactly, but where the face has been de-identified through remodeling. We demonstrate that our approach preserves privacy far better than existing techniques, without compromising downstream medical analyses on the brain.