Abstract:
Most recent generation methods synthesize images from either complex textual descriptions or scene graphs. However, users need to elaborate attributes and relationships of objects in the scene, and scene graphs are more difficult to obtain. To simplify the burden of users, in this work, we propose a Label2im model to generate images from object labels directly with the help of a Knowledge Graph (KG), e.g. Visual Genome. To acquire rational interactions between objects, we explore possible relationships from the KG. Considering that there is a large gap between the label domain and image domain, we propose to learn knowledge representations of the scene graph from the KG to ensure the semantic consistency. First, given several object labels, we design a Scene Graph Selection Module (SGSM) to explore interactions between objects in the KG and generate a set of scene graphs. Second, the structure representation and knowledge embedding of the scene graph are learned and integrated in the Scene Graph Representation Module (SGRM), which leads to rational scene layouts. Based on the scene layouts and KG, we employ the Cascaded Refinement Network (CRN) to generate the final image. To encode knowledge information in the generation process, we propose a Triplet Attention Module (TAM) which is embedded in the CRN. We verify the effectiveness of the proposed method on the Visual Genome dataset and demonstrate that our method is able to generate complex images with rich content and fine details.
