Generative pre-trained transformer based on image content and user personality for caption creation in social media

Social media is a platform for sharing information and interactions between users, often involving images with captions that reflect the user's personality. Each user creates distinct captions based on personal traits, making a personalized caption generator beneficial. Currently, existing social media caption generators have limitations, such as requiring payment for full features, lack of support for Bahasa (Indonesian Language), dependency on user input to generate captions, and suboptimal object detection accuracy. To address these issues, a new method is proposed for generating social media captions based on image content and user personality to simplify the caption creation process. This caption generator will be optimized in Bahasa. The content of the image will be explored through image objects and scenery. Image objects are identified using a Graph Convolutional Network (GCN) for personality classification. At the same time, a Convolutional Neural Network (CNN) approach will be employed to detect objects within images, and VGG16 will be used to detect scenery. Then, these three models are combined with a GPT to generate new captions. The model will be trained on public datasets, and subjective evaluation will be used for testing. The outcome of this research is expected to produce relevant captions based on the user's personality, making the captioning process more efficient and relevant to the personality.