User conditional hashtag prediction for images

Emily Denton, Jason Weston, Manohar Paluri, Lubomir Bourdev, Rob Fergus

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Understanding the content of user's image posts is a particularly interesting problem in social networks and web settings. Current machine learning techniques focus mostly on curated training sets of image-label pairs, and perform image classification given the pixels within the image. In this work we instead leverage the wealth of information available from users: firstly, we employ user hashtags to capture the description of image content; and secondly, we make use of valuable contextual information about the user. We show how user metadata (age, gender, etc.) combined with image features derived from a convolutional neural network can be used to perform hashtag prediction. We explore two ways of combining these heterogeneous features into a learning framework: (i) simple concatenation; and (ii) a 3-way multiplicative gating, where the image model is conditioned on the user metadata. We apply these models to a large dataset of de-identified Facebook posts and demonstrate that modeling the user can significantly improve the tag prediction quality over current state-of-the-art methods.

Original languageEnglish (US)
Title of host publicationKDD 2015 - Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining
PublisherAssociation for Computing Machinery
Number of pages10
ISBN (Electronic)9781450336642
StatePublished - Aug 10 2015
Event21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2015 - Sydney, Australia
Duration: Aug 10 2015Aug 13 2015

Publication series

NameProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining


Other21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2015


  • Deep learning
  • Hashtagging
  • Large scale image annotation
  • Social media
  • User modeling

ASJC Scopus subject areas

  • Software
  • Information Systems


Dive into the research topics of 'User conditional hashtag prediction for images'. Together they form a unique fingerprint.

Cite this