PALM: Personalized Attention-based Language Model for Long-tail Query Understanding in Enterprise Search Systems

Enterprise search systems face significant challenges in handling long-tail queries, which constitute a substantial portion of search traffic but often receive inadequate attention in traditional systems. This paper introduces PALM (Personalized Attention-based Language Model), a novel framework designed to enhance long-tail query understanding in enterprise search environments. PALM integrates personalization capabilities with an advanced attention mechanism to improve search accuracy for infrequent queries while maintaining high performance on common queries. The framework employs a unique hierarchical architecture that combines user context, query semantics, and organizational knowledge through a sophisticated attention mechanism. The system features an innovative query embedding approach that adapts to individual user contexts while leveraging collective organizational knowledge. Extensive experiments on a large-scale enterprise dataset, comprising over 5 million queries from 50,000 users, demonstrate PALM's superior performance compared to state-of-the-art baselines. The results show significant improvements across multiple metrics, with a 17.5% increase in MAP for ultra-rare queries and a 10.4% overall improvement in NDCG@10. The framework exhibits robust performance across different organizational units and query types, making it particularly valuable for enterprise environments where query patterns are highly diverse and context-dependent. Our ablation studies confirm the effectiveness of each component in the PALM architecture, while case analyses provide insights into the framework's practical applications.