Depth Recommendation Systems and Feature Combination Selection: Unleashing the Power of TayFCS

In today’s digital landscape, where information is vast and attention spans are short, depth recommendation systems (DRS) have become pivotal in delivering personalized user experiences. From streaming platforms curating your next watchlist to e-commerce sites suggesting products that align with your preferences, these systems are the backbone of personalized content delivery. But have you ever wondered what makes these recommendations so spot-on? The answer lies in how these systems model and understand the complex interactions between users and items. Today, we’re diving deep into a crucial aspect of DRS – feature combination selection, and exploring how the innovative TayFCS framework is revolutionizing this space.

The Nitty-Gritty of Depth Recommendation Systems and Feature Interactions

At the heart of DRS is the ability to model intricate user-item interaction behaviors. Researchers have developed a suite of methods, such as DeepFM, DCN, and PNN, to capture implicit feature interactions guided by supervision signals. However, the limitations of model capacity and data volume often hinder these methods from grasping fine-grained and high-order feature interactions based solely on individual input features. To circumvent this, the industry commonly employs explicit feature combinations using Cartesian products. By incorporating these combinations as additional features, the complexity of modeling feature co-occurrence patterns is reduced.

But here’s the catch: as the order of feature combinations increases, so does the number of combinations, leading to an explosion in computational complexity and introducing heavy information redundancy. This makes selecting a subset of effective feature combinations a critical challenge for practical recommender systems. Traditional feature selection methods, which typically focus on individual features, fall short when it comes to high-order feature combination selection due to the exponential growth in time complexity when evaluating feature combinations one by one.

Enter TayFCS: A Game-Changing Feature Combination Selection Framework

To address these challenges, researchers have proposed TayFCS (Taylor-based Feature Combination Selection), a lightweight yet powerful framework designed to enhance model performance while reducing noise and managing memory consumption. Let’s break down how TayFCS works its magic.

The Taylor Expansion Scorer (TayScorer): Where the Magic Begins

The Taylor Expansion Scorer (TayScorer) is the cornerstone of TayFCS. It leverages field-wise Taylor expansion on the base model to approximate the importance of feature combinations. Instead of repeatedly running experiments with feature addition and removal, TayScorer computes importance based on sub-component gradients. This can be achieved with a single backward pass using a trained recommendation model. The beauty of this approach lies in its efficiency – it avoids the exponential time complexity that comes with evaluating all potential feature combinations individually.

Logistic Regression Elimination (LRE): Tackling Redundancy Head-On

To further reduce information redundancy among feature combinations and their sub-components, TayFCS introduces Logistic Regression Elimination (LRE). This module estimates information gain based on model prediction performance. By sorting features in descending order of importance and adding them to a surrogate model in batches according to window size, LRE iteratively removes features that don’t provide positive gains. This ensures that the final set of feature combinations is both informative and non-redundant.

Under the Hood: How TayFCS Operates

TayFCS operates in two main phases: efficient analysis of feature importance and elimination of redundant features. Let’s delve deeper into each phase.

Efficient Feature Importance Analysis with TayScorer

TayScorer begins by formalizing the relationship between deep models and features. It performs a Taylor expansion on a well-trained model to obtain the importance of corresponding combined features. However, traditional Taylor expansions have extremely high time complexity. To overcome this, TayFCS employs an efficient approximation method inspired by the Information Matrix Equality (IME). This allows for a rapid estimation of combined feature importance with time complexity reduced from O(n²) to O(n).

Redundancy Elimination with LRE

LRE addresses the issue of redundancy by using a greedy algorithm to filter out high-scoring but redundant features. It employs a logistic regression model to assess whether additional features contribute useful information. By shuffling features one by one and evaluating the impact on model performance, LRE identifies and removes redundant features, ensuring that only those that bring positive gains remain.

Experimental Validation: Putting TayFCS to the Test

The effectiveness and efficiency of TayFCS have been validated through extensive experiments on three benchmark datasets: Frappe, iPinYou, and Avazu. These datasets, derived from real-world scenarios, provide a robust testing ground for evaluating the performance of TayFCS.

Key Findings from the Experiments

The experimental results reveal several key insights:

1. Dataset Variability: The impact of feature combinations varies across datasets. For instance, on the Frappe and Avazu datasets, feature combinations yield significant performance improvements, while on the iPinYou dataset, the improvement is smaller. This can be attributed to the sparse positive samples in the iPinYou dataset, which limit the amount of useful gradient information.

2. Model-Specific Improvements: Different models exhibit varying improvements within the same dataset. For example, DeepFM performs best on the Frappe dataset, likely due to its smaller number of fields, which allows FM to capture higher-order interaction information through cross-product on selected feature combinations. On the iPinYou dataset, DNN achieves the best performance, possibly because other interaction modules struggle to learn due to the sparsity of positive samples.

3. Superiority of TayFCS: Among the selection methods, TayFCS stands out. While randomly selecting feature combinations shows decent results in some cases, it often provides the lowest improvement and sometimes even causes a significant performance drop. AutoField+ and TayFCS consistently achieve top-2 performance, indicating that feature combinations constructed at the input layer help the model learn complex user-item interaction relationships that cannot be achieved through model architecture changes alone.

Visual Importance Results: Seeing is Believing

To provide an intuitive understanding of the feature combination scores, researchers visualized the importance scores using heatmaps. For the Frappe dataset, it’s evident that feature combinations involving user and item are the most important. Additionally, combinations of country and daytime play a crucial role in capturing variations related to the environment and timing of advertisement display. For the iPinYou dataset, important features primarily involve combinations related to slotid and useragent, followed by features related to slotvisibility or slotformat. These visualizations demonstrate that TayFCS effectively highlights important features while distinguishing them from less important ones.

Transfer Learning: The Proof of Concept

A critical aspect of any feature selection method is its ability to generalize to other models. To assess this, researchers used DCN and MaskNet as target models and DNN as the source model. The results showed that incorporating feature combinations selected by TayFCS into DCN and MaskNet further improved performance. For DCN, the AUC improved from 0.7942 to 0.8024, while on the well-designed MaskNet model (initially at 0.7964), the AUC reached 0.8046 after adding feature combinations on the Avazu dataset. This not only demonstrates the effective transferability of the combination results produced by TayFCS but also indicates that relying solely on model optimization is insufficient for adequately modeling user-item relationships.

Hyperparameter Analysis: Striking the Right Balance

The relationship between the number of added feature combinations and model performance was also examined. For the Avazu dataset on DeepFM and Wide & Deep models, TayFCS showed significant improvement over the baseline model when K=5 and reached its best performance at K=15. However, at K=20, model performance slightly declined. This indicates that the informative combinations subset is a small fraction of the features. As more features are added, excessive information can make it difficult for the model to distinguish which features are more important for measuring user behavior, leading to some performance degradation. The performance at K=10 and K=15 is very similar, and considering the balance between required resources and performance, K=10 is generally preferred. Moreover, TayFCS consistently outperforms AutoField+ across different K values, while randomly selecting the combination subset fails to provide significant performance improvement, further highlighting the effectiveness of TayFCS.

Efficiency Analysis: Speed Meets Precision

The efficiency of analyzing each feature combination is crucial, as excessive time spent on exploration can be resource-intensive and costly. In this subsection, the total time required to analyze feature combinations was compared between TayFCS and previous feature engineering works. On the Frappe dataset, due to its relatively small size, the time consumption of TayFCS is comparable to that of AutoField+. However, on the iPinYou and Avazu datasets, which have more feature fields, TayFCS significantly outperforms AutoField+ in terms of time efficiency. This is because TayFCS employs TayScorer, which approximates higher-order Taylor expansions efficiently. TayScorer estimates higher-order importance scores with a single gradient pass, eliminating the need for multiple training or backpropagation steps. The LRE process only requires training a lightweight logistic regression model once. As a result, its overall time complexity remains similar to that of original training.

Ablation Studies: Understanding the Components

Ablation studies were conducted to explore the impact of each component of TayFCS. The experiments revealed that each component is crucial. Random feature selection performs the worst, as it is blind and lacks depth. The hash table is essential for models on the Avazu dataset when combinations involve many features. Without it, performance drops sharply, nearly matching that of random combinations. However, for Frappe, it is less important because the most important combinations’ features do not exceed the threshold τ. Moreover, removing LRE results in a performance drop on both Frappe and Avazu, indicating that LRE effectively removes redundant features. This highlights the effectiveness and adaptability of TayFCS across different datasets.

Inference Time: The Real-World Consideration

Inference time is equally important for online services. Evaluations on the test dataset revealed that adding feature combinations based on the hash embedding table in TayFCS does not significantly increase inference overhead in most cases. For example, in Frappe on DNN, inference time increased by 10%, and on iPinYou and Avazu, the increases were 12.5% and 13.7%, respectively. A similar trend was observed on Wide & Deep. The increase in time cost mainly comes from two factors: first, the feature combination embedding takes more time for table lookup; second, more combinations require wider hidden states in the model. Though the added feature combinations account for 50%, 31.2%, and 41.7% of the original feature fields, the increase in inference time is small. This suggests that inference latency does not grow linearly with the number of added features, since the model’s computation (excluding embeddings) also takes a large portion of the time. It is also possible that the hash operations used in the embedding table make the lookup overhead relatively low. Despite the slight increase in time, the improvement in inference performance justifies this additional overhead, as there is no free lunch.

Online Test: Where the Rubber Meets the Road

TayFCS was deployed in the CVR (Conversion Rate) prediction task on an ad platform. The CVR model is updated daily with billions of samples and hundreds of finely crafted features. To prevent training dataset overfitting, 10% of the validation set (about 50 thousand samples) was used for TayFCS for feature combination selection. TayFCS took approximately 1 hour to screen all possible sparse feature combinations and selected the top 12 features. In the A/B tests, the control group used the TAML model with the original features, while the experimental group added 12 combination features from TayFCS (8 2nd-order and 4 3rd-order combinations). Both groups had 5% traffic from randomly divided users. After two weeks of observation, the TayFCS group saw a 13.9% CVR improvement and 0.73% revenue growth, now serving the whole traffic. TayFCS has become a foundational tool in the platform for feature combination selection, resulting in improved user engagement.

Conclusion: The Future of Feature Combination Selection

TayFCS addresses the inefficiency and redundancy in current combined feature selection by integrating the IME theory to derive an efficient approximation of Taylor expansion and introducing LRE to reduce feature redundancy. Based on the valid feature combination scores obtained from this process, TayFCS selects the top important ones and constructs them as new regular features in the embedding layer to help the model better capture interaction patterns. Extensive experiments on classical base models across three datasets validate the effectiveness of this method. The results clearly show that TayFCS improves prediction accuracy while demonstrating high efficiency in the analysis process and strong potential for real-world deployment. This may provide a practical and novel perspective for optimizing depth recommendation systems.

However, TayFCS is not without its limitations. While it efficiently selects useful feature combinations and these combinations are model-agnostic, bringing improvements in prediction accuracy with acceptable inference latency, the additional combinations require extra embedding space, leading to increased memory cost. In future work, the aim is to explore combinations at the feature value level to further enhance accuracy while minimizing memory overhead.

In the ever-evolving landscape of depth recommendation systems, TayFCS stands as a beacon of innovation, offering a promising path forward for feature combination selection. As we continue to push the boundaries of what these systems can achieve, frameworks like TayFCS will undoubtedly play a crucial role in shaping the future of personalized content delivery.