Multivariate Performance Measures
Bipartite ranking is a two-label ranking problem involving relevant and non-relevant elements.
Two Groups Instead of Every Pair
Some ranking problems do not need to distinguish every element from every other element. The important question may be simpler: does each element belong to the relevant group or the non-relevant group? Bipartite ranking describes this two-label setting.
Bipartite ranking is a two-label ranking problem involving relevant and non-relevant elements.
Reading the Feedback Vector
The feedback vector y records the two groups with numerical labels. An entry of 1 identifies a relevant element, while an entry of -1 identifies a non-relevant element. Each position in y therefore carries the feedback label for the corresponding element.
Interpreting a feedback vector
Consider the feedback vector y = [1, -1, 1]. What does each entry communicate?
First position: The value 1 identifies the first element as relevant.
Second position: The value -1 identifies the second element as non-relevant.
Third position: The value 1 identifies the third element as relevant.
The vector describes two relevant elements and one non-relevant element.
From Scores to Binary Labels
A predicted ranking vector y' contains predicted values for the elements. Each predicted value y'i is compared with a threshold θ. The binary prediction is obtained through sign(y'i - θ). Values above the threshold produce the positive sign, corresponding to the relevant label 1; values below the threshold produce the negative sign, corresponding to the non-relevant label -1.
Converting predicted values
Use θ = 0 with the predicted ranking vector y' = [0.8, -0.4, 0.2]. Convert each value into a binary label using sign(y'i - θ).
First value: The value 0.8 is above θ = 0, so its sign is positive and its binary label is 1.
Second value: The value -0.4 is below θ = 0, so its sign is negative and its binary label is -1.
Third value: The value 0.2 is above θ = 0, so its sign is positive and its binary label is 1.
The converted binary vector is [1, -1, 1].
Selecting the Threshold
The threshold θ is usually set to 0. With that choice, the conversion asks whether each predicted value is on the positive or negative side of zero. However, zero is not mandatory in every problem. Additional problem constraints may motivate choosing another threshold.
Mistakes in Label Conversion
Treating bipartite ranking as a problem that must order every element distinctly.
Bipartite ranking focuses on separating elements into the two groups relevant and non-relevant.
Fix:
First identify the two-label structure: relevant elements receive 1 and non-relevant elements receive -1.Reversing the meanings of 1 and -1 in the feedback vector.
The feedback-vector convention assigns 1 to relevant elements and -1 to non-relevant elements.
Fix:
Use 1 for relevant and -1 for non-relevant.Applying the threshold without subtracting θ.
The stated conversion is sign(y'i - θ), so the predicted value must be compared with the selected threshold.
Fix:
Subtract θ from each predicted value before interpreting its sign.Assuming the threshold must always be zero.
Zero is usual, but the threshold may be chosen using additional constraints.
Fix:
Start with θ = 0 when appropriate, then check whether the problem requires a different value.
Check Your Understanding
A predicted ranking vector contains the values 0.6 and -0.2. Using θ = 0, determine the binary label produced for each value. Then explain what would need to be reconsidered if an additional problem constraint required a different threshold.
Hints
- Compare each predicted value with θ.
- A value above the threshold receives the positive label 1, while a value below the threshold receives the negative label -1.
- A different threshold changes the comparison used in sign(y'i - θ).
- Bipartite ranking handles a two-group ranking problem: relevant versus non-relevant elements. The feedback vector y uses 1 for relevant elements and -1 for non-relevant elements. A predicted ranking value y'i becomes a binary label through sign(y'i - θ). The threshold θ is usually 0, but additional problem constraints can motivate another choice.
Key Takeaways
- Bipartite ranking separates elements into relevant and non-relevant groups.
- The feedback vector uses 1 for relevant elements and -1 for non-relevant elements.
- Each predicted ranking value is converted with sign(y'i - θ).
- The threshold is usually 0, but additional problem constraints may require another value.