Skip to content

Commit 9a30e23

Browse files
shahidki31srowen
authored andcommitted
[SPARK-26351][MLLIB] Update doc and minor correction in the mllib evaluation metrics
## What changes were proposed in this pull request? Currently, there are some minor inconsistencies in doc compared to the code. In this PR, I am correcting those inconsistencies. 1) Links related to the evaluation metrics in the docs are not working 2) Minor correction in the evaluation metrics formulas in docs. ## How was this patch tested? NA Closes #23589 from shahidki31/docCorrection. Authored-by: Shahid <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent 6c18d8d commit 9a30e23

File tree

2 files changed

+13
-11
lines changed

2 files changed

+13
-11
lines changed

docs/mllib-evaluation-metrics.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -413,13 +413,13 @@ A ranking system usually deals with a set of $M$ users
413413

414414
$$U = \left\{u_0, u_1, ..., u_{M-1}\right\}$$
415415

416-
Each user ($u_i$) having a set of $N$ ground truth relevant documents
416+
Each user ($u_i$) having a set of $N_i$ ground truth relevant documents
417417

418-
$$D_i = \left\{d_0, d_1, ..., d_{N-1}\right\}$$
418+
$$D_i = \left\{d_0, d_1, ..., d_{N_i-1}\right\}$$
419419

420-
And a list of $Q$ recommended documents, in order of decreasing relevance
420+
And a list of $Q_i$ recommended documents, in order of decreasing relevance
421421

422-
$$R_i = \left[r_0, r_1, ..., r_{Q-1}\right]$$
422+
$$R_i = \left[r_0, r_1, ..., r_{Q_i-1}\right]$$
423423

424424
The goal of the ranking system is to produce the most relevant set of documents for each user. The relevance of the
425425
sets and the effectiveness of the algorithms can be measured using the metrics listed below.
@@ -439,21 +439,21 @@ $$rel_D(r) = \begin{cases}1 & \text{if $r \in D$}, \\ 0 & \text{otherwise}.\end{
439439
Precision at k
440440
</td>
441441
<td>
442-
$p(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{k} \sum_{j=0}^{\text{min}(\left|D\right|, k) - 1} rel_{D_i}(R_i(j))}$
442+
$p(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{k} \sum_{j=0}^{\text{min}(Q_i, k) - 1} rel_{D_i}(R_i(j))}$
443443
</td>
444444
<td>
445-
<a href="https://en.wikipedia.org/wiki/Information_retrieval#Precision_at_K">Precision at k</a> is a measure of
445+
<a href="https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Precision_at_K">Precision at k</a> is a measure of
446446
how many of the first k recommended documents are in the set of true relevant documents averaged across all
447447
users. In this metric, the order of the recommendations is not taken into account.
448448
</td>
449449
</tr>
450450
<tr>
451451
<td>Mean Average Precision</td>
452452
<td>
453-
$MAP=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{\left|D_i\right|} \sum_{j=0}^{Q-1} \frac{rel_{D_i}(R_i(j))}{j + 1}}$
453+
$MAP=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{N_i} \sum_{j=0}^{Q_i-1} \frac{rel_{D_i}(R_i(j))}{j + 1}}$
454454
</td>
455455
<td>
456-
<a href="https://en.wikipedia.org/wiki/Information_retrieval#Mean_average_precision">MAP</a> is a measure of how
456+
<a href="https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision">MAP</a> is a measure of how
457457
many of the recommended documents are in the set of true relevant documents, where the
458458
order of the recommendations is taken into account (i.e. penalty for highly relevant documents is higher).
459459
</td>
@@ -462,10 +462,10 @@ $$rel_D(r) = \begin{cases}1 & \text{if $r \in D$}, \\ 0 & \text{otherwise}.\end{
462462
<td>Normalized Discounted Cumulative Gain</td>
463463
<td>
464464
$NDCG(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{IDCG(D_i, k)}\sum_{j=0}^{n-1}
465-
\frac{rel_{D_i}(R_i(j))}{\text{ln}(j+2)}} \\
465+
\frac{rel_{D_i}(R_i(j))}{\text{log}(j+2)}} \\
466466
\text{Where} \\
467-
\hspace{5 mm} n = \text{min}\left(\text{max}\left(|R_i|,|D_i|\right),k\right) \\
468-
\hspace{5 mm} IDCG(D, k) = \sum_{j=0}^{\text{min}(\left|D\right|, k) - 1} \frac{1}{\text{ln}(j+2)}$
467+
\hspace{5 mm} n = \text{min}\left(\text{max}\left(Q_i, N_i\right),k\right) \\
468+
\hspace{5 mm} IDCG(D, k) = \sum_{j=0}^{\text{min}(\left|D\right|, k) - 1} \frac{1}{\text{log}(j+2)}$
469469
</td>
470470
<td>
471471
<a href="https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG">NDCG at k</a> is a

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,8 @@ class RankingMetrics[T: ClassTag](predictionAndLabels: RDD[(Array[T], Array[T])]
138138
var dcg = 0.0
139139
var i = 0
140140
while (i < n) {
141+
// Base of the log doesn't matter for calculating NDCG,
142+
// if the relevance value is binary.
141143
val gain = 1.0 / math.log(i + 2)
142144
if (i < pred.length && labSet.contains(pred(i))) {
143145
dcg += gain

0 commit comments

Comments
 (0)