Search for clj-concord doesn't find clj-concordion

[holyjak]

I believe that both https://clojars.org/search?q=clj-concord&format=json and https://clojars.org/search?q=clj-concordion&format=json should include clj-concordion yet they do not, though the former includes lot of projects that have "clj-" but no "concord" in their name.

I see that clj-concord has 2758 "hits" so perhaps clj-concordion is just not on the first page of 24. But the ranking/matching algorithm seems really weird ...

[tobias]

I don't know much about lucene, but if someone who does know know more about it and wants to look at solving this, the search code lives in https://github.com/clojars/clojars-web/blob/master/src/clojars/search.clj. Also see #721.

[holyjak]

thanks! I want to fix search for cljdocs so I can look at this one just as well, when I only find the time...

…

On Sun, May 19, 2019, 7:30 PM Toby Crawley ***@***.***> wrote: I don't know much about lucene, but if someone who does know know more about it and wants to look at solving this, the search code lives in https://github.com/clojars/clojars-web/blob/master/src/clojars/search.clj. Also see #721 <#721>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#719?email_source=notifications&email_token=AAEYSPVUYVGE3HTYIGJUSIDPWGFB3A5CNFSM4HKWSFUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVXGXGY#issuecomment-493775771>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEYSPUOWEYRY64FNCI4M2DPWGFB3ANCNFSM4HKWSFUA> .

[holyjak]

Hi @tobias ! If you get to work on search, let me know, we can perhaps leverage the code both for clojars and cljdoc. I have re-written search for cljdoc but have to yet incorporate download counts to provide better results. It is better then what it used to be but still not optimal, as can be seen in this thread cljdoc/cljdoc#308

Here is cljdoc in-progress PR to incorporate download counts cljdoc/cljdoc#359 but I am stuck between it having too little effect and having undesirably strong effect. Tuning Lucene is not trivial for me :) I will look more at how clojars does it. Thanks!

[tobias]

Hi @holyjak! I finally got around to working on search, and used your cljdoc implementation as the basis for that. It was really helpful, thanks! (see #806 (comment) for a few more details).

For download counts, the method Clojars was using is no longer supported by Lucene. So I switched to including the calculated boost value in the index, then boosting the query by that value.

The boost is calculated here:

clojars-web/src/clojars/search.clj

Lines 206 to 213 in d98939c

	(def download-score-weight 50)
	(defn- calculate-document-boost
	[stats {:as _jar :keys [artifact-id group-id]}]
	(let [total (stats/total-downloads stats)]
	(* download-score-weight
	(/ (or (stats/download-count stats group-id artifact-id) 0)
	(max 1 total)))))

added to the index here:

clojars-web/src/clojars/search.clj

Lines 195 to 197 in d98939c

	;; adds a boost field based on the ratio of downloads of the jar to the
	;; total number of downloads. This is then applied to the query below.
	(.add (DoubleDocValuesField. boost-field-name download-boost))))

and used in the query here:

clojars-web/src/clojars/search.clj

Line 300 in d98939c

query (FunctionScoreQuery/boostByValue query (DoubleValuesSource/fromDoubleField boost-field-name))

Hopefully that helps with boosting cljdocs results with dowload counts.

As far as the root issue reported here: even with my changes, searching for clj-concord won't find clj-concordion, since I didn't change the searching to be fuzzy, so you would have to use clj-concord* or concord*. However, results for those searches should be better now.

[holyjak]

Thank you very much, that is wonderful!

[tobias]

I don't think we have a good way to fix the root issue here (searching for clj-concord does not find clj-concordian) without making all searches fuzzy, and doing that would make search less useful overall unfortunately. So I'm going to close this issue.