Deprecation banner on module which doesn't appear to be deprecated

[bigpreshkt]

I see that #2436 has started showing deprecation banners.

I see one for, e.g. https://metacpan.org/pod/CHI

However, I don't see anything to suggest that module is deprecated, and if I ask the API:

[davidp@columbia:~]$ jq .deprecated <(curl -s https://fastapi.metacpan.org/v1/release/CHI)
"false"

Similarly, no sign that it should be deprecated at https://explorer.metacpan.org/?url=/module/JSWARTZ/CHI-0.60/lib%2FCHI.pm

Am I missing something obvious, or is this displaying the banner wrongly?

[bleargh45]

That does appear to be incorrect; let me go take a look at that for you right now.

[bleargh45]

Now seeing what the discrepancy was that caused this to occur...

When using the local metacpan-docker setup to do development with, the deprecated field is properly cast as a Boolean, and returns false. The Production API, however, returns it as a string value, "false":

$ curl --silent http://localhost:5000/module/CHI        | jq .deprecated
false

$ curl --silent http://fastapi.metacpan.org/module/CHI  | jq .deprecated
"false"

[bleargh45]

And, more frustratingly, it also appears that the boolean-ness of the values are not consistent across the API Index:

$ curl --silent http://fastapi.metacpan.org/release/Alien-Prototype-Carousel  | jq .deprecated
true

$ curl --silent http://fastapi.metacpan.org/release/CHI  | jq .deprecated
"false"

$ curl --silent http://fastapi.metacpan.org/module/Dist::Zilla::Plugin::EOLTests | jq .deprecated
true

$ curl --silent http://fastapi.metacpan.org/module/Dist::Zilla::Plugin::Test::EOL | jq .deprecated
"false"

So... for the moment, simplest approach looks like checking not just for the 1/0-ness of the Bool, but also for "true/false" (as strings). At least, until we can determine what's up with the underlying index, and can figure out how to address that.

[oalders]

@mickeyn any idea what's going on with the booleans?

[bleargh45]

If I had to wager a guess... I think what may have happened is that when the deprecated field was added to the index and backfilled, the backfill accidentally got done with a string value; when I've checked various releases/distributions, it feels like "more recent Releases got a Boolean, older ones got a String".

$ curl --silent http://fastapi.metacpan.org/release/GTERMARS/Data-FormValidator-EmailValid-0.05 | jq .release.deprecated
"false"

$ curl --silent http://fastapi.metacpan.org/release/GTERMARS/Data-FormValidator-EmailValid-0.06 | jq .release.deprecated
false

(do note, though, that there are several years between those two releases)

[oalders]

Ah! I just re-indexed 0.05 and it looks to no longer return the string.

[oalders]

It might make sense just to re-index everything again. @mickeyn what do you think?

[bleargh45]

Reading through the old ElasticSearch docs, I see reference to an "Update by Query API", which sounds like it could be useful (to someone who could craft a better ES search than I). Sounds like it can take records that match a given query, and then reindex them to pick up new fields. Example provided in the docs is a new field was added to the index, and the existing docs need to be reindexed to pick it up.

Unsure if that's useful here, but figured I'd throw it out there just in case.

[oalders]

I think that API is probably helpful for indexing plain documents, but a lot of what we're doing is parsing the documents ourselves and setting fields based on that, so it won't cover all of our needs.

[mickeyn]

@oalders mass reindexing older versions is always tricky because it's done against current CPAN index and permissions files it will probably cause some authorization issues as we've seen before.

The boolean issue is probably a Perl problem (not that ES issue in this old version is out of the question).
We have boolean fields set in the mapping and we rely on Bool type of the Moose attributes where we only set 1 / 0 - something to dig deeper into (maybe we should be using a different type? set different values?).

[bleargh45]

If I've found the right chunk of code, then I'm left to question if we're even using the mapping... down in metacpan/metacpan-api/lib/MetaCPAN/Document/Release/Set.pm#L55 it calls $self->filter(...)->sort(...)->raw->first.

That ->raw part is what caught my eye; is that not just returning the raw hash-ref of data from ES, as opposed to running it through the mapping and/or object transformation first?

Unfortunately, I'm somewhat hindered here as the metacpan-docker setup doesn't generate this same "string false", thus making it harder for me to be of assistance in helping sort out where the proper fix for this lies.

[mohawk2]

I know this isn't going to be what's fun to hear (or action), but I fear that figuring out how to repro the string-false is essential to correctly addressing this.

[oalders]

My first thought was that it's an issue with our boolean handling in Perl. Having said that, if we can't reproduce the problem with the Docker setup and if the following is always correct after having been re-indexed

$ curl --silent http://fastapi.metacpan.org/release/GTERMARS/Data-FormValidator-EmailValid-0.05 | jq .release.deprecated

then I'm inclined to think maybe the bug was in our indexing process at some point or just the initial backfill, as @bleargh45 has suggested.

[oalders]

I guess we could just check that field for everything in the index and see if there's a pattern. :)