From 65eb83be9d700745b560051857bb6470792e4c85 Mon Sep 17 00:00:00 2001 From: Graham Knop Date: Tue, 28 Jun 2016 14:01:42 -0400 Subject: [PATCH 0001/1850] update links for new org and repo names --- News.md | 8 ++++---- README.md | 6 +++--- lib/MetaCPAN/Web/Model/API.pm | 4 ++-- root/about.html | 4 ++-- root/about/contact.html | 4 ++-- root/about/contributors.html | 16 ++++++++-------- root/about/development.html | 10 +++++----- root/about/faq.html | 4 ++-- root/about/missing_modules.html | 4 ++-- root/lab.html | 2 +- root/static/js/github.js | 2 +- root/wrapper.html | 6 +++--- t/model/changes.t | 2 +- 13 files changed, 36 insertions(+), 36 deletions(-) diff --git a/News.md b/News.md index 2f879221ce..4a36ff12e6 100644 --- a/News.md +++ b/News.md @@ -11,7 +11,7 @@ tight configs... and now so do we. I'd like to thank then for providing both examples, support and an IRC channel where Donald Stufft from the Python infrastructure team was able to help out. -All our configs and systems are in our [github repos](https://github.com/CPAN-API/) +All our configs and systems are in our [github repos](https://github.com/metacpan/) so feel free to review / improve or 'borrow' configs from. Title: Liquid Web is Sponsoring MetaCPAN @@ -38,8 +38,8 @@ We have moved all sites over to [Fastly](http://www.fastly.com) We have further plans... but this will do for now, many thanks to our sponsors for making this possible. -[network-infrastructure](https://github.com/CPAN-API/network-infrastructure) -and [metacpan-puppet](https://github.com/CPAN-API/metacpan-puppet) have all +[network-infrastructure](https://github.com/metacpan/network-infrastructure) +and [metacpan-puppet](https://github.com/metacpan/metacpan-puppet) have all our configs if you want to see details of anything. @@ -65,7 +65,7 @@ If you had either http://search.cpan.org/ or http://cpansearch.perl.org/ in your desktop `hosts` file you'll need to [update the IP](https://metacpan.org/about/faq#cani_automatically_redirectlinkspointingatsearch.cpan.orgtometacpan.org) -We will be documenting this a bit more under [Network Infrastructure](https://github.com/CPAN-API/network-infrastructure) +We will be documenting this a bit more under [Network Infrastructure](https://github.com/metacpan/network-infrastructure) but that's a work in progress. diff --git a/README.md b/README.md index 4e85a49b38..b2c57121e5 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ -[![Build Status](https://travis-ci.org/CPAN-API/metacpan-web.png?branch=master)](https://travis-ci.org/CPAN-API/metacpan-web) -[![Coverage Status](https://coveralls.io/repos/CPAN-API/metacpan-web/badge.png)](https://coveralls.io/r/CPAN-API/metacpan-web) +[![Build Status](https://travis-ci.org/metacpan/metacpan-web.png?branch=master)](https://travis-ci.org/metacpan/metacpan-web) +[![Coverage Status](https://coveralls.io/repos/metacpan/metacpan-web/badge.png)](https://coveralls.io/r/metacpan/metacpan-web) ## GETTING STARTED -We strongly recommend using [metacpan-developer](https://github.com/CPAN-API/metacpan-developer), +We strongly recommend using [metacpan-developer](https://github.com/metacpan/metacpan-developer), this will give you a virtual machine already configured and ready to start developing on. $ vagrant ssh diff --git a/lib/MetaCPAN/Web/Model/API.pm b/lib/MetaCPAN/Web/Model/API.pm index dcff97322d..b6f369969c 100644 --- a/lib/MetaCPAN/Web/Model/API.pm +++ b/lib/MetaCPAN/Web/Model/API.pm @@ -33,7 +33,7 @@ has api_secure => ( $AnyEvent::HTTP::PERSISTENT_TIMEOUT = 0; $AnyEvent::HTTP::USERAGENT = 'Mozilla/5.0 (compatible; U; MetaCPAN-Web/1.0; ' - . '+https://github.com/CPAN-API/metacpan-web)'; + . '+https://github.com/metacpan/metacpan-web)'; } sub cv { @@ -134,7 +134,7 @@ sub _strip_source_prefix_from_fields { # "A stored boolean field is being returned as a string, not as a boolean" # when requested via "fields". To work around this we can specify # "_source.blah" in "fields", then we strip the "_source." prefix here. - # https://github.com/CPAN-API/metacpan-web/issues/881 + # https://github.com/metacpan/metacpan-web/issues/881 # https://github.com/elasticsearch/elasticsearch/issues/2551 # See .../API/Release.pm for examples of this. diff --git a/root/about.html b/root/about.html index 1c9b25e9e1..e95c2900e6 100644 --- a/root/about.html +++ b/root/about.html @@ -26,8 +26,8 @@ vs [https://fastapi.metacpan.org/v1/module/Moose](https://fastapi.metacpan.org/v1/module/Moose) MetaCPAN is [a community effort](/about/contributors), with all the code freely -available on GitHub ([www](https://github.com/CPAN-API/metacpan-web), -[api](https://github.com/CPAN-API/cpan-api)). +available on GitHub ([www](https://github.com/metacpan/metacpan-web), +[api](https://github.com/metacpan/metacpan-api)). ## Using the API diff --git a/root/about/contact.html b/root/about/contact.html index d54136ff17..5194213669 100644 --- a/root/about/contact.html +++ b/root/about/contact.html @@ -28,11 +28,11 @@ ## Front-end -This is what you are looking at right now. The source code is on [github](https://github.com/CPAN-API/metacpan-web). +This is what you are looking at right now. The source code is on [github](https://github.com/metacpan/metacpan-web). ## Back-end -The front-end is a slim wrapper around the back-end. It provides a REST API to data around CPAN and its ecosystem. The code is also on [github](https://github.com/CPAN-API/cpan-api). +The front-end is a slim wrapper around the back-end. It provides a REST API to data around CPAN and its ecosystem. The code is also on [github](https://github.com/metacpan/metacpan-api). diff --git a/root/about/contributors.html b/root/about/contributors.html index ca5c2a9f51..13ca2bfa47 100644 --- a/root/about/contributors.html +++ b/root/about/contributors.html @@ -2,14 +2,14 @@ - -

Networks stats for the last 30 days

- -<% UNLESS fastly_stats.keys %> - This must be run on a box that has access to - the fastly_api_key, production usually -<% END %> - -<% - # Define styles to use - css_category_lookup = { - requests => 'category10', - bandwidth => 'category20b', - hit_ratio => 'category20c', - }; - - # Now loop the sites we have stats on - FOREACH site IN fastly_stats.keys.sort; - - # Data for this site - site_data = fastly_stats.$site; - - # A key for the div/data - sitekey = site.replace(/\./,'_'); -%> -

<% site %>

- - <% FOREACH type IN [ 'requests', 'bandwidth' ] %> - -

<% type | ucfirst %>: <% site_data.totals.$type | comma %>

- -
- - - <% END %> - -<% END %> - - diff --git a/root/inc/about-bar.html b/root/inc/about-bar.html index 3a945068ca..df1736de80 100644 --- a/root/inc/about-bar.html +++ b/root/inc/about-bar.html @@ -28,8 +28,5 @@ class="active"<% END %>> Distribution Metadata - class="active"<% END %>> - Network Stats - diff --git a/t/html.t b/t/html.t index e2300f2ac8..e7a645a3c9 100644 --- a/t/html.t +++ b/t/html.t @@ -12,7 +12,7 @@ my %skip = map { $_ => 1 } ( 'root/inc/dependencies-graph.html', 'root/author.html', 'root/mirrors.html', 'root/source.html', 'root/wrapper.html', 'root/about/contributors.html', - 'root/inc/favorite.html', 'root/about/stats.html', + 'root/inc/favorite.html', ); my $rule = Path::Iterator::Rule->new; From 3e7997a4ff3af9f920b0e72487d61f4d4e1be917 Mon Sep 17 00:00:00 2001 From: Brad Lhotsky Date: Fri, 18 Nov 2016 17:57:45 -0600 Subject: [PATCH 0059/1850] Fix undefined deref in author email When an author is missing an email, don't die, seed as an empty list. --- lib/MetaCPAN/Web/Model/ReleaseInfo.pm | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/MetaCPAN/Web/Model/ReleaseInfo.pm b/lib/MetaCPAN/Web/Model/ReleaseInfo.pm index c169eb0bd9..d2fa4d4753 100644 --- a/lib/MetaCPAN/Web/Model/ReleaseInfo.pm +++ b/lib/MetaCPAN/Web/Model/ReleaseInfo.pm @@ -82,8 +82,10 @@ sub groom_contributors { $authors = [ grep { $_ ne 'unknown' } @$authors ]; my $author_info = { - email => - [ lc "$release->{author}\@cpan.org", @{ $author->{email} }, ], + email => [ + lc "$release->{author}\@cpan.org", + @{ $author->{email} || [] }, + ], name => $author->{name}, }; my %seen = map { $_ => $author_info } From a70eb2913feb466318ac47f64f97dba1558f60a7 Mon Sep 17 00:00:00 2001 From: Graham Knop Date: Fri, 18 Nov 2016 19:05:16 -0600 Subject: [PATCH 0060/1850] fix release-info test --- t/model/release-info.t | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/model/release-info.t b/t/model/release-info.t index 254f99ca1a..1745c98790 100644 --- a/t/model/release-info.t +++ b/t/model/release-info.t @@ -138,7 +138,7 @@ sub normalize_issues_ok { distribution => # Default to rt url, but let data override. - { bugs => { source => "${rt_prefix}X", %$bugs } }, + { bugs => { rt => { source => "${rt_prefix}X", %$bugs } } }, } ); is_deeply $instance->normalize_issues, $exp, $desc; From 4e59931cefc89dc1f5a7e1400748a5040bcdbadd Mon Sep 17 00:00:00 2001 From: Graham Knop Date: Fri, 18 Nov 2016 18:37:34 -0600 Subject: [PATCH 0061/1850] use Log::Log4perl for logging --- app.psgi | 6 ++- cpanfile | 3 +- cpanfile.snapshot | 78 ++++++++++++++++++++++++++--- lib/MetaCPAN/Web.pm | 17 ++++++- lib/MetaCPAN/Web/Controller/Root.pm | 2 + 5 files changed, 96 insertions(+), 10 deletions(-) diff --git a/app.psgi b/app.psgi index 9fda8a2c12..b5b785e420 100644 --- a/app.psgi +++ b/app.psgi @@ -14,7 +14,6 @@ my $dev_mode; BEGIN { $root_dir = File::Basename::dirname(__FILE__); $dev_mode = $ENV{PLACK_ENV} && $ENV{PLACK_ENV} eq 'development'; - $dev_mode and require Carp::Always; } BEGIN { @@ -30,6 +29,11 @@ use File::Path (); use MetaCPAN::Web; use Plack::Builder; +BEGIN { + $SIG{__WARN__} = sub { MetaCPAN::Web->log->warn(@_) }; + $dev_mode and require Devel::Confess and Devel::Confess->import; +} + my $tempdir = "$root_dir/var/tmp"; # explicitly call ->to_app on every Plack::App::* for performance diff --git a/cpanfile b/cpanfile index 961a03b047..7c1c25ff7c 100644 --- a/cpanfile +++ b/cpanfile @@ -1,4 +1,3 @@ -requires 'Carp::Always'; requires 'AnyEvent::Curl::Multi'; requires 'CHI'; requires 'CPAN::Changes', '0.21'; @@ -27,6 +26,7 @@ requires 'Data::Printer', '0.38'; requires 'DateTime', '1.24'; requires 'DateTime::Format::HTTP'; requires 'DateTime::Format::ISO8601'; +requires 'Devel::Confess'; requires 'Digest::MD5'; requires 'Digest::SHA1'; requires 'Search::Elasticsearch', '== 2.03'; @@ -50,6 +50,7 @@ requires 'JavaScript::Minifier::XS'; requires 'List::AllUtils'; requires 'List::MoreUtils'; requires 'List::Util'; +requires 'Log::Log4perl::Catalyst'; requires 'Locale::Country','3.28'; requires 'Module::Build::Tiny', '0.037'; requires 'Module::Runtime'; diff --git a/cpanfile.snapshot b/cpanfile.snapshot index e5c4b0fa13..4f92a4a9be 100644 --- a/cpanfile.snapshot +++ b/cpanfile.snapshot @@ -264,13 +264,6 @@ DISTRIBUTIONS perl 5.006 strict 0 warnings 0 - Carp-Always-0.13 - pathname: F/FE/FERREIRA/Carp-Always-0.13.tar.gz - provides: - Carp::Always 0.13 - requirements: - Carp 0 - ExtUtils::MakeMaker 0 Carp-Assert-0.21 pathname: N/NE/NEILB/Carp-Assert-0.21.tar.gz provides: @@ -1535,6 +1528,18 @@ DISTRIBUTIONS Test::Requires 0 parent 0 perl 5.008001 + Devel-Confess-0.009003 + pathname: H/HA/HAARG/Devel-Confess-0.009003.tar.gz + provides: + Devel::Confess 0.009003 + Devel::Confess::Builtin 0.009003 + Devel::Confess::Source undef + Devel::Confess::_Util undef + requirements: + Carp 0 + ExtUtils::MakeMaker 0 + Scalar::Util 0 + perl 5.006 Devel-GlobalDestruction-0.13 pathname: H/HA/HAARG/Devel-GlobalDestruction-0.13.tar.gz provides: @@ -2450,6 +2455,65 @@ DISTRIBUTIONS perl 5.008001 strict 0 warnings 0 + Log-Log4perl-1.48 + pathname: M/MS/MSCHILLI/Log-Log4perl-1.48.tar.gz + provides: + L4pResurrectable 0.01 + Log::Log4perl 1.48 + Log::Log4perl::Appender undef + Log::Log4perl::Appender::Buffer undef + Log::Log4perl::Appender::DBI undef + Log::Log4perl::Appender::File undef + Log::Log4perl::Appender::Limit undef + Log::Log4perl::Appender::RRDs undef + Log::Log4perl::Appender::Screen undef + Log::Log4perl::Appender::ScreenColoredLevels undef + Log::Log4perl::Appender::Socket undef + Log::Log4perl::Appender::String undef + Log::Log4perl::Appender::Synchronized undef + Log::Log4perl::Appender::TestArrayBuffer undef + Log::Log4perl::Appender::TestBuffer undef + Log::Log4perl::Appender::TestFileCreeper undef + Log::Log4perl::Catalyst undef + Log::Log4perl::Config undef + Log::Log4perl::Config::BaseConfigurator undef + Log::Log4perl::Config::DOMConfigurator 0.03 + Log::Log4perl::Config::PropertyConfigurator undef + Log::Log4perl::Config::Watch undef + Log::Log4perl::DateFormat undef + Log::Log4perl::Filter undef + Log::Log4perl::Filter::Boolean undef + Log::Log4perl::Filter::LevelMatch undef + Log::Log4perl::Filter::LevelRange undef + Log::Log4perl::Filter::MDC undef + Log::Log4perl::Filter::StringMatch undef + Log::Log4perl::InternalDebug undef + Log::Log4perl::JavaMap undef + Log::Log4perl::JavaMap::ConsoleAppender undef + Log::Log4perl::JavaMap::FileAppender undef + Log::Log4perl::JavaMap::JDBCAppender undef + Log::Log4perl::JavaMap::NTEventLogAppender undef + Log::Log4perl::JavaMap::RollingFileAppender undef + Log::Log4perl::JavaMap::SyslogAppender undef + Log::Log4perl::JavaMap::TestBuffer undef + Log::Log4perl::Layout undef + Log::Log4perl::Layout::NoopLayout undef + Log::Log4perl::Layout::PatternLayout undef + Log::Log4perl::Layout::PatternLayout::Multiline undef + Log::Log4perl::Layout::SimpleLayout undef + Log::Log4perl::Level undef + Log::Log4perl::Logger undef + Log::Log4perl::MDC undef + Log::Log4perl::NDC undef + Log::Log4perl::Resurrector undef + Log::Log4perl::Util undef + Log::Log4perl::Util::Semaphore undef + Log::Log4perl::Util::TimeTracker undef + requirements: + ExtUtils::MakeMaker 0 + File::Path 2.0606 + File::Spec 0.82 + Test::More 0.45 MCE-1.708 pathname: M/MA/MARIOROY/MCE-1.708.tar.gz provides: diff --git a/lib/MetaCPAN/Web.pm b/lib/MetaCPAN/Web.pm index de14c57032..9ac1b5a56f 100644 --- a/lib/MetaCPAN/Web.pm +++ b/lib/MetaCPAN/Web.pm @@ -10,7 +10,8 @@ use Catalyst qw/ Static::Simple Authentication +MetaCPAN::Role::Fastly::Catalyst - /; + /, '-Log=warn,error,fatal'; +use Log::Log4perl::Catalyst; extends 'Catalyst'; @@ -46,6 +47,20 @@ sub token { shift->request->session->get('token'); } +__PACKAGE__->log( + Log::Log4perl::Catalyst->new( + \q{ +log4perl.rootLogger=DEBUG, OUTPUT + +log4perl.appender.OUTPUT=Log::Log4perl::Appender::Screen +log4perl.appender.OUTPUT.stderr=1 + +log4perl.appender.OUTPUT.layout=PatternLayout +log4perl.appender.OUTPUT.layout.ConversionPattern=[%d] [%p] [%X{url}] %m%n +} + ) +); + __PACKAGE__->setup(); __PACKAGE__->meta->make_immutable; diff --git a/lib/MetaCPAN/Web/Controller/Root.pm b/lib/MetaCPAN/Web/Controller/Root.pm index 845b8393cb..24ee5dcb4c 100644 --- a/lib/MetaCPAN/Web/Controller/Root.pm +++ b/lib/MetaCPAN/Web/Controller/Root.pm @@ -27,6 +27,8 @@ sub auto : Private { if ( my $token = $c->token ) { $c->authenticate( { token => $token } ); } + Log::Log4perl::MDC->put( "ip", $c->req->address ); + Log::Log4perl::MDC->put( "url", $c->req->uri . '' ); return 1; } From b876d6c206972e075e8a932aec93bf0ce8d3534f Mon Sep 17 00:00:00 2001 From: Graham Knop Date: Fri, 18 Nov 2016 19:27:39 -0600 Subject: [PATCH 0062/1850] autoflush logs --- lib/MetaCPAN/Web.pm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/MetaCPAN/Web.pm b/lib/MetaCPAN/Web.pm index 9ac1b5a56f..3c2a768eb4 100644 --- a/lib/MetaCPAN/Web.pm +++ b/lib/MetaCPAN/Web.pm @@ -57,7 +57,10 @@ log4perl.appender.OUTPUT.stderr=1 log4perl.appender.OUTPUT.layout=PatternLayout log4perl.appender.OUTPUT.layout.ConversionPattern=[%d] [%p] [%X{url}] %m%n -} +}, + { + autoflush => 1, + }, ) ); From 8158f3abf6f724a7a3ce6863213632796dcffe89 Mon Sep 17 00:00:00 2001 From: Graham Knop Date: Fri, 18 Nov 2016 17:24:13 -0600 Subject: [PATCH 0063/1850] account for files with no modules in dist page --- lib/MetaCPAN/Web/Controller/Release.pm | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/MetaCPAN/Web/Controller/Release.pm b/lib/MetaCPAN/Web/Controller/Release.pm index 09fa86da55..2fb288bf9c 100644 --- a/lib/MetaCPAN/Web/Controller/Release.pm +++ b/lib/MetaCPAN/Web/Controller/Release.pm @@ -184,9 +184,9 @@ sub _files_to_categories { ); my @modules - = is_arrayref( $f->{module} ) - ? @{ $f->{module} } - : $f->{module}; + = is_arrayref( $f->{module} ) ? @{ $f->{module} } + : defined $f->{module} ? $f->{module} + : (); if ( $f->{documentation} and @modules ) { push @{ $ret->{modules} }, $f; From 1f25d3ad8838812c1a6a9751953acd364baa300f Mon Sep 17 00:00:00 2001 From: Graham Knop Date: Fri, 18 Nov 2016 19:37:54 -0600 Subject: [PATCH 0064/1850] fix autoflush option --- lib/MetaCPAN/Web.pm | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/MetaCPAN/Web.pm b/lib/MetaCPAN/Web.pm index 3c2a768eb4..59f25d48a7 100644 --- a/lib/MetaCPAN/Web.pm +++ b/lib/MetaCPAN/Web.pm @@ -58,9 +58,7 @@ log4perl.appender.OUTPUT.stderr=1 log4perl.appender.OUTPUT.layout=PatternLayout log4perl.appender.OUTPUT.layout.ConversionPattern=[%d] [%p] [%X{url}] %m%n }, - { - autoflush => 1, - }, + autoflush => 1, ) ); From 91f635fa6c536d2a1739b4c3ba7a354e76aa429f Mon Sep 17 00:00:00 2001 From: Leo Lapworth Date: Sat, 19 Nov 2016 01:46:37 +0000 Subject: [PATCH 0065/1850] block robots from searching based on size of results, they should just use defaults --- root/robots.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/root/robots.txt b/root/robots.txt index 92a8f7265d..28938f5fa3 100644 --- a/root/robots.txt +++ b/root/robots.txt @@ -6,5 +6,8 @@ Disallow: /diff/ Disallow: /source/ Disallow: /raw/ +# Do not allow changing the default per page as is not useful +Disallow: /*?*size=* + Sitemap: https://metacpan.org/static/sitemaps/authors.xml.gz Sitemap: https://metacpan.org/static/sitemaps/releases.xml.gz From 892c732543f38d56712850fee86a22e96d6a2158 Mon Sep 17 00:00:00 2001 From: Olaf Alders Date: Fri, 18 Nov 2016 21:05:13 -0600 Subject: [PATCH 0066/1850] Don't die when pagination runs out of results. --- lib/MetaCPAN/Web/Controller/Author.pm | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/lib/MetaCPAN/Web/Controller/Author.pm b/lib/MetaCPAN/Web/Controller/Author.pm index 9fbf81202c..f881133c3d 100644 --- a/lib/MetaCPAN/Web/Controller/Author.pm +++ b/lib/MetaCPAN/Web/Controller/Author.pm @@ -112,24 +112,26 @@ sub releases : Chained('root') PathPart Args(0) { my @releases = map { single_valued_arrayref_to_scalar( $_->{fields} ) } @{ $releases->{hits}->{hits} }; - my $pageset = Data::Pageset->new( + $c->stash( { - total_entries => $releases->{hits}->{total}, - entries_per_page => $page_size, - current_page => $page, - pages_per_set => 10, - mode => 'slide' + author => $author, + page_size => $page_size, + releases => \@releases, } ); - $c->stash( + return unless $releases->{hits}->{total}; + + my $pageset = Data::Pageset->new( { - releases => \@releases, - author => $author, - pageset => $pageset, - page_size => $page_size, + current_page => $page, + entries_per_page => $page_size, + mode => 'slide', + pages_per_set => 10, + total_entries => $releases->{hits}->{total}, } ); + $c->stash( { pageset => $pageset } ); } sub _calc_aggregated { From e2eaa8d101774d33273ec04913810a810ce7c952 Mon Sep 17 00:00:00 2001 From: Olaf Alders Date: Fri, 18 Nov 2016 21:27:14 -0600 Subject: [PATCH 0067/1850] Silence a warning that happens when a user possibly has a cookie but is not logged in. --- lib/MetaCPAN/Web/Controller/Account/Favorite.pm | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/MetaCPAN/Web/Controller/Account/Favorite.pm b/lib/MetaCPAN/Web/Controller/Account/Favorite.pm index 807b0ed45b..5936410d23 100644 --- a/lib/MetaCPAN/Web/Controller/Account/Favorite.pm +++ b/lib/MetaCPAN/Web/Controller/Account/Favorite.pm @@ -8,6 +8,13 @@ sub auto : Private { # Needed to clear the cache my $user = $c->model('API::User')->get_profile( $c->token )->recv; + + # Probably we shouldn't be getting to this point without a defined user, + # but sometimes we do. Might have been an issue with the v0 => v1 + # migration. + + $c->detach('/forbidden') unless $user; + $c->stash->{user} = $user; return 1; From 7d41a31dbb6aea76403a2bbe3e41cbcf59fce5a2 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Fri, 18 Nov 2016 19:41:12 -0800 Subject: [PATCH 0068/1850] Module search: Page correctly when fetching unique distributions The page size shouldn't dynamically scale without the starting point of each page also scaling in step. Otherwise, the pages ending up overlapping once $run is >1. This is the simpler fix. A slightly more complicated fix would be adjusting "from" instead to scale in step, but that seems unnecessary. --- lib/MetaCPAN/Web/Model/API/Module.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/MetaCPAN/Web/Model/API/Module.pm b/lib/MetaCPAN/Web/Model/API/Module.pm index 28c98bdbe2..0596847aeb 100644 --- a/lib/MetaCPAN/Web/Model/API/Module.pm +++ b/lib/MetaCPAN/Web/Model/API/Module.pm @@ -248,7 +248,7 @@ sub _search { return $self->search( $query, { - size => $run * $RESULTS_PER_RUN, + size => $RESULTS_PER_RUN, from => ( $run - 1 ) * $RESULTS_PER_RUN, fields => [qw(distribution)], $run == 1 From efc55a281665db6595b45b01e0dc6d573290ea67 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Fri, 18 Nov 2016 19:43:19 -0800 Subject: [PATCH 0069/1850] Module search: Avoid requesting duplicate fields from ES Requesting a field more than once produces the value more than once which causes the new ES to produce an array value for the field containing duplicate values. This busts our collapsed search which ended up making ref addresses unique instead of distribution names. With the root cause removed, the map which papered over the symptoms is not longer needed. --- lib/MetaCPAN/Web/Model/API/Module.pm | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/MetaCPAN/Web/Model/API/Module.pm b/lib/MetaCPAN/Web/Model/API/Module.pm index 0596847aeb..1489cc8f3d 100644 --- a/lib/MetaCPAN/Web/Model/API/Module.pm +++ b/lib/MetaCPAN/Web/Model/API/Module.pm @@ -130,8 +130,7 @@ sub search_collapsed { && $data->{hits}->{total} && $data->{hits}->{total} > $hits + ( $run - 2 ) * $RESULTS_PER_RUN ); - @distributions - = map { $_->[0] } splice( @distributions, $from, $page_size ); + @distributions = splice( @distributions, $from, $page_size ); # Everything else will fail (slowly and quietly) without distributions. if ( !@distributions ) { @@ -415,6 +414,12 @@ sub search { } ); + # Ensure our requested fields are unique so that Elasticsearch doesn't + # return us the same value multiple times in an unexpected arrayref. For + # example, distribution is listed both above and in ->_search, which calls + # this function (->search) and gets merged with the query above. + $search->{fields} = [ uniq @{ $search->{fields} || [] } ]; + return $self->request( '/file/_search', $search ); } From a15e24e51cabd710eca6a24c1c5214732101ef85 Mon Sep 17 00:00:00 2001 From: Graham Knop Date: Sat, 19 Nov 2016 10:57:45 -0600 Subject: [PATCH 0070/1850] bring boosts back to what we used with v0 --- lib/MetaCPAN/Web/Model/API/Module.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/MetaCPAN/Web/Model/API/Module.pm b/lib/MetaCPAN/Web/Model/API/Module.pm index 1489cc8f3d..94644ffb1c 100644 --- a/lib/MetaCPAN/Web/Model/API/Module.pm +++ b/lib/MetaCPAN/Web/Model/API/Module.pm @@ -292,7 +292,7 @@ sub search { term => { 'documentation' => { value => $query, - boost => 100 + boost => 20 } } }, @@ -300,7 +300,7 @@ sub search { term => { 'module.name' => { value => $query, - boost => 100 + boost => 20 } } }, From d743f81bb62bc4a4ad730e6155274ca6c3886177 Mon Sep 17 00:00:00 2001 From: Olaf Alders Date: Sat, 19 Nov 2016 13:08:06 -0600 Subject: [PATCH 0071/1850] Re-attempt to fix e2eaa8d101. --- lib/MetaCPAN/Web/Controller/Account/Favorite.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/MetaCPAN/Web/Controller/Account/Favorite.pm b/lib/MetaCPAN/Web/Controller/Account/Favorite.pm index 5936410d23..838417ea56 100644 --- a/lib/MetaCPAN/Web/Controller/Account/Favorite.pm +++ b/lib/MetaCPAN/Web/Controller/Account/Favorite.pm @@ -13,7 +13,7 @@ sub auto : Private { # but sometimes we do. Might have been an issue with the v0 => v1 # migration. - $c->detach('/forbidden') unless $user; + $c->detach('/forbidden') unless $user && $user->{user}; $c->stash->{user} = $user; From f0299337c8d235827188c1d9aded1f5a7b2675a2 Mon Sep 17 00:00:00 2001 From: Olaf Alders Date: Sat, 19 Nov 2016 13:50:01 -0600 Subject: [PATCH 0072/1850] Fixes warning in Changes date formatter. --- lib/MetaCPAN/Web/Model/API/Changes/Parser.pm | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/lib/MetaCPAN/Web/Model/API/Changes/Parser.pm b/lib/MetaCPAN/Web/Model/API/Changes/Parser.pm index 64a7a02681..64d0764be9 100644 --- a/lib/MetaCPAN/Web/Model/API/Changes/Parser.pm +++ b/lib/MetaCPAN/Web/Model/API/Changes/Parser.pm @@ -46,16 +46,19 @@ sub parse { $date = $1; } - # handle localtime-like timestamps + # handle localtime-like timestamps + # May Tue 03 17:25:00 2005 + # /changes/distribution/Catalyst-View-PSP + # XXX haarg is going to rip this out and replace it with something better. elsif ( $note - =~ s{^\D{3}\s+(\D{3})\s+(\d{1,2})\s+([\d:]+)?\D*(\d{4})}{} + =~ s{^(\D{3})\s+(\D{3})\s+(\d{1,2})\s+([\d:]+)?\D*(\d{4})}{} ) { - if ($3) { + if ($4) { # unfortunately ignores TZ data $date = sprintf( '%d-%02d-%02dT%sZ', - $4, $months{$1}, $2, $3 ); + $5, $months{$1}, $3, $4 ); } else { $date From ed3e71cb034601e818cff2854c0a4fe9c0429f4d Mon Sep 17 00:00:00 2001 From: Leo Lapworth Date: Sat, 19 Nov 2016 20:09:15 +0000 Subject: [PATCH 0073/1850] stop errors and forbidden caching --- cpanfile | 2 +- cpanfile.snapshot | 17 ++++++++++++----- lib/MetaCPAN/Web/Controller/Root.pm | 4 ++++ 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/cpanfile b/cpanfile index 7c1c25ff7c..6878c83bba 100644 --- a/cpanfile +++ b/cpanfile @@ -62,7 +62,7 @@ requires 'MooseX::StrictConstructor'; requires 'MooseX::Fastly::Role', '0.02'; requires 'PerlIO::gzip'; requires 'MetaCPAN::Moose'; -requires 'MetaCPAN::Role', '0.05'; +requires 'MetaCPAN::Role', '0.06'; requires 'MooseX::Types::Common::Numeric'; requires 'MooseX::Types::Common::String'; requires 'MooseX::Types::Moose'; diff --git a/cpanfile.snapshot b/cpanfile.snapshot index 4f92a4a9be..7ac4a847d0 100644 --- a/cpanfile.snapshot +++ b/cpanfile.snapshot @@ -264,6 +264,13 @@ DISTRIBUTIONS perl 5.006 strict 0 warnings 0 + Carp-Always-0.13 + pathname: F/FE/FERREIRA/Carp-Always-0.13.tar.gz + provides: + Carp::Always 0.13 + requirements: + Carp 0 + ExtUtils::MakeMaker 0 Carp-Assert-0.21 pathname: N/NE/NEILB/Carp-Assert-0.21.tar.gz provides: @@ -2592,12 +2599,12 @@ DISTRIBUTIONS perl 5.006 strict 0 warnings 0 - MetaCPAN-Role-0.05 - pathname: L/LL/LLAP/MetaCPAN-Role-0.05.tar.gz + MetaCPAN-Role-0.06 + pathname: L/LL/LLAP/MetaCPAN-Role-0.06.tar.gz provides: - MetaCPAN::Role 0.05 - MetaCPAN::Role::Fastly 0.05 - MetaCPAN::Role::Fastly::Catalyst 0.05 + MetaCPAN::Role 0.06 + MetaCPAN::Role::Fastly 0.06 + MetaCPAN::Role::Fastly::Catalyst 0.06 requirements: Carp 0 CatalystX::Fastly::Role::Response 0.04 diff --git a/lib/MetaCPAN/Web/Controller/Root.pm b/lib/MetaCPAN/Web/Controller/Root.pm index 24ee5dcb4c..64a3629d8f 100644 --- a/lib/MetaCPAN/Web/Controller/Root.pm +++ b/lib/MetaCPAN/Web/Controller/Root.pm @@ -61,6 +61,8 @@ sub default : Path { sub not_found : Private { my ( $self, $c ) = @_; + $c->cdn_never_cache(1); + $c->stash( { template => 'not_found.html', @@ -72,6 +74,8 @@ sub not_found : Private { sub forbidden : Private { my ( $self, $c ) = @_; + $c->cdn_never_cache(1); + $c->stash( { template => 'forbidden.html' } ); $c->response->status(403); } From 4302a53b1d0a08cf5a41dcf19e9142877f366b13 Mon Sep 17 00:00:00 2001 From: Olaf Alders Date: Sat, 19 Nov 2016 14:13:19 -0600 Subject: [PATCH 0074/1850] Avoid "splice() offset past end of array" warning. --- lib/MetaCPAN/Web/Model/API/Module.pm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/MetaCPAN/Web/Model/API/Module.pm b/lib/MetaCPAN/Web/Model/API/Module.pm index 94644ffb1c..6395b783fa 100644 --- a/lib/MetaCPAN/Web/Model/API/Module.pm +++ b/lib/MetaCPAN/Web/Model/API/Module.pm @@ -130,7 +130,11 @@ sub search_collapsed { && $data->{hits}->{total} && $data->{hits}->{total} > $hits + ( $run - 2 ) * $RESULTS_PER_RUN ); - @distributions = splice( @distributions, $from, $page_size ); + # Avoid "splice() offset past end of array" warning. + @distributions + = $from > @distributions + ? () + : splice( @distributions, $from, $page_size ); # Everything else will fail (slowly and quietly) without distributions. if ( !@distributions ) { From 6513562597f5dd2e4ad9f1edd24ce9bf59158ce4 Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Sat, 19 Nov 2016 14:44:50 -0800 Subject: [PATCH 0075/1850] Fix the Bootstrap .clearfix class to actually fix clearing I have no idea how or why clear: both turned into clear: none in commit e11081f, which ostensibly upgraded to Bootstrap 3.1.1. That release of Bootstrap had clear: both in the &:after rule, so perhaps a spurious local modification made it into the commit. The upshot of this is that the keyboard shortcuts modal dialog isn't busted any more. --- root/static/less/bootstrap/mixins.less | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/root/static/less/bootstrap/mixins.less b/root/static/less/bootstrap/mixins.less index 1d32c0ef98..71723dba4a 100644 --- a/root/static/less/bootstrap/mixins.less +++ b/root/static/less/bootstrap/mixins.less @@ -23,7 +23,7 @@ display: table; // 2 } &:after { - clear: none; + clear: both; } } From a2336a8e8327d7d94130c3b2ad3c4d45bc05d26a Mon Sep 17 00:00:00 2001 From: Thomas Sibley Date: Sat, 19 Nov 2016 14:58:16 -0800 Subject: [PATCH 0076/1850] Link to fastapi.metacpan.org instead of the longer API-docs.md URL Easier to maintain and a single place to update. --- root/about/faq.html | 2 +- root/wrapper.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/root/about/faq.html b/root/about/faq.html index b501a0d1d0..356d7d9fb8 100644 --- a/root/about/faq.html +++ b/root/about/faq.html @@ -64,7 +64,7 @@ ## Where can I find the API docs? -The API docs can be found [in the API repository](https://github.com/metacpan/metacpan-api/blob/master/docs/API-docs.md). +The API docs can be found by visiting [fastapi.metacpan.org](https://fastapi.metacpan.org). API requests need to be sent to fastapi.metacpan.org. ## How can I try the API? diff --git a/root/wrapper.html b/root/wrapper.html index 1040a4ebf7..3257020ec5 100644 --- a/root/wrapper.html +++ b/root/wrapper.html @@ -223,7 +223,7 @@