Import news articles from Content Publisher (WHIT-2440)

[ChrisBAshton]

Merging PR blocked on:

• Not sending 'headers' property when publishing news articles to Publishing API
• WHIT-2498 Sending extra 'details' (change history, political, etc)
• WHIT-2386 Associations (mostly organisations) being added to StandardEdition
• WHIT-2384 Deciding how we override lead image
• WHIT-2484 Press Releases being a valid StandardEdition format

Running the import blocked on:

• WHIT-2519 Putting out comms before retiring Content Publisher

Testing

I've successfully tested importing ALL articles locally and on integration, including route overwriting ✅

Example doc on Content Publisher: https://content-publisher.integration.publishing.service.gov.uk/documents/36d03d5e-eac6-4c18-9d29-f02f6bbf6cc1:en

Content Publisher	Whitehall

Differences:

• No organisations yet (pending WHIT-2386)
• No visible history mode yet (pending WHIT-2498)
• 'Flattened' document history

Live article (via Content Publisher)	Integration article (via Whitehall)

Differences:

• No organisations yet (pending WHIT-2386)
• History mode SHOULD now be working (since WHIT-2498)
• No lead image (pending WHIT-2384)

Changenote history now works (as of WHIT-2498):

As does publishing new major versions:

What

Sister PR to alphagov/content-publisher#3311.

This PR takes the output from the above (the Content Publisher export) and imports those news stories and press releases into Whitehall. It carries over associations, images and attachments, as well as a condensed document history, but we haven't bothered to try to replicate each individual edition - only the published editions.

Example rake task to import a single news article:

rake import:news_article["../content-publisher/tmp/secretary-of-state-for-work-and-pensions-attends-g7-labour-and-employment-ministerial-meeting-in-germany.json"]

How to run

Importing a single news article via its JSON:

$ kubectl get pods
# returns list of pods, including
# whitehall-admin-c4c7c957c-9q966

$ kubectl cp ~/Downloads/sample-article.json whitehall-admin-c4c7c957c-9q966:/tmp
# copies the file

$ kubectl exec whitehall-admin-c4c7c957c-9q966 -- rake import:news_article["/tmp/sample-article.json"]
# runs the rake task

Importing all of the news articles:

$ kubectl get pods
# returns list of Whitehall application pods (such as whitehall-admin-c4c7c957c-9q966)
# and Whitehall worker pods (such as whitehall-admin-worker-777ff7f7b7-lhfkb)

# Recursively copy the local directory into the worker pods
$ kubectl cp /path/to/your/news-articles whitehall-admin-worker-777ff7f7b7-lhfkb:/tmp/news-articles
$ kubectl cp /path/to/your/news-articles whitehall-admin-worker-another-one-lhfkb:/tmp/news-articles

# Now do the same for one of the application pods
$ kubectl cp /path/to/your/news-articles whitehall-admin-c4c7c957c-9q966:/tmp/news-articles

# Finally, run the rake task on the application pod you uploaded to.
# (Best do this directly on the pod, in case of time-outs)
$ kubectl exec whitehall-admin-c4c7c957c-9q966 -- rake import:news_articles_in_directory["/tmp/news-articles"]
# runs the rake task

Explanation:

1. We need to upload the JSON files to the application pod so that the workers get enqueued in the first place. (The results of Dir.glob("#{args[:dir_path]}/**/*.json").each is what powers the enqueuing - so we need to ensure all the files are there to be globbed!)
2. We need to upload the JSON files to ALL the worker pods so that every worker has access to the given JSON file. (Workers will just look locally for /tmp/news-articles/<file>.json, so it needs to exist locally).
3. Note that you will need to re-upload all of the JSON files to all of the pods as above, every time there is a deployment to Whitehall (which tears down the pods and rebuilds them)

Sense check that it all imported correctly:

Document.where(document_type: "StandardEdition").map(&:live_edition).select { |ed| ed && ["press_release", "news_story"].include?(ed.configurable_document_type)}.count
=> 2979
^ there should be 2979, the same as the number of JSON files in the local export

expected_slugs = Dir.glob("/tmp/news-articles/**/*.json").map { |filepath| filepath.split("/").last.sub(".json", "") }
=> 
["1-3-billion-investment-to-deliver-homes-infrastructure-and-jobs",
...
actual_slugs = Document.where(document_type: "StandardEdition").pluck(:slug)
=> 
["10-downing-street",
...

expected_slugs - actual_slugs
=> ["helicopter-services-deal-raises-competition-concerns-1", "non-executive-director-appointed-to-gov-facility-services-limited-1"]

^ This last one should be an empty array. If for whatever reason there are some stragglers, they can be imported by running the bulk task again, or importing individually (rake import:news_article["/tmp/news-articles/non-executive-director-appointed-to-gov-facility-services-limited-1.json"]).

Finally, we can destroy all StandardEdition news articles (Edition.where(configurable_document_type: ["news_article", "press_release"]).map(&:document).map(&:destroy)) if we want to test re-import everything (e.g. after making a tweak).

Post merge

Post merge, we should bulk import the news articles as above, and then revert this PR (none of the changes are required to remain once the articles have been migrated).

Why

Migrating content out of Content Publisher will allow us to retire that long-deprecated app 🎉
It will also be a good first test case for our new config-driven news articles format.

In terms of the comprehensiveness of transfer of content, we're striking a balance of replicating just the published news articles, as best as we can, without worrying about replicating the entire document history. We're also not bothering to transfer drafts nor deleted/'gone' content. There is no draft newer than early 2024 and the nature of news articles is such that it's extremely unlikely an update will need to be applied. Publishers will be able to create new drafts on the migrated content once it's in Whitehall. Similarly, a 'gone' news article is extremely unlikely to need to be brought back, and publishers can always replicate the 'gone' news article manually if needed.

JIRA: https://gov-uk.atlassian.net/browse/WHIT-2440

---

⚠️ This repo is Continuously Deployed: make sure you follow the guidance ⚠️

This application is owned by the Whitehall Experience team. Please let us know in #govuk-whitehall-experience-tech when you raise any PRs.

Follow these steps if you are doing a Rails upgrade.