CLDC-4191: Q84 Referral CSV and XML export (#3157)

* CLDC-4151: Note deprecations of existing referral questions

* CLDC-4151: Add new cols to lettings log model

* CLDC-4151: Add new referrals questions

* CLDC-4151: Remove referral value check

* CLDC-4151: Add new fields to bulk upload

use new fields as presented in bulk upload

* CLDC-4151: Update tests

also add bulk upload file to test with

* CLDC-4151: Ignore tests failing for later tickets

* fixup! CLDC-4151: Add new referrals questions

start q nums from 2026

* CLDC-4151: Add model tests for new questions

* fixup! CLDC-4151: Add new fields to bulk upload

export old_visible_id

set MAX_COLUMNS correctly

check .prp? for prp cols

* fixup! CLDC-4151: Update tests

clarify tests that may update

* fixup! CLDC-4151: Add new fields to bulk upload

handle the old_visible_id not existing

use override org ID only for the output owning ID. it's only passed as eg ORG1

* CLDC-4151: Add new questions to log factory

* fixup! CLDC-4151: Add new fields to bulk upload

leave field_131 validations till later

* CLDC-4188: Add LA flow

splits the referral_register question and pages into two, as its not easy to have the answers be dependent

if the owning organisation changes type, reset the referral register question. the other questions do not need to be reset as they are no longer routed to

* CLDC-4188: Infer referral_register for renewals

needs a new dependencies array for 2026

this handles inferring as well as clearing inferred answers if the prior answer changes

* CLDC-4188: Add validation between prevten and referral_register

block two other validations from previous years

* CLDC-4188: Update tests

* CLDC-4188: Ignore tests failing for future tickets

* CLDC-4188: Update existing tests

needed as should_reset_referral_register? calls a .find()

* fixup! CLDC-4188: Add validation between prevten and referral_register

use .prp?

improve grammar

* fixup! CLDC-4188: Infer referral_register for renewals

extract dependencies to common list

* fixup! CLDC-4188: Update tests

use before and context blocks better

* fixup! CLDC-4188: Update tests

lint

* fixup! CLDC-4188: Update tests

remove unneeded referral_type set

* CLDC-4189: Add PRP flow

* CLDC-4189: Update validation between prevten and referral_register

* CLDC-4189: Add tests

* fixup! CLDC-4189: Add PRP flow

fix typo

Co-authored-by: Oscar Richardson <116292912+oscar-richardson-softwire@users.noreply.github.com>

* fixup! CLDC-4189: Add tests

fix typo

* fixup! CLDC-4189: Add tests

fix missing cases in page specs

use a loop and before blocks for household validation tests

* CLDC-4189: Ensure log is not classed as generan needs if prevten isnt answered

* fixup! CLDC-4189: Add tests

add cases for other internal transfer flow

* CLDC-4190: Invalidate all referral fields if any are wrong

* CLDC-4190: Remove other validations on referral fields

* CLDC-4190: Ignore referral validation if BU is renewal

* CLDC-4190: Add tests

* fixup! CLDC-4190: Invalidate all referral fields if any are wrong

add punctuation to comment

name validation fields better

* fixup! CLDC-4190: Add tests

make test names clearer

* CLDC-4191: Update CSV export tests

the export happens by default, no changes needed here

* CLDC-4191: Add new referral fields to 2026 exports

remove old referral fields from 2026 exports

ensure new referral fields don't show in pre 2026 exports

* CLDC-4190: Final field fixes

---------

Co-authored-by: Oscar Richardson <116292912+oscar-richardson-softwire@users.noreply.github.com>

app/services/exports/lettings_log_export_constants.rb

@@ -77,7 +77,6 @@ module Exports::LettingsLogExportConstants
     "reason",
     "reasonother",
     "reasonpref",
-    "referral",
     "refused",
     "reghome",
     "renttype",
@@ -158,6 +157,7 @@ module Exports::LettingsLogExportConstants
     "chcharge",
     "national",
     "offered",
+    "referral",
   ]
 
   YEAR_2022_EXPORT_FIELDS = Set[
@@ -165,6 +165,7 @@ module Exports::LettingsLogExportConstants
     "chcharge",
     "national",
     "offered",
+    "referral",
   ]
 
   YEAR_2023_EXPORT_FIELDS = Set[
@@ -172,6 +173,7 @@ module Exports::LettingsLogExportConstants
     "chcharge",
     "national",
     "offered",
+    "referral",
   ]
 
   YEAR_2024_EXPORT_FIELDS = Set[
@@ -192,6 +194,7 @@ module Exports::LettingsLogExportConstants
     "pscharge_value_check",
     "supcharg_value_check",
     "carehome_charges_value_check",
+    "referral",
   ]
 
   YEAR_2025_EXPORT_FIELDS = Set[
@@ -210,6 +213,7 @@ module Exports::LettingsLogExportConstants
     "scharge_value_check",
     "pscharge_value_check",
     "supcharg_value_check",
+    "referral",
   ]
 
   YEAR_2026_EXPORT_FIELDS = Set[
@@ -227,6 +231,9 @@ module Exports::LettingsLogExportConstants
     "scharge_value_check",
     "pscharge_value_check",
     "supcharg_value_check",
+    "referral_register",
+    "referral_noms",
+    "referral_org",
   ]
 
   (1..8).each do |index|

spec/fixtures/exports/general_needs_log_26_27.xml

@@ -123,7 +123,9 @@
     <totelder>0</totelder>
     <totadult>2</totadult>
     <nocharge/>
-    <referral>2</referral>
+    <referral_register>1</referral_register>
+    <referral_noms/>
+    <referral_org/>
     <brent>200.0</brent>
     <scharge>50.0</scharge>
     <pscharge>40.0</pscharge>

spec/services/csv/lettings_log_csv_service_spec.rb

@@ -336,8 +336,7 @@ RSpec.describe Csv::LettingsLogCsvService do
           context "when the current user is a support user" do
             let(:user) { create(:user, :support, organisation:, email: "s.port@jeemayle.com") }
 
-            # TODO: CLDC-4191 Reinstate this test when we update log export
-            xit "exports the CSV with all values correct" do
+            it "exports the CSV with all values correct" do
               expected_content = CSV.read("spec/fixtures/files/lettings_log_csv_export_labels_26.csv")
               values_to_delete = %w[id]
               values_to_delete.each do |attribute|
@@ -351,8 +350,7 @@ RSpec.describe Csv::LettingsLogCsvService do
           context "when the current user is not a support user" do
             let(:user) { create(:user, :data_provider, organisation:, email: "choreographer@owtluk.com") }
 
-            # TODO: CLDC-4191 Reinstate this test when we update log export
-            xit "exports the CSV with all values correct" do
+            it "exports the CSV with all values correct" do
               expected_content = CSV.read("spec/fixtures/files/lettings_log_csv_export_non_support_labels_26.csv")
               values_to_delete = %w[id]
               values_to_delete.each do |attribute|
@@ -370,8 +368,7 @@ RSpec.describe Csv::LettingsLogCsvService do
           context "when the current user is a support user" do
             let(:user) { create(:user, :support, organisation:, email: "s.port@jeemayle.com") }
 
-            # TODO: CLDC-4191 Reinstate this test when we update log export
-            xit "exports the CSV with all values correct" do
+            it "exports the CSV with all values correct" do
               expected_content = CSV.read("spec/fixtures/files/lettings_log_csv_export_codes_26.csv")
               values_to_delete = %w[id]
               values_to_delete.each do |attribute|
@@ -385,8 +382,7 @@ RSpec.describe Csv::LettingsLogCsvService do
           context "when the current user is not a support user" do
             let(:user) { create(:user, :data_provider, organisation:, email: "choreographer@owtluk.com") }
 
-            # TODO: CLDC-4191 Reinstate this test when we update log export
-            xit "exports the CSV with all values correct" do
+            it "exports the CSV with all values correct" do
               expected_content = CSV.read("spec/fixtures/files/lettings_log_csv_export_non_support_codes_26.csv")
               values_to_delete = %w[id]
               values_to_delete.each do |attribute|

spec/services/exports/lettings_log_export_service_spec.rb

@@ -536,8 +536,7 @@ RSpec.describe Exports::LettingsLogExportService do
         let(:expected_data_filename) { "core_2026_2027_apr_mar_f0001_inc0001_pt001.xml" }
         let(:xml_export_file) { File.open("spec/fixtures/exports/general_needs_log_26_27.xml", "r:UTF-8") }
 
-        # TODO: CLDC-4191 Reinstate this test when we update log export
-        xit "generates an XML export file with the expected content within the ZIP file" do
+        it "generates an XML export file with the expected content within the ZIP file" do
           expected_content = replace_entity_ids(lettings_log, xml_export_file.read)
           expect(storage_service).to receive(:write_file).with(expected_zip_filename, any_args) do |_, content|
             entry = Zip::File.open_buffer(content).find_entry(expected_data_filename)