CLDC-4190: Q84 Referral BU validations (#3155)

* CLDC-4151: Note deprecations of existing referral questions

* CLDC-4151: Add new cols to lettings log model

* CLDC-4151: Add new referrals questions

* CLDC-4151: Remove referral value check

* CLDC-4151: Add new fields to bulk upload

use new fields as presented in bulk upload

* CLDC-4151: Update tests

also add bulk upload file to test with

* CLDC-4151: Ignore tests failing for later tickets

* fixup! CLDC-4151: Add new referrals questions

start q nums from 2026

* CLDC-4151: Add model tests for new questions

* fixup! CLDC-4151: Add new fields to bulk upload

export old_visible_id

set MAX_COLUMNS correctly

check .prp? for prp cols

* fixup! CLDC-4151: Update tests

clarify tests that may update

* fixup! CLDC-4151: Add new fields to bulk upload

handle the old_visible_id not existing

use override org ID only for the output owning ID. it's only passed as eg ORG1

* CLDC-4151: Add new questions to log factory

* fixup! CLDC-4151: Add new fields to bulk upload

leave field_131 validations till later

* CLDC-4188: Add LA flow

splits the referral_register question and pages into two, as its not easy to have the answers be dependent

if the owning organisation changes type, reset the referral register question. the other questions do not need to be reset as they are no longer routed to

* CLDC-4188: Infer referral_register for renewals

needs a new dependencies array for 2026

this handles inferring as well as clearing inferred answers if the prior answer changes

* CLDC-4188: Add validation between prevten and referral_register

block two other validations from previous years

* CLDC-4188: Update tests

* CLDC-4188: Ignore tests failing for future tickets

* CLDC-4188: Update existing tests

needed as should_reset_referral_register? calls a .find()

* fixup! CLDC-4188: Add validation between prevten and referral_register

use .prp?

improve grammar

* fixup! CLDC-4188: Infer referral_register for renewals

extract dependencies to common list

* fixup! CLDC-4188: Update tests

use before and context blocks better

* fixup! CLDC-4188: Update tests

lint

* fixup! CLDC-4188: Update tests

remove unneeded referral_type set

* CLDC-4189: Add PRP flow

* CLDC-4189: Update validation between prevten and referral_register

* CLDC-4189: Add tests

* fixup! CLDC-4189: Add PRP flow

fix typo

Co-authored-by: Oscar Richardson <116292912+oscar-richardson-softwire@users.noreply.github.com>

* fixup! CLDC-4189: Add tests

fix typo

* fixup! CLDC-4189: Add tests

fix missing cases in page specs

use a loop and before blocks for household validation tests

* CLDC-4189: Ensure log is not classed as generan needs if prevten isnt answered

* fixup! CLDC-4189: Add tests

add cases for other internal transfer flow

* CLDC-4190: Invalidate all referral fields if any are wrong

* CLDC-4190: Remove other validations on referral fields

* CLDC-4190: Ignore referral validation if BU is renewal

* CLDC-4190: Add tests

* fixup! CLDC-4190: Invalidate all referral fields if any are wrong

add punctuation to comment

name validation fields better

* fixup! CLDC-4190: Add tests

make test names clearer

* CLDC-4190: Final field fixes

---------

Co-authored-by: Oscar Richardson <116292912+oscar-richardson-softwire@users.noreply.github.com>

app/services/bulk_upload/lettings/year2026/row_parser.rb

@@ -466,8 +466,6 @@ class BulkUpload::Lettings::Year2026::RowParser
   validate :validate_needs_type_present, on: :after_log
   validate :validate_data_types, on: :after_log
   validate :validate_relevant_collection_window, on: :after_log
-  validate :validate_la_with_local_housing_referral, on: :after_log
-  validate :validate_cannot_be_la_referral_if_general_needs_and_la, on: :after_log
   validate :validate_leaving_reason_for_renewal, on: :after_log
   validate :validate_only_one_housing_needs_type, on: :after_log
   validate :validate_no_disabled_needs_conjunction, on: :after_log
@@ -478,6 +476,7 @@ class BulkUpload::Lettings::Year2026::RowParser
   validate :validate_reasonable_preference_dont_know, on: :after_log
   validate :validate_condition_effects, on: :after_log
   validate :validate_if_log_already_exists, on: :after_log, if: -> { FeatureToggle.bulk_upload_duplicate_log_check_enabled? }
+  validate :validate_referral_fields, on: :after_log
 
   validate :validate_owning_org_data_given, on: :after_log
   validate :validate_owning_org_exists, on: :after_log
@@ -738,7 +737,7 @@ private
   end
 
   def validate_prevten_value_when_renewal
-    return unless field_7 == 1
+    return unless renewal?
     return if field_100.blank? || [6, 30, 31, 32, 33, 34, 35, 38].include?(field_100)
 
     errors.add(:field_100, I18n.t("#{ERROR_BASE_KEY}.prevten.invalid"))
@@ -845,7 +844,7 @@ private
   end
 
   def validate_leaving_reason_for_renewal
-    if field_7 == 1 && ![50, 51, 52, 53].include?(field_98)
+    if renewal? && ![50, 51, 52, 53].include?(field_98)
       errors.add(:field_98, I18n.t("#{ERROR_BASE_KEY}.reason.renewal_reason_needed"))
     end
   end
@@ -858,16 +857,8 @@ private
     field_4 == 2
   end
 
-  def validate_cannot_be_la_referral_if_general_needs_and_la
-    if field_116 == 4 && general_needs? && owning_organisation && owning_organisation.la?
-      errors.add :field_116, I18n.t("#{ERROR_BASE_KEY}.referral.general_needs_prp_referred_by_la")
-    end
-  end
-
-  def validate_la_with_local_housing_referral
-    if field_116 == 3 && owning_organisation && owning_organisation.la?
-      errors.add(:field_116, I18n.t("#{ERROR_BASE_KEY}.referral.nominated_by_local_ha_but_la"))
-    end
+  def renewal?
+    field_7 == 1
   end
 
   def validate_relevant_collection_window
@@ -1050,6 +1041,57 @@ private
     end
   end
 
+  def field_referral_register_la_valid?
+    if owning_organisation&.la?
+      [1, 2, 3, 4].include?(field_116)
+    else
+      field_116.blank?
+    end
+  end
+
+  def field_referral_register_prp_valid?
+    if owning_organisation&.prp?
+      [5, 6, 7, 8, 9].include?(field_154)
+    else
+      field_154.blank?
+    end
+  end
+
+  def field_referral_noms_valid?
+    case field_154
+    when 6
+      [1, 2, 3, 4].include?(field_155)
+    when 7
+      [5, 6, 7, 8].include?(field_155)
+    else
+      field_155.blank?
+    end
+  end
+
+  def field_referral_org_valid?
+    case field_155
+    when 1
+      [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].include?(field_156)
+    when 7
+      [11, 12, 13, 14, 15, 16, 17, 18, 19, 20].include?(field_156)
+    else
+      field_156.blank?
+    end
+  end
+
+  def referral_fields_valid?
+    field_referral_register_la_valid? && field_referral_register_prp_valid? && field_referral_noms_valid? && field_referral_org_valid?
+  end
+
+  def validate_referral_fields
+    return if renewal?
+    return if referral_fields_valid?
+
+    %i[field_116 field_154 field_155 field_156].each do |field|
+      errors.add(field, I18n.t("#{ERROR_BASE_KEY}.referral.invalid_option"))
+    end
+  end
+
   def field_mapping_for_errors
     {
       lettype: [:field_11],
@@ -1780,6 +1822,11 @@ private
 
   def referral_register
     return unless owning_organisation
+    # by default CORE will ingest all these fields and nil questions that aren't asked.
+    # here, we specifically want the log to be invalid if any of the referral fields are wrong.
+    # BU will only consider a log invalid if its incomplete.
+    # so, nil these fields if any are invalid.
+    return unless referral_fields_valid?
 
     if owning_organisation.la?
       field_116
@@ -1789,12 +1836,20 @@ private
   end
 
   def referral_noms
-    field_155
+    return unless owning_organisation
+    return unless referral_fields_valid?
+
+    if owning_organisation.prp?
+      field_155
+    end
   end
 
   def referral_org
     return unless owning_organisation
+    return unless referral_fields_valid?
 
-    field_156
+    if owning_organisation.prp?
+      field_156
+    end
   end
 end

config/locales/validations/lettings/2026/bulk_upload.en.yml

@@ -40,8 +40,7 @@ en:
           reason:
             renewal_reason_needed: "The reason for leaving must be \"End of social or private sector tenancy - no fault\", \"End of social or private sector tenancy - evicted due to anti-social behaviour (ASB)\", \"End of social or private sector tenancy - evicted due to rent arrears\" or \"End of social or private sector tenancy - evicted for any other reason\"."
           referral:
-            general_needs_prp_referred_by_la: "The source of the referral cannot be referred by local authority housing department for a general needs log."
-            nominated_by_local_ha_but_la: "The source of the referral cannot be Nominated by local housing authority as your organisation is a local authority."
+            invalid_option: "Your answers for each part of \"What is the source of referral for this letting?\" are incompatible with each other. Use the bulk upload specification or paper form to see which combinations are valid (available from ‘Collection resources’ on the homepage)."
           scheme:
             must_relate_to_org: "This scheme code does not belong to the owning organisation or managing organisation."
           location:

spec/services/bulk_upload/lettings/year2026/row_parser_spec.rb

@@ -225,9 +225,6 @@ RSpec.describe BulkUpload::Lettings::Year2026::RowParser do
             field_115: "2",
 
             field_116: "1",
-            field_154: "1",
-            field_155: "1",
-            field_156: "1",
 
             field_117: "1",
             field_118: "2",
@@ -651,12 +648,12 @@ RSpec.describe BulkUpload::Lettings::Year2026::RowParser do
         end
 
         context "when an invalid value error has been added" do
-          let(:attributes) { setup_section_params.merge({ field_116: "100" }) }
+          let(:attributes) { setup_section_params.merge({ field_115: "100" }) }
 
           it "does not add an additional error" do
             parser.valid?
-            expect(parser.errors[:field_116].length).to eq(1)
-            expect(parser.errors[:field_116]).to include(match I18n.t("validations.lettings.2026.bulk_upload.invalid_option", question: ""))
+            expect(parser.errors[:field_115].length).to eq(1)
+            expect(parser.errors[:field_115]).to include(match I18n.t("validations.lettings.2026.bulk_upload.invalid_option", question: ""))
           end
         end
       end
@@ -1163,10 +1160,239 @@ RSpec.describe BulkUpload::Lettings::Year2026::RowParser do
       end
     end
 
-    # TODO: CLDC-4191: Add tests for the new referral fields
-    # describe "#field_116" do # referral
-    #
-    # end
+    describe "#field_116, field_154, field_155, field_156" do # referral
+      context "when org is LA" do
+        let(:owning_org) { create(:organisation, :la, :with_old_visible_id) }
+
+        let(:org_attributes) { { bulk_upload:, field_1: owning_org.old_visible_id } }
+
+        context "and not renewal" do
+          let(:renewal_attributes) { org_attributes.merge({ field_7: nil }) }
+
+          context "and field_116 is valid" do
+            let(:attributes) { renewal_attributes.merge({ field_116: 1 }) }
+
+            it "does not add an error" do
+              parser.valid?
+              expect(parser.errors[:field_116]).to be_blank
+              expect(parser.errors[:field_154]).to be_blank
+              expect(parser.errors[:field_155]).to be_blank
+              expect(parser.errors[:field_156]).to be_blank
+            end
+          end
+
+          context "and field_116 is invalid" do
+            let(:attributes) { renewal_attributes.merge({ field_116: 5 }) } # PRP option
+
+            it "adds errors to all referral fields" do
+              parser.valid?
+              expect(parser.errors[:field_116]).to be_present
+              expect(parser.errors[:field_154]).to be_present
+              expect(parser.errors[:field_155]).to be_present
+              expect(parser.errors[:field_156]).to be_present
+            end
+          end
+
+          context "and field_116 is blank" do
+            let(:attributes) { renewal_attributes.merge({ field_116: nil }) }
+
+            it "adds errors to all referral fields" do
+              parser.valid?
+              expect(parser.errors[:field_116]).to be_present
+              expect(parser.errors[:field_154]).to be_present
+              expect(parser.errors[:field_155]).to be_present
+              expect(parser.errors[:field_156]).to be_present
+            end
+          end
+
+          context "and other fields are given" do
+            let(:attributes) { renewal_attributes.merge({ field_116: 1, field_154: 5, field_155: 1, field_152: 1 }) }
+
+            it "adds errors to all referral fields" do
+              parser.valid?
+              expect(parser.errors[:field_116]).to be_present
+              expect(parser.errors[:field_154]).to be_present
+              expect(parser.errors[:field_155]).to be_present
+              expect(parser.errors[:field_156]).to be_present
+            end
+          end
+        end
+
+        context "and is renewal" do
+          let(:attributes) { org_attributes.merge({ field_7: 1, field_116: 1, field_154: 5, field_155: 1, field_156: 1 }) }
+
+          it "does not add an error for referral fields" do
+            parser.valid?
+            expect(parser.errors[:field_116]).to be_blank
+            expect(parser.errors[:field_154]).to be_blank
+            expect(parser.errors[:field_155]).to be_blank
+            expect(parser.errors[:field_156]).to be_blank
+          end
+        end
+      end
+
+      context "when org is PRP" do
+        let(:owning_org) { create(:organisation, :prp, :with_old_visible_id) }
+
+        let(:org_attributes) { { bulk_upload:, field_1: owning_org.old_visible_id } }
+
+        context "and not renewal" do
+          let(:renewal_attributes) { org_attributes.merge({ field_7: nil }) }
+
+          context "and field_154 is valid and does not expect an answer for field_155" do
+            let(:attributes) { renewal_attributes.merge({ field_154: 5 }) }
+
+            it "does not add an error" do
+              parser.valid?
+              expect(parser.errors[:field_116]).to be_blank
+              expect(parser.errors[:field_154]).to be_blank
+              expect(parser.errors[:field_155]).to be_blank
+              expect(parser.errors[:field_156]).to be_blank
+            end
+
+            context "and later fields are given" do
+              let(:attributes) { renewal_attributes.merge({ field_154: 5, field_155: 1, field_156: 1 }) }
+
+              it "adds errors to all referral fields" do
+                parser.valid?
+                expect(parser.errors[:field_116]).to be_present
+                expect(parser.errors[:field_154]).to be_present
+                expect(parser.errors[:field_155]).to be_present
+                expect(parser.errors[:field_156]).to be_present
+              end
+            end
+          end
+
+          context "and field_154 is invalid" do
+            let(:attributes) { renewal_attributes.merge({ field_154: 1 }) } # LA option
+
+            it "adds errors to all referral fields" do
+              parser.valid?
+              expect(parser.errors[:field_116]).to be_present
+              expect(parser.errors[:field_154]).to be_present
+              expect(parser.errors[:field_155]).to be_present
+              expect(parser.errors[:field_156]).to be_present
+            end
+          end
+
+          context "and field_154 is blank" do
+            let(:attributes) { renewal_attributes.merge({ field_154: nil }) }
+
+            it "adds errors to all referral fields" do
+              parser.valid?
+              expect(parser.errors[:field_116]).to be_present
+              expect(parser.errors[:field_154]).to be_present
+              expect(parser.errors[:field_155]).to be_present
+              expect(parser.errors[:field_156]).to be_present
+            end
+          end
+
+          context "and field_154 is valid and expects an answer for field_155" do
+            let(:field_154_attributes) { renewal_attributes.merge({ field_154: 6 }) }
+
+            context "and field_155 is valid and does not expect an answer for field_156" do
+              let(:attributes) { field_154_attributes.merge({ field_155: 2 }) }
+
+              it "does not add an error" do
+                parser.valid?
+                expect(parser.errors[:field_116]).to be_blank
+                expect(parser.errors[:field_154]).to be_blank
+                expect(parser.errors[:field_155]).to be_blank
+                expect(parser.errors[:field_156]).to be_blank
+              end
+
+              context "and later fields are given" do
+                let(:attributes) { field_154_attributes.merge({ field_155: 2, field_156: 1 }) }
+
+                it "adds errors to all referral fields" do
+                  parser.valid?
+                  expect(parser.errors[:field_116]).to be_present
+                  expect(parser.errors[:field_154]).to be_present
+                  expect(parser.errors[:field_155]).to be_present
+                  expect(parser.errors[:field_156]).to be_present
+                end
+              end
+            end
+
+            context "and field_155 is invalid" do
+              let(:attributes) { field_154_attributes.merge({ field_155: 5 }) } # needs field_154 to be 7
+
+              it "adds errors to all referral fields" do
+                parser.valid?
+                expect(parser.errors[:field_116]).to be_present
+                expect(parser.errors[:field_154]).to be_present
+                expect(parser.errors[:field_155]).to be_present
+                expect(parser.errors[:field_156]).to be_present
+              end
+            end
+
+            context "and field_155 is blank" do
+              let(:attributes) { field_154_attributes.merge({ field_155: nil }) }
+
+              it "adds errors to all referral fields" do
+                parser.valid?
+                expect(parser.errors[:field_116]).to be_present
+                expect(parser.errors[:field_154]).to be_present
+                expect(parser.errors[:field_155]).to be_present
+                expect(parser.errors[:field_156]).to be_present
+              end
+            end
+
+            context "and field_155 is valid and expects an answer for field_156" do
+              let(:field_155_attributes) { field_154_attributes.merge({ field_155: 1 }) }
+
+              context "and field_156 is valid" do
+                let(:attributes) { field_155_attributes.merge({ field_156: 1 }) }
+
+                it "does not add an error" do
+                  parser.valid?
+                  expect(parser.errors[:field_116]).to be_blank
+                  expect(parser.errors[:field_154]).to be_blank
+                  expect(parser.errors[:field_155]).to be_blank
+                  expect(parser.errors[:field_156]).to be_blank
+                end
+              end
+
+              context "and field_156 is invalid" do
+                let(:attributes) { field_155_attributes.merge({ field_156: 11 }) } # needs field_155 to be 7
+
+                it "adds errors to all referral fields" do
+                  parser.valid?
+                  expect(parser.errors[:field_116]).to be_present
+                  expect(parser.errors[:field_154]).to be_present
+                  expect(parser.errors[:field_155]).to be_present
+                  expect(parser.errors[:field_156]).to be_present
+                end
+              end
+
+              context "and field_156 is blank" do
+                let(:attributes) { field_155_attributes.merge({ field_156: nil }) }
+
+                it "adds errors to all referral fields" do
+                  parser.valid?
+                  expect(parser.errors[:field_116]).to be_present
+                  expect(parser.errors[:field_154]).to be_present
+                  expect(parser.errors[:field_155]).to be_present
+                  expect(parser.errors[:field_156]).to be_present
+                end
+              end
+            end
+          end
+        end
+
+        context "and is renewal" do
+          let(:attributes) { org_attributes.merge({ field_7: 1, field_116: 1, field_154: 5, field_155: 1, field_156: 1 }) }
+
+          it "does not add an error for referral fields" do
+            parser.valid?
+            expect(parser.errors[:field_116]).to be_blank
+            expect(parser.errors[:field_154]).to be_blank
+            expect(parser.errors[:field_155]).to be_blank
+            expect(parser.errors[:field_156]).to be_blank
+          end
+        end
+      end
+    end
 
     describe "fields 7, 8, 9 => startdate" do
       context "when any one of these fields is blank" do