@ -6,7 +6,6 @@ If your household enters a new social housing tenancy or purchases a social hous
The information is provided via ‘<%= t('service_name') %>’, a service funded and managed by MHCLG. It collects information on the tenants or residents, tenancy or sale, and the dwelling itself. Some of this data is personal and sensitive, so MHCLG is responsible for ensuring it’s processed in line with data protection legislation.
The information is provided via ‘<%= t('service_name') %>’, a service funded and managed by MHCLG. It collects information on the tenants or residents, tenancy or sale, and the dwelling itself. Some of this data is personal and sensitive, so MHCLG is responsible for ensuring it’s processed in line with data protection legislation.
## Why do we share this information?
## Why do we share this information?
Information collected via CORE is shared with other government departments and agencies. It’s shared with the Greater London Authority and the Regulator of Social Housing. Data providers can also access data for their organisations via CORE. Data is only shared for research and statistical purposes.
Information collected via CORE is shared with other government departments and agencies. It’s shared with the Greater London Authority and the Regulator of Social Housing. Data providers can also access data for their organisations via CORE. Data is only shared for research and statistical purposes.
@ -27,16 +26,16 @@ Information collected via CORE relates to your tenancy, the dwelling you are liv
Collected data will be held for as long as necessary for research and statistical purposes. When no longer needed, data will be deleted in a safe manner. We’re aware some collected data is particularly sensitive. For example:
Collected data will be held for as long as necessary for research and statistical purposes. When no longer needed, data will be deleted in a safe manner. We’re aware some collected data is particularly sensitive. For example:
* ethnic group
- ethnic group
* if previous tenure is a hospital, prison or approved probation hostel support
- if previous tenure is a hospital, prison or approved probation hostel support
* if household left last settled home because discharged from prison, a long stay hospital or other institution
- if household left last settled home because discharged from prison, a long stay hospital or other institution
* if referral source is probation or prison, youth offending or community mental health team, or health service
- if referral source is probation or prison, youth offending or community mental health team, or health service
MHCLG publishes data annually, in aggregate form, as part of a report and complementary tables.
MHCLG publishes data annually, in aggregate form, as part of a report and complementary tables.
* For annual lettings data, visit: [https://www.gov.uk/government/collections/rents-lettings-and-tenancies](https://www.gov.uk/government/collections/rents-lettings-and-tenancies)
- For annual lettings data, visit: [https://www.gov.uk/government/collections/rents-lettings-and-tenancies](https://www.gov.uk/government/collections/rents-lettings-and-tenancies)
* For annual sales data, visit: [https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers](https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers)
- For annual sales data, visit: [https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers](https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers)
Detail-level data is anonymised and protected, minimising identification risk. It's held with the UK Data Archive.
Detail-level data is anonymised and protected, minimising identification risk. It's held with the UK Data Archive.
- With [ESBuild](https://esbuild.github.io/) - fast and does some transpiling but [doesn't support ES5/IE11](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/203)
- With [ESBuild](https://esbuild.github.io/) - fast and does some transpiling but [doesn't support ES5/IE11](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/203)
@ -12,6 +12,7 @@ There might be short extensions to the deadline, so shortly after the last day t
Also, if incorrect data is found during QA process, data providers might be asked to correct it. Once the data has been through its first QA processes and is as present and correct as possible, the ability to edit and delete logs is closed. This is typically in late summer/autumn, but it depends on the statistical analysis.
Also, if incorrect data is found during QA process, data providers might be asked to correct it. Once the data has been through its first QA processes and is as present and correct as possible, the ability to edit and delete logs is closed. This is typically in late summer/autumn, but it depends on the statistical analysis.
To accommodate the different end dates, we will now store 3 different dates on the form definition:
To accommodate the different end dates, we will now store 3 different dates on the form definition:
- Submission deadline (submission_deadline) - this is the date displayed at the top of a completed log in lettings and sales - "You can review and make changes to this log until 9 June 2024.". Nothing happens on this date
- Submission deadline (submission_deadline) - this is the date displayed at the top of a completed log in lettings and sales - "You can review and make changes to this log until 9 June 2024.". Nothing happens on this date
- New logs end date (new_logs_end_date) - no new logs for that collection year can be submitted, but logs can be edited
- New logs end date (new_logs_end_date) - no new logs for that collection year can be submitted, but logs can be edited
- Edit and delete logs end date (edit_end_date) - logs can no longer be edited or deleted. Completed logs can still be viewed. Materials / references to the collection year are removed.
- Edit and delete logs end date (edit_end_date) - logs can no longer be edited or deleted. Completed logs can still be viewed. Materials / references to the collection year are removed.
In order to use the app as an API, you will need to configure requests to the API as so:
In order to use the app as an API, you will need to configure requests to the API as so:
* Configure your request with Basic Auth. Set the username to be the same as `API_USER` and password as the `API_KEY` (`API_USER` and `API_KEY` are environment variables that should be set for the application)
- Configure your request with Basic Auth. Set the username to be the same as `API_USER` and password as the `API_KEY` (`API_USER` and `API_KEY` are environment variables that should be set for the application)
* Check the endpoint you are calling is an action that is `create`, `show` or `update`
- Check the endpoint you are calling is an action that is `create`, `show` or `update`
* Check you are setting the following request headers:
- Check you are setting the following request headers:
*`Content-Type = application/json`
-`Content-Type = application/json`
* `Action = application/json` N.B. If you use `*/*` instead, the request won't be recognised as an API request`
- `Action = application/json` N.B. If you use `*/*` instead, the request won't be recognised as an API request`
Currently only the logs controller is configured to accept and authenticate API requests, when the above API environment variables are set.
Currently only the logs controller is configured to accept and authenticate API requests, when the above API environment variables are set.
Currently, there are four environments with infrastructure:
Currently, there are four environments with infrastructure:
- Meta
- Meta
- Development (Review Apps)
- Development (Review Apps)
- Staging
- Staging
- Production
- Production
### Meta
### Meta
This holds the Terraform “backend” and the ECR(s).
This holds the Terraform “backend” and the ECR(s).
The Terraform “backend” consists of:
The Terraform “backend” consists of:
- S3 buckets - for storing Terraform state files. One for all non-production environments (including the meta environment itself), and another just for production.
- S3 buckets - for storing Terraform state files. One for all non-production environments (including the meta environment itself), and another just for production.
- DynamoDB - for managing access and locking of all state files.
- DynamoDB - for managing access and locking of all state files.
The ECR(s) are:
The ECR(s) are:
- core - holds the application Docker images.
- core - holds the application Docker images.
- db-migration - holds the Docker images curated to help migrate a DB from PaaS to AWS.
- db-migration - holds the Docker images curated to help migrate a DB from PaaS to AWS.
- s3-migration - holds the Docker images curated to help migrate S3 files from PaaS to AWS.
- s3-migration - holds the Docker images curated to help migrate S3 files from PaaS to AWS.
N.B. the migration ECRs may or may not be present, depending on if the Terraform has been configured to create migration infrastructure. The migration infrastructure is only used to help migrate the DB and S3 from PaaS to AWS, so is usually therefore only temporarily present.
N.B. the migration ECRs may or may not be present, depending on if the Terraform has been configured to create migration infrastructure. The migration infrastructure is only used to help migrate the DB and S3 from PaaS to AWS, so is usually therefore only temporarily present.
### Development / Staging / Production
### Development / Staging / Production
These are the main environments holding the “application” infrastructure.
These are the main environments holding the “application” infrastructure.
Though not exhaustive, each of them will generally contain the following key components:
Though not exhaustive, each of them will generally contain the following key components:
- ECS Fargate cluster
- ECS Fargate cluster
- RDS (PostgreSQL database)
- RDS (PostgreSQL database)
- ElastiCache (Redis data store)
- ElastiCache (Redis data store)
@ -43,6 +49,7 @@ Though not exhaustive, each of them will generally contain the following key com
- WAF (Firewall)
- WAF (Firewall)
### Development / Review Apps
### Development / Review Apps
The development environment is used for Review Apps, and has some infrastructure that is created per-review-app and some that is shared by all apps.
The development environment is used for Review Apps, and has some infrastructure that is created per-review-app and some that is shared by all apps.
In general, each review app has its own ECS Fargate cluster and Redis instances (plus any infrastructure to enable this), while the rest is shared.
In general, each review app has its own ECS Fargate cluster and Redis instances (plus any infrastructure to enable this), while the rest is shared.
@ -64,7 +71,6 @@ To deploy you need to:
6. Post success message on Slack.
6. Post success message on Slack.
7. Tag tickets as ‘Released’ and move tickets to done on JIRA.
7. Tag tickets as ‘Released’ and move tickets to done on JIRA.
## CI/CD
## CI/CD
When a commit is made to `main` the following GitHub action jobs are triggered:
When a commit is made to `main` the following GitHub action jobs are triggered:
@ -95,6 +101,7 @@ One reason a review app deployment might fail is that it is attempting to run mi
## Destroying/recreating infrastructure
## Destroying/recreating infrastructure
Things to watch out for when destroying/creating infra:
Things to watch out for when destroying/creating infra:
- All resources
- All resources
- The lifecycle meta-argument prevent_destroy will stop you destroying things. Best to set this to false before trying to destroy!
- The lifecycle meta-argument prevent_destroy will stop you destroying things. Best to set this to false before trying to destroy!
- AWS CloudWatch (for general application / infrastructure logging)
- AWS CloudWatch (for general application / infrastructure logging)
- Sentry (for application error logging)
- Sentry (for application error logging)
### CloudWatch
### CloudWatch
The CloudWatch service can be accessed from the AWS Console. You should authenticate onto the infrastructure environment whose logs you want to check.
The CloudWatch service can be accessed from the AWS Console. You should authenticate onto the infrastructure environment whose logs you want to check.
From CloudWatch, navigate to the desired log group (e.g. for the app task running on ECS) and open the desired log stream, in order to read its log “events”.
From CloudWatch, navigate to the desired log group (e.g. for the app task running on ECS) and open the desired log stream, in order to read its log “events”.
Alternatively, you can also navigate to a specific AWS service / resource in question (e.g. ECS tasks), selecting the instance of interest (e.g. a specific ECS task), and finding the “logs” tab (or similar) to view the log “events”.
Alternatively, you can also navigate to a specific AWS service / resource in question (e.g. ECS tasks), selecting the instance of interest (e.g. a specific ECS task), and finding the “logs” tab (or similar) to view the log “events”.
### Sentry
### Sentry
To access Sentry, ensure you have been added to the MHCLG account.
To access Sentry, ensure you have been added to the MHCLG account.
Generally error logs in Sentry will also be present somewhere in the CloudWatch logs, but they will be easier to assess here (e.g. number of occurrences over a time period). The logs in Sentry are created by the application when it makes Rails.logger.error calls.
Generally error logs in Sentry will also be present somewhere in the CloudWatch logs, but they will be easier to assess here (e.g. number of occurrences over a time period). The logs in Sentry are created by the application when it makes Rails.logger.error calls.
## Debugging
## Debugging
### Application infrastructure
### Application infrastructure
For debugging / investigating infrastructure issues you can use the AWS CloudWatch automatic dashboards. (e.g. is there a lack of physical space on the database, how long has the ECS had very high compute usage for etc.)
For debugging / investigating infrastructure issues you can use the AWS CloudWatch automatic dashboards. (e.g. is there a lack of physical space on the database, how long has the ECS had very high compute usage for etc.)
They can be found in the CloudWatch service on AWS console, by going to dashboards → automatic dashboards, and selecting the desired dashboard (e.g. Elastic Container Service).
They can be found in the CloudWatch service on AWS console, by going to dashboards → automatic dashboards, and selecting the desired dashboard (e.g. Elastic Container Service).
Alternatively, you can also navigate to the AWS resource in question (e.g. RDS database), selecting the instance of interest, and selecting the “monitoring” / ”metrics” tab (or similar), as this can provide alternate useful information also.
Alternatively, you can also navigate to the AWS resource in question (e.g. RDS database), selecting the instance of interest, and selecting the “monitoring” / ”metrics” tab (or similar), as this can provide alternate useful information also.
### Exec into a container
### Exec into a container
You can open a terminal directly on a running container / app, in order to run some commands that may help with debugging an issue.
You can open a terminal directly on a running container / app, in order to run some commands that may help with debugging an issue.
To do this, you will need to “exec” into the container.
To do this, you will need to “exec” into the container.
#### Prerequisites
#### Prerequisites
- AWS CLI
- AWS CLI
- AWS Session manager plugin Install the Session Manager plugin for the AWS CLI - AWS Systems Manager
- AWS Session manager plugin Install the Session Manager plugin for the AWS CLI - AWS Systems Manager
- AWS access
- AWS access
#### Accessing the rails console
#### Accessing the rails console
Prerequisite:
Prerequisite:
Configure AWS auth following the [documentation in the infra repo](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data-infrastructure/blob/main/docs/development_setup.md). This also details how to enter a subshell with suitable AWS credentials.
Configure AWS auth following the [documentation in the infra repo](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data-infrastructure/blob/main/docs/development_setup.md). This also details how to enter a subshell with suitable AWS credentials.
You can then use `aws ecs execute-command --cluster "core-$env-app" --task <taskid> --interactive --command <command>` to run the relevant command on a specific task.
You can then use `aws ecs execute-command --cluster "core-$env-app" --task <taskid> --interactive --command <command>` to run the relevant command on a specific task.
### Database
### Database
In order to investigate or look more closely at the database, you can exec into a container as above, and use the rails console to query the database.
In order to investigate or look more closely at the database, you can exec into a container as above, and use the rails console to query the database.
## Factories for Lettings Log, Sales Log, Organisation, and User
## Factories for Lettings Log, Sales Log, Organisation, and User
Each of these factories has nested relationships and callbacks that ensure associated objects are created and linked properly. For instance, creating a `lettings_log` involves creating or associating with a `user`, which in turn is linked to an `organisation`, potentially leading to creating `organisation_rent_periods` and a `data_protection_confirmation`.
Each of these factories has nested relationships and callbacks that ensure associated objects are created and linked properly. For instance, creating a `lettings_log` involves creating or associating with a `user`, which in turn is linked to an `organisation`, potentially leading to creating `organisation_rent_periods` and a `data_protection_confirmation`.
This documentation outlines the objects that are created and/or persisted to the database when using FactoryBot to create or build models for LettingsLog, SalesLog, Organisation, and User. There are other factories, but they are simpler, less frequently used and don't have as much resource hierarchy.
This documentation outlines the objects that are created and/or persisted to the database when using FactoryBot to create or build models for LettingsLog, SalesLog, Organisation, and User. There are other factories, but they are simpler, less frequently used and don't have as much resource hierarchy.
### Lettings Log
### Lettings Log
Objects Created/Persisted:
Objects Created/Persisted:
- **User**: The `assigned_to` user is created.
- **User**: The `assigned_to` user is created.
- **Organisation**: The `assigned_to` user’s organisation created by `User` factory.
- **Organisation**: The `assigned_to` user’s organisation created by `User` factory.
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with `assigned_to` user as a `data_protection_officer`
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with `assigned_to` user as a `data_protection_officer`
- **OrganisationRentPeriod**: If `log.period` is present and the `managing_organisation` does not have an `OrganisationRentPeriod` for that period, a new `OrganisationRentPeriod` is created and associated with `managing_organisation`.
- **OrganisationRentPeriod**: If `log.period` is present and the `managing_organisation` does not have an `OrganisationRentPeriod` for that period, a new `OrganisationRentPeriod` is created and associated with `managing_organisation`.
Example Usage:
Example Usage:
```
```
let(:lettings_log) { create(:lettings_log) }
let(:lettings_log) { create(:lettings_log) }
```
```
### Sales Log
### Sales Log
Objects Created/Persisted:
Objects Created/Persisted:
- **User**: The `assigned_to` user is created.
- **User**: The `assigned_to` user is created.
- **Organisation**: The `assigned_to` user’s organisation created by `User` factory.
- **Organisation**: The `assigned_to` user’s organisation created by `User` factory.
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with `assigned_to` user as a `data_protection_officer`
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with `assigned_to` user as a `data_protection_officer`
Example Usage:
Example Usage:
```
```
let(:sales_log) { create(:sales_log) }
let(:sales_log) { create(:sales_log) }
```
```
### Organisation
### Organisation
Objects Created/Persisted:
Objects Created/Persisted:
- **OrganisationRentPeriod**: For each rent period in transient attribute `rent_periods`, an `OrganisationRentPeriod` is created.
- **OrganisationRentPeriod**: For each rent period in transient attribute `rent_periods`, an `OrganisationRentPeriod` is created.
- **DataProtectionConfirmation**: If `with_dsa` is `true` (default), a `DataProtectionConfirmation` is created with a `data_protection_officer`
- **DataProtectionConfirmation**: If `with_dsa` is `true` (default), a `DataProtectionConfirmation` is created with a `data_protection_officer`
- **User**: Data protection officer that signs the data protection confirmation
- **User**: Data protection officer that signs the data protection confirmation
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with this user as a `data_protection_officer`
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with this user as a `data_protection_officer`
Rachael Booth
2 years ago