Draft
Conversation
Remove stale resources cleanup since cleanup by tenant is executed afterwards anyways
executeMainMigrationPhases: migrate entities first and then relations
Dedup postgres functions are not used, remove sql files with them
deleteDocumentsByTenantId. Elasticsearch deletion is needed only for instance index
Rework entities selection for member upload and last_updated_at logic. Now it uses last_updated_date for sub-resources; local instances(with holdings,items) are fetched by tenantId and shared instances are fetched when having holdings with member tenantId. Required work:
On executeMainMigrationPhases (from staging to main) inserts set last_updated_date as constant "2000-01-01" for all entities that have it (only holdings don't have it) so it's not processed by background job.
last_updated_date='2000-01-01' to be used as filter on upload stage for sub-resources. Upload stage for instances have tenantId filter which is enough to get all required entities
Disable reindex status trigger (which blocks background job) on reindex start (only on member reindex) and enable on upload end. This will allow sub-resources for other tenant being processed. Notes: Since we set last_updated_date to "2000-01-01" - entities inserted during reindex will not be processed.
Create alternative reindex status trigger which will only handle statuses without blocking background job. Same trigger but without usage of locks. Enable on reindex start, disable on upload end (only on member reindex). This will allow reindex status updates on "processed count" changes while preserving ability to process resources from other member tenants by the background job
Remove staging table statistics
Rework target tenant id cache with spring cache
Add staging statuses/timings to reindex status
Add already running reindex validation on full reindex
Implements MSEARCH-1100
…e it's used for truncation
- Add deduplication for staging subjects/contributors/call-numbers/classifications - Remove redundant partitioning of staging tables Refs: MSEARCH-1107
# Conflicts: # NEWS.md # README.md # src/main/java/org/folio/search/SearchApplication.java # src/main/java/org/folio/search/integration/message/interceptor/PopulateInstanceBatchInterceptor.java # src/main/java/org/folio/search/service/reindex/ReindexOrchestrationService.java # src/main/java/org/folio/search/service/reindex/ReindexService.java # src/main/java/org/folio/search/service/reindex/jdbc/ContributorRepository.java # src/main/java/org/folio/search/service/reindex/jdbc/ItemRepository.java # src/main/java/org/folio/search/service/reindex/jdbc/MergeInstanceRepository.java # src/main/java/org/folio/search/service/reindex/jdbc/UploadInstanceRepository.java # src/test/java/org/folio/search/integration/KafkaMessageListenerTest.java # src/test/java/org/folio/search/service/reindex/jdbc/ReindexJdbcRepositoriesIT.java # src/test/java/org/folio/search/service/reindex/jdbc/UploadRangeRepositoriesIT.java
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Describe the purpose of this pull request. Why is this change necessary? What problem does it solve?
Approach
How does this change fulfill the purpose? Provide a high-level overview of the technical approach taken to address the problem.
Changes Checklist
Related Issues
MSEARCH-XXX
List any Jira issues related to this pull request.
Learning and Resources (if applicable)
Discuss any research conducted during the development of this pull request. Include links to relevant blog posts, patterns, libraries, or addons that were used to solve the problem.
Screenshots (if applicable)
If this pull request involves any visual changes or new features, consider including screenshots or GIFs to illustrate the changes.