Skip to content

Enabling export of multiple instances #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fredrodlima merged 8 commits into from
Apr 16, 2025
Merged

Enabling export of multiple instances #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
596c055
InstanceExport.ps1: Updating script to enable multi instances export
fredrodlima Apr 11, 2025
8c27244
README.md: Updating texts for instructions on how to run the script
fredrodlima Apr 11, 2025
fe30b37
Update README.md
fredrodlima Apr 12, 2025
3411cc1
InstanceExport: Removing $instance on Log Cmdlet
fredrodlima Apr 15, 2025
4bbbe3c
Update README.md
fredrodlima Apr 15, 2025
054acb3
InstanceExport: Extracting ExportInstanceData
fredrodlima Apr 15, 2025
6d9e34e
Merge remote-tracking branch 'origin/enabling-export-multiple-instanc…
fredrodlima Apr 15, 2025
21f02b0
InstanceExport: log start message sooner
bigdogwillfeed Apr 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
324 changes: 189 additions & 135 deletions InstanceExport/PowerShell/InstanceExport.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ Function CreateDirectoryIfDoesNotExist {
try {
New-Item -Path $directoryPath -ItemType "directory" | Out-Null
$msg = "$directoryPath created"
Log -msg $msg -displayMsg $msg -logLevel "info" -currentDate $currentDate
Log -msg $msg -displayMsg $msg -logLevel "info" -currentDate $currentDate
}
catch {
$msg = "Error trying to create $directoryPath directory"
Expand All @@ -121,7 +121,7 @@ Function GetPassword() {
return ConvertFrom-SecureString -SecureString $password -AsPlainText
}
catch {
Log -msg $_.Exception.Message -logLevel "displayInfo" -currentDate $currentDate
Log -msg $_.Exception.Message -logLevel "displayInfo" -currentDate $currentDate
}
# fallback to powershell 5
$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)
Expand All @@ -131,7 +131,7 @@ Function GetPassword() {
}
# we're probably on some non-windows environment... fall back to unsecured
$msg = "Warning: Unable to handle password securely. Falling back to plain text password"
Log -msg $msg -logLevel "displayInfo" -currentDate $currentDate
Log -msg $msg -logLevel "displayInfo" -currentDate $currentDate
return Read-Host "$msg. To continue, please enter the password again"
}

Expand Down Expand Up @@ -180,157 +180,211 @@ Function ServerHealtCheck($Uri) {
return $response
}


function Export {
<#
.Description
This function exports an given instancce and save it to your disk in the specified exportFilePath from a JSON manifest file
#>
$currentDate = Get-Date -Format "MM_dd_yyyy_HH_mm_ss"

$manifestFileExists = Test-Path -Path $manifestFilePath

if ($manifestFileExists) {
$manifestJson = Get-Content -Raw -Path $manifestFilePath | ConvertFrom-Json
if ($manifestJson.CreateDate) {
$currentDate = Get-Date $manifestJson.CreateDate -Format "MM_dd_yyyy_HH_mm_ss"
}
else {
$msg = "Error in reading manifest file"
$displayMsg = "An error occurred when reading manifest file"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Exit 1
}
Function ExportInstanceData() {

$message = "Starting exporting data for instance $instance"
Log -msg $message -displayMsg $message -logLevel "success" -currentDate $currentDate

try {
CreateDirectoryIfDoesNotExist -directoryPath $exportFilePath -currentDate $currentDate
try {
CreateDirectoryIfDoesNotExist -directoryPath $directoryPath -currentDate $currentDate
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Exit 1
}

try {
$accessTokenResponseModel = Login
}
catch {
Log -msg "An error occurred" -displayMsg "$($_.Exception.Response.StatusCode.value__): An error occurred when logging in" -logLevel "error" -currentDate $currentDate
Log -msg "Authorization error for Instance Export" -logLevel "error" -currentDate $currentDate
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate
Log -msg "Url: api/login" -logLevel "error" -currentDate $currentDate
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate
}
if ($accessTokenResponseModel) {

$accessToken = $accessTokenResponseModel.access_token

$header = @{
"authorization" = "Bearer $accessToken"
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Exit 1

$currentCategory = "";
$i = 0;
for (; $i -lt $manifestJson.Entries.Length; $i = $i + 1) {
$entry = $manifestJson.Entries[$i]
if ($currentCategory -ne $entry.Category) {
$currentCategory = $entry.Category;
$message = "Starting extraction of $($entry.Category) category."
Log -msg $message -displayMsg $message -logLevel "displayInfo" -currentDate $currentDate
}
$message = "Extracting data from $($entry.Url)"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$directoryPath = $exportFilePath + "/" + $instance + "/" + $entry.Path

try {
CreateDirectoryIfDoesNotExist -directoryPath $directoryPath -currentDate $currentDate
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}
$fileName = $entry.FileName.Split([IO.Path]::GetInvalidFileNameChars()) -join ''
$filePath = "$directoryPath/$fileName"
$fileAlreadyExists = Test-Path -Path $filePath -PathType Leaf
if (!$fileAlreadyExists -or $overwrite) {
$entryExportParameters = @{
Method = "GET"
Uri = "https://$($manifestJson.SubDomain).$($manifestJson.HostName)$($entry.Url)"
Headers = $header
ContentType = "application/json"
}

try {
if (!$entry.fileName.Contains(".json")) {
Invoke-RestMethod @entryExportParameters -OutFile $filePath | Out-Null
}
else {
$GetEntriesResponse = Invoke-RestMethod @entryExportParameters
$GetEntriesResponse | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath
}
}
catch {
$message = "Error occurred when extracting data from $($entry.Url)"
Log -msg "An error occurred" -displayMsg $message -logLevel "error" -currentDate $currentDate
Log -msg $message -logLevel "error" -currentDate $currentDate
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate
Log -msg "Url: $($entry.Url)" -logLevel "error" -currentDate $currentDate
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate
Log -msg "Exception: $($_.Exception)" -logLevel "error" -currentDate $currentDate
Write-Verbose $_.Exception
Write-Verbose $_.Exception.Response
if ($_.Exception.Response.StatusCode -eq 503) {
$message = "Lost connection to server. Waiting to restablish connection."
Write-ColorOutput red $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
do {
$response = ServerHealtCheck("https://$($manifestJson.SubDomain).$($manifestJson.HostName)/en/healthcheck");
} while ($response.Exception)
$message = "Server connection reestablished. Resuming export"
Write-ColorOutput green $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$i = $i - 1;
}
}
}
else {
$message = "Skipping $filePath because file already exists"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
}
}

try {
$accessTokenResponseModel = Login
Logout($header)
}
catch {
Log -msg "An error occurred" -displayMsg "$($_.Exception.Response.StatusCode.value__): An error occurred when logging in" -logLevel "error" -currentDate $currentDate
Log -msg "Authorization error for Instance Export" -logLevel "error" -currentDate $currentDate
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate
Log -msg "Url: api/login" -logLevel "error" -currentDate $currentDate
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate
$message = "Error occurred when logging out"
Log -msg $message -displayMsg $message -logLevel "error" -currentDate $currentDate
}

$msg = "Exporting data for instance $instance was succesful."
Log -msg $msg -displayMsg $msg -logLevel "success" -currentDate $currentDate
}
else {
$msg = "Exporting Instance run finished with errors."
$displayMsg = "Exporting Instance run was not successful. Check the log file for more details."
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}
}

if ($accessTokenResponseModel) {

$accessToken = $accessTokenResponseModel.access_token

$header = @{
"authorization" = "Bearer $accessToken"
function Export {
<#
.Description
This function exports an given instancce and save it to your disk in the specified exportFilePath from a JSON manifest file
#>
$currentDate = Get-Date -Format "MM_dd_yyyy_HH_mm_ss"

$manifestFilePathExists = Test-Path -Path $manifestFilePath

$target = Get-Item $manifestFilePath

if ($manifestFilePathExists) {
if (!$target.PSIsContainer) {
Copy link
Contributor Author

@fredrodlima fredrodlima Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This checks if the manifestFilePath is not a container (a file) and if yes it follows the previous script flow. Otherwise we follow the updated pattern (with a provided folder)

$manifestJson = Get-Content -Raw -Path $manifestFilePath | ConvertFrom-Json
$instance = $manifestJson.subdomain
Copy link
Contributor Author

@fredrodlima fredrodlima Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$instance now is using subdomain from json file


if ($manifestJson.CreateDate) {
$currentDate = Get-Date $manifestJson.CreateDate -Format "MM_dd_yyyy_HH_mm_ss"
}

$currentCategory = "";
$i = 0;
for (; $i -lt $manifestJson.Entries.Length; $i = $i + 1) {
$entry = $manifestJson.Entries[$i]
if ($currentCategory -ne $entry.Category) {
$currentCategory = $entry.Category;
$message = "Starting extraction of $($entry.Category) category."
Log -msg $message -displayMsg $message -logLevel "displayInfo" -currentDate $currentDate
}
$message = "Extracting data from $($entry.Url)"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$directoryPath = $exportFilePath + "/" + $entry.Path

try {
CreateDirectoryIfDoesNotExist -directoryPath $directoryPath -currentDate $currentDate
}
catch {
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}
$fileName = $entry.FileName.Split([IO.Path]::GetInvalidFileNameChars()) -join ''
$filePath = "$directoryPath/$fileName"
$fileAlreadyExists = Test-Path -Path $filePath -PathType Leaf
if (!$fileAlreadyExists -or $overwrite) {
$entryExportParameters = @{
Method = "GET"
Uri = "https://$($manifestJson.SubDomain).$($manifestJson.HostName)$($entry.Url)"
Headers = $header
ContentType = "application/json"
}

try {
if (!$entry.fileName.Contains(".json")) {
Invoke-RestMethod @entryExportParameters -OutFile $filePath | Out-Null
}
else {
$GetEntriesResponse = Invoke-RestMethod @entryExportParameters
$GetEntriesResponse | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath
}
}
catch {
$message = "Error occurred when extracting data from $($entry.Url)"
Log -msg "An error occurred" -displayMsg $message -logLevel "error" -currentDate $currentDate
Log -msg $message -logLevel "error" -currentDate $currentDate
Log -msg "StatusCode: $($_.Exception.Response.StatusCode.value__)" -logLevel "error" -currentDate $currentDate
Log -msg "Url: $($entry.Url)" -logLevel "error" -currentDate $currentDate
Log -msg "StatusDescription: $($_.Exception.Response.StatusDescription)" -logLevel "error" -currentDate $currentDate
Log -msg "Exception: $($_.Exception)" -logLevel "error" -currentDate $currentDate
Write-Verbose $_.Exception
Write-Verbose $_.Exception.Response
if ($_.Exception.Response.StatusCode -eq 503) {
$message = "Lost connection to server. Waiting to restablish connection."
Write-ColorOutput red $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
do {
$response = ServerHealtCheck("https://$($manifestJson.SubDomain).$($manifestJson.HostName)/en/healthcheck");

} while ($response.Exception)
$message = "Server connection reestablished. Resuming export"
Write-ColorOutput green $message
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
$i = $i - 1;
}
}
}
else {
$message = "Skipping $filePath because file already exists"
Log -msg $message -displayMsg $message -logLevel "info" -currentDate $currentDate
}
else {
$msg = "Error in reading manifest file"
$displayMsg = "An error occurred when reading manifest file"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Exit 1
}

$directoryPath = $exportFilePath + "/" + $instance
ExportInstanceData

$msg = "Exporting Instance run finished."
Log -msg $msg -displayMsg $msg -logLevel "success" -currentDate $currentDate

Log -msg $msg -displayMsg $msg -logLevel "success" -currentDate $currentDate
}
else {
$manifestFilePath = $manifestFilePath.TrimEnd('\') # Trim any trailing backslash if it exists

$manifestJsonFiles = Get-ChildItem -Path "$manifestFilePath\*" -Include "*.json"

try {
Logout($header)
CreateDirectoryIfDoesNotExist -directoryPath "$exportFilePath" -currentDate $currentDate
}
catch {
$message = "Error occurred when logging out"
Log -msg $message -displayMsg $message -logLevel "error" -currentDate $currentDate
$msg = "Error trying to create $directoryPath directory"
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
Exit 1
}

if ($manifestJsonFiles) {

foreach ($manifestFile in $manifestJsonFiles) {

$manifestFileExists = Test-Path -Path $manifestFile

if ($manifestFileExists) {
$manifestJson = Get-Content -Raw -Path $manifestFile | ConvertFrom-Json

$instance = $manifestJson.subdomain
Copy link
Contributor Author

@fredrodlima fredrodlima Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$instance now is using subdomain from json file


$directoryPath = $exportFilePath + "/" + $instance

ExportInstanceData
}
else {
$displayMsg = "An error occurred when looking for manifest file"
Log -msg "Error in finding manifest file" -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}
}

$msg = "Exporting Instance script run finished."
Log -msg $msg -displayMsg $msg -logLevel "success" -currentDate $currentDate
}
else {
$msg = "Manifest file path doesn't contain valid json file(s)."
Log -msg $msg -displayMsg $msg -logLevel "error" -currentDate $currentDate

$msg = "Exporting Instance script run finished with errors."
Copy link
Contributor Author

@fredrodlima fredrodlima Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improving a tiny bit the Log here so users are aware that an error has occurred in their script run

Log -msg $msg -displayMsg $msg -logLevel "error" -currentDate $currentDate
}
}
else {
$msg = "Exporting Instance run finished with errors."
$displayMsg = "Exporting Instance run was not successful. Check the log file for more details."
Log -msg $msg -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}

}
else {
try {
CreateDirectoryIfDoesNotExist -directoryPath $exportFilePath -currentDate $currentDate
}
catch {
$displayMsg = "An error occurred when trying to create a new directory"
Log -msg "Error trying to create $directoryPath directory" -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
}

$displayMsg = "An error occurred when looking for manifest file"
Log -msg "Error in finding manifest file" -displayMsg $displayMsg -logLevel "error" -currentDate $currentDate
$msg = "Manifest file path provided doesn't exist."
Log -msg $msg -displayMsg $msg -logLevel "error" -currentDate $currentDate
}
}
Export
Loading