Search and media indexing improvements by ethanrous · Pull Request #54 · ethanrous/weblens

ethanrous · 2026-02-08T03:10:05Z

No description provided.

Copilot

Pull request overview

This PR improves search and media indexing by adding HDIR-based scoring, refining file/share permissions, centralizing query-param driven search/timeline state, and modernizing related UI + API endpoints (including feature flags).

Changes:

Add Feature Flags API (/flags) and wire UI/admin tooling to toggle HDIR processing.
Introduce HDIR search scoring surfaced through API models and UI (e.g., hdirScore), plus related cache cleanup changes.
Refactor file browser search/navigation to use URL query params, add regex/recursive search, and improve shared-file navigation/highlighting.

Reviewed changes

Copilot reviewed 141 out of 156 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
weblens-vue/weblens-nuxt/types/wlError.ts	Adds a concrete `WLError` class to normalize Nuxt/Axios errors.
weblens-vue/weblens-nuxt/types/weblensShare.ts	Adds `clone()` to force reactive updates after mutating share state.
weblens-vue/weblens-nuxt/types/weblensMedia.ts	Removes recognition tags and adds `hdirScore` field.
weblens-vue/weblens-nuxt/types/weblensFile.ts	Adds permissions + shared navigation helpers; updates URL helpers and GoTo logic.
weblens-vue/weblens-nuxt/types/user.ts	Renames enum variants to uppercase (e.g., `ADMIN`).
weblens-vue/weblens-nuxt/types/table.ts	Adds `Text` column type and tightens `TableColumn` typing.
weblens-vue/weblens-nuxt/stores/location.ts	Centralizes search/timeline in URL params; adds highlight hash handling and redirects.
weblens-vue/weblens-nuxt/pages/settings/users.vue	Fixes `v-model` usage and updates permission checks to new enum names.
weblens-vue/weblens-nuxt/pages/settings/dev.vue	Adds HDIR feature-flag toggle and updates drop-media call signature.
weblens-vue/weblens-nuxt/pages/settings.vue	Hides admin settings behind `ADMIN` permission gate.
weblens-vue/weblens-nuxt/pages/files/[fileID].vue	Passes parent-match override during searching; uses `fileFetchError`.
weblens-vue/weblens-nuxt/pages/files.vue	Refactors presentation rendering into slot-based content/info components.
weblens-vue/weblens-nuxt/package.json	Adds `httpxy`, moves `vite` alias into deps, removes `less`.
weblens-vue/weblens-nuxt/modules/webSocketProxy.js	Switches to dynamic import for `httpxy` and makes module setup async.
weblens-vue/weblens-nuxt/eslint.config.mjs	Enforces `vue/valid-v-model`.
weblens-vue/weblens-nuxt/components/organism/TaskProgress.vue	Tweaks task UI labeling (“Indexing Media”).
weblens-vue/weblens-nuxt/components/organism/ShareModal.vue	Updates user-select event, uses `Text` table cells, forces reactivity via cloning.
weblens-vue/weblens-nuxt/components/organism/Presentation.vue	Converts presentation to slot-based media/info + explicit next/previous callbacks.
weblens-vue/weblens-nuxt/components/organism/MediaTimeline.vue	Moves pagination/loading state to store; updates timeline toggling.
weblens-vue/weblens-nuxt/components/organism/FileScroller.vue	Adds error rendering + optional parent match filtering for searching.
weblens-vue/weblens-nuxt/components/organism/FileHeader.vue	Updates timeline toggle and improves “navigate up” logic for shares.
weblens-vue/weblens-nuxt/components/molecule/UserSearch.vue	Changes select event name + improves dropdown/select behavior.
weblens-vue/weblens-nuxt/components/molecule/Searchbar.vue	Uses location-store search; improves key hint UI and filter keyboard handling.
weblens-vue/weblens-nuxt/components/molecule/PresentationMediaInfo.vue	New component for media URL + geo display in presentation.
weblens-vue/weblens-nuxt/components/molecule/PresentationMediaContent.vue	New component for rendering video/pdf/image content in presentation.
weblens-vue/weblens-nuxt/components/molecule/PresentationFileInfo.vue	New component to show filename or navigate to source file by media ID.
weblens-vue/weblens-nuxt/components/molecule/NoResults.vue	Updates “no results” messaging to use centralized search + filters.
weblens-vue/weblens-nuxt/components/molecule/MediaSearchFilters.vue	Updates checkbox event name to `checked:changed`.
weblens-vue/weblens-nuxt/components/molecule/FileSquare.vue	Adds title tooltip and truncation/layout tweaks.
weblens-vue/weblens-nuxt/components/molecule/FileSortControls.vue	Updates sort-direction handling and options mapping; refactors option models.
weblens-vue/weblens-nuxt/components/molecule/FileSearchFilters.vue	Adds regex search toggle; updates checkbox event name.
weblens-vue/weblens-nuxt/components/molecule/FileCard.vue	Adds DOM id for hash scroll + highlight-on-load behavior.
weblens-vue/weblens-nuxt/components/molecule/ErrorCard.vue	Adds computed error messages for 403/404 and centers display.
weblens-vue/weblens-nuxt/components/molecule/ContextMenuActions.vue	Switches permission logic to `targetFile.Can*()` methods.
weblens-vue/weblens-nuxt/components/atom/WeblensOptions.vue	Removes chevron icon; tweaks Tailwind classes and import type usage.
weblens-vue/weblens-nuxt/components/atom/WeblensInput.vue	Moves key-hint rendering into consumer slot; changes clear button behavior.
weblens-vue/weblens-nuxt/components/atom/WeblensCheckbox.vue	Adds unique ids and changes emitted event name; updates SVG attributes.
weblens-vue/weblens-nuxt/components/atom/Table.vue	Adds `Text` cell support; updates checkbox event wiring.
weblens-vue/weblens-nuxt/components/atom/MediaImage.vue	Hooks image clicks into presentation store when clicks are enabled.
weblens-vue/weblens-nuxt/api/FileBrowserApi.ts	Removes folder-data function (now handled elsewhere) and trims imports.
services/reshape/share.go	Deduplicates permission conversion and forces `CanView = true`.
services/reshape/media.go	Adds batch options to include HDIR similarity scores.
services/reshape/file.go	Adds `permissions` into `FileInfo` payload.
services/notify/caster.go	Renames task start-time accessor in interface (`GetStartTime`).
services/media/import.go	Removes recognition tag initialization.
services/media/hdir.go	Reworks HDIR text search, adds text-encoding caching, improves errors.
services/media/agno/agno.go	Adds macOS LDFLAGS and debug timing logs; tweaks EXIF logging.
services/jobs/file_parser.go	Switches to feature flags, updates LeafMap signature usage, improves trace logs.
services/file/file_service_setup.go	Validates absolute paths and refactors filesystem load/caching logic.
services/file/file_service.go	Moves cache-file helpers into dedicated file.
services/file/cache_files.go	Adds cache file cleanup utility (`RemoveCacheFilesWithFilter`).
services/ctxservice/app_context.go	Adjusts cache capacity default.
services/auth/access_service.go	Clarifies comment wording for admin access.
scripts/start.bash	Refactors dev/start flow, enables cache/profiling defaults, adds HDIR env.
scripts/lib/mongo.bash	Switches mongo launch/cleanup to docker compose; changes defaults.
scripts/lib/meta.bash	Removes strict shell options.
scripts/lib/hdir.bash	Adds HDIR container helpers.
scripts/lib/docker-common.bash	Removes strict shell options.
scripts/lib/build.bash	Refactors build helpers; adds HDIR image build.
scripts/lib/all.bash	Refactors env defaults and sources new helper scripts.
scripts/lib/agno.bash	Adds Agno build/lifecycle helper.
scripts/install-deps.sh	Adds Vulkan-related runtime deps.
scripts/gogogadgetdocker.bash	Uses `dockerc`, switches to GHCR tagging, sources shared lib.
scripts/docker-compose.yml.example	Removes outdated example compose.
scripts/docker-compose.yml	Removes outdated compose.
scripts/devel.bash	Removes legacy dev script.
scripts/build-mongo.bash	Removes legacy mongo build script.
scripts/build-agno.bash	Adds standalone Agno build entrypoint.
routers/startup.go	Runs pprof with explicit mux/handlers.
routers/api/v1/websocket/websocket.go	Emits TaskCreated notification immediately for scan directory.
routers/api/v1/tower/rest_tower.go	Removes transaction wrapper and uses idempotent init calls.
routers/api/v1/tower/rest_admin.go	Replaces config endpoints with feature flags endpoints + swagger docs.
routers/api/v1/media/rest_media.go	Adds search scoring plumbing and safer cache cleanup; changes drop-media signature.
routers/api/v1/file/rest_share.go	Removes dead commented code for album shares.
routers/api/v1/file/rest_folders.go	Adds server-side sorting for folder children; disables media prefetch block.
routers/api/v1/file/rest_files.go	Adds regex/recursive search and server-side sorting; optimizes media fetch.
routers/api/v1/api.go	Adds long timeout for media batch route; renames `/config` -> `/flags`.
modules/structs/tower_params.go	Introduces `FeatureFlagParams` alias for flag updates.
modules/structs/media.go	Renames `fileIds` -> `fileIDs`, adds `hdirScore`, removes recognition tags.
modules/structs/file.go	Adds `permissions` to `FileInfo`.
modules/startup/register.go	Adds richer startup hook logging.
modules/slices/slices.go	Adds `Sort` and `Delete` wrappers.
modules/slices/natsort.go	Adds natural-sort compare helper.
modules/log/dev_logger.go	Prints extra structured fields in dev console logger.
modules/fs/abs_path.go	Improves error message for non-absolute path.
modules/config/config.go	Adjusts defaults and enforces absolute paths; adds `WEBLENS_HDIR_URI`.
models/task/worker_pool.go	Improves worker/task logging and avoids executing canceled tasks.
models/task/task.go	Removes `StartedAt` API and simplifies error log message.
models/media/media_test.go	Removes recognition tag tests.
models/media/cache_file.go	Adds cache filename parsing and error handling improvements.
models/file/file_test.go	Removes `RecursiveMap`/`LeafMap` tests.
models/file/file.go	Changes `LeafMap` signature to require ctx + child provider.
models/featureflags/feature_flag_test.go	Adds placeholder feature flags test file.
models/featureflags/feature_flag_model.go	Adds feature flags storage model + cache integration.
models/db/collection.go	Adds helpers for creating collection/search indexes.
models/config/config_test.go	Removes legacy config tests.
models/config/config_model.go	Removes legacy config model.
hdir/pyproject.toml	Simplifies metadata and relaxes python constraint.
hdir/open.main.py	Switches to open_clip flow, adds encode-text endpoint and env-based cache path.
hdir/main.py	Removes legacy CLIP-based server.
hdir/.python-version	Pins local python version.
go.work	Updates Go toolchain version.
go.mod	Updates Go toolchain version.
e2e/rest_tower_test.go	Updates E2E test to use FeatureFlags API.
docs/swagger.yaml	Updates API docs for flags/search params/file/media model changes.
docs/swagger.json	Updates API docs for flags/search params/file/media model changes.
docs/docs.go	Updates embedded swagger docs for flags/search params/file/media model changes.
docker/mongo.compose.yaml	Adds mongod+mongot stack with search and replica set init.
docker/mongo-core.env	Adds compose env for core tower role.
docker/hdir.Dockerfile	Rebuilds HDIR image using `uv` + preload step.
docker/dev.Dockerfile	Removes legacy dev Dockerfile.
docker/Dockerfile	Reworks build pipeline (pnpm caching, agno build, debian runtime, vulkan deps).
api/ts/package.json	Bumps axios/tsup/typescript versions.
api/ts/AllApi.ts	Adds FeatureFlags API factory to TS client.
api/test/api_feature_flags_test.go	Adds generated FeatureFlags API tests (skipped).
api/model_media_info.go	Updates generated MediaInfo: `fileIDs`, adds `hdirScore`, removes recognition tags.
api/model_file_info.go	Adds `permissions` to generated FileInfo.
api/model_config.go	Removes generated Config model.
api/model_bundle.go	Adds generated Bundle model for feature flags.
api/docs/MediaInfo.md	Updates MediaInfo docs for `fileIDs` + `hdirScore`.
api/docs/MediaAPI.md	Documents new DropMedia `username` query parameter.
api/docs/FolderAPI.md	Documents new folder sort query params.
api/docs/FilesAPI.md	Documents new search query params (sort/recursive/regex).
api/docs/FileInfo.md	Documents FileInfo permissions.
api/docs/FeatureFlagsAPI.md	Adds FeatureFlags API docs.
api/docs/Bundle.md	Adds Bundle model docs.
api/client.go	Switches from ConfigAPI to FeatureFlagsAPI in client struct.
api/api_media.go	Adds `username` query parameter builder for DropMedia.
api/api_folder.go	Adds `sortProp`/`sortOrder` query params for GetFolder.
api/api_files.go	Adds sort/recursive/regex query params for SearchByFilename.
api/api_feature_flags.go	Replaces generated config endpoints with feature flags endpoints.
api/README.md	Updates API list/models for FeatureFlags/Bundle.
api/.openapi-generator/FILES	Updates generated file list (removes config, adds feature flags).
agno	Updates submodule pointer.
Makefile	Adds `agno` target and renames docker publish target to `container`.
.dockerignore	Expands ignores and normalizes directories.
.air.toml	Watches `.a` and expands excluded directories.

Files not reviewed (2)

api/ts/pnpm-lock.yaml: Language not supported
weblens-vue/weblens-nuxt/pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

services/reshape/media.go

Copilot · 2026-02-08T03:14:05Z

models/db/collection.go

+func (c *ContextualizedCollection[T]) NewIndex(model mongo.IndexModel, opts ...*options.CreateIndexesOptions) error {
+	indexName := model.Options.Name
+	log.FromContext(c.ctx).Trace().Msgf("Creating index [%s] on collection [%s]", *indexName, c.collection.Name())
+
+	existingIndexes, err := c.collection.Indexes().List(c.ctx)
+	if err != nil {
+		return nil
+	}


Two issues here: (1) return nil silently swallows errors from listing indexes; this should return err. (2) model.Options or model.Options.Name can be nil, which will panic when dereferenced in the log statement—validate and return a clear error if name is missing.

Copilot · 2026-02-08T03:14:05Z

models/db/collection.go

+	existingIndexes, err := c.collection.SearchIndexes().List(c.ctx, &options.SearchIndexesOptions{Name: indexName, Type: indexType})
+	if err != nil {
+		return nil
+	}


Like NewIndex, NewSearchIndex is swallowing errors with return nil. This hides operational failures (permissions, connectivity, etc.). Return err so callers can react appropriately.

Copilot · 2026-02-08T03:14:05Z

routers/api/v1/file/rest_files.go

+		var less int
+
+		switch sortBy {
+		case "modified":
+			// less = files[i].ModTime().Before(files[j].ModTime())
+			less = int(f1.ModTime - f2.ModTime)
+		case "size":
+			less = int(f1.Size - f2.Size)
+		default: // Sort by name
+			path1, _ := fs.ParsePortable(f1.PortablePath)
+			path2, _ := fs.ParsePortable(f2.PortablePath)
+			less = slices.NatSortCompare(path1.Filename(), path2.Filename())
+		}
+
+		if sortDir == "desc" {
+			less = -less
+		}
+
+		return less


Swagger enums added in this PR use updatedAt (not modified), so sortProp=updatedAt will currently fall into the default (name) sort. Also, converting int64 diffs to int risks overflow and is not a strict comparator (ties/overflow can break ordering). Prefer explicit comparisons: return -1/0/1 based on <, >, and handle updatedAt as the key.

Suggested change

var less int

switch sortBy {

case "modified":

// less = files[i].ModTime().Before(files[j].ModTime())

less = int(f1.ModTime - f2.ModTime)

case "size":

less = int(f1.Size - f2.Size)

default: // Sort by name

path1, _ := fs.ParsePortable(f1.PortablePath)

path2, _ := fs.ParsePortable(f2.PortablePath)

less = slices.NatSortCompare(path1.Filename(), path2.Filename())

}

if sortDir == "desc" {

less = -less

}

return less

var cmp int

switch sortBy {

case "modified", "updatedAt":

if f1.ModTime < f2.ModTime {

cmp = -1

} else if f1.ModTime > f2.ModTime {

cmp = 1

} else {

cmp = 0

}

case "size":

if f1.Size < f2.Size {

cmp = -1

} else if f1.Size > f2.Size {

cmp = 1

} else {

cmp = 0

}

default: // Sort by name

path1, _ := fs.ParsePortable(f1.PortablePath)

path2, _ := fs.ParsePortable(f2.PortablePath)

cmp = slices.NatSortCompare(path1.Filename(), path2.Filename())

}

if sortDir == "desc" {

cmp = -cmp

}

return cmp

weblens-vue/weblens-nuxt/stores/location.ts

Copilot · 2026-02-08T03:14:08Z

modules/slices/natsort.go

+					if an != bn {
+						// #nosec G40
+						return int(an - bn)
+					}


Subtracting two uint64 values and converting to int can overflow (and also breaks strict comparator expectations). Return -1/0/1 using comparisons instead (e.g., if an < bn return -1; if an > bn return 1).

Suggested change

if an != bn {

// #nosec G40

return int(an - bn)

}

if an < bn {

return -1

}

if an > bn {

return 1

}

Copilot · 2026-02-08T03:14:08Z

models/featureflags/feature_flag_test.go

@@ -0,0 +1 @@
+package featureflags_test


Feature flags are newly introduced (defaulting, DB upsert, caching), but this test file is only a placeholder. Add unit tests covering: Default(), first-run GetFlags() behavior when the doc is missing (creates defaults), and SaveFlags() updating cache + persistence.

Copilot · 2026-02-08T03:14:09Z

models/media/cache_file.go

+// ParseCacheFileName parses the cache file name and returns the media ID, quality, and page number.
+func ParseCacheFileName(filename string) (mID string, quality Quality, pageNum int, err error) {
+	var qualityStr string
+
+	var pageNumStr string
+
+	matches := cacheFileFormatRegex.FindStringSubmatch(filename)
+	if len(matches) < 3 {
+		return "", "", 0, wlerrors.WithStack(errInvalidCacheFilename)
+	}


This introduces filename parsing and a strict regex, but there are no tests shown for it. Add unit tests for: low-res vs high-res, page suffix handling, invalid filenames, invalid page numbers, and unsupported qualities.

routers/startup.go

modules/structs/tower_params.go

The mongod container is exiting immediately with code 0 in CI before healthcheck runs. This adds logging to capture container logs, state, and Docker context info to diagnose the root cause. Also fixes ensure_weblens_net to use dockerc consistently instead of mixing bare docker and sudo docker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The launch_mongo output is piped through show_as_subtask which discards stdout when WEBLENS_VERBOSE=false (the default). Redirect all debug output to stderr so it appears in CI logs regardless. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The mongod container runs as user 'mongod' (uid 999) but the volume directories created by sudo docker are owned by root. Pre-create the _build/db/core/{mongod,mongot} directories with open permissions so the container can write to them. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use ${1:-} instead of $1 to safely handle the optional argument when running under set -euo pipefail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Make the host-side port in mongo.compose.yaml configurable via MONGO_HOST_PORT env var (defaults to 27017). launch_mongo now passes its port argument through a temp env file to docker compose, since sudo strips environment variables. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Use ctxservice.NewAppContext() instead of struct literal in test helper to properly initialize cacheLock (fixes nil pointer dereference in GetCache) - Show last 200 lines of log file when tests fail for better CI debugging visibility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace log.GlobalLogger().Error() + t.FailNow() pattern with require.NoError() across all e2e tests. The previous pattern wrote errors to the log file (invisible in CI output), making test failures impossible to diagnose. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The initTower hook wrapped InitializeCoreServer/InitializeBackupServer in db.WithTransaction, but these functions call startup.RunStartups which runs registerIndexes — and MongoDB's listIndexes command cannot execute inside a multi-document transaction. This matches the REST handler which already calls these functions without a transaction wrapper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ethanrous added 3 commits January 31, 2026 20:30

Search and media indexing improvements

9d0e3ef

major fixes for search and media indexing

c2b24bb

fix linter errors

4752a32

ethanrous marked this pull request as ready for review February 8, 2026 03:10

ethanrous requested a review from Copilot February 8, 2026 03:10

Copilot AI reviewed Feb 8, 2026

View reviewed changes

ethanrous and others added 10 commits February 7, 2026 22:47

fix copilot suggestions

bac5355

Bump agno

9c34103

Fix unbound variable error in build_agno with set -u

ffd326b

Use ${1:-} instead of $1 to safely handle the optional argument when running under set -euo pipefail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ethanrous merged commit 8dfaa4d into main Feb 8, 2026
3 checks passed

ethanrous deleted the search-and-media-indexing-improvements branch February 8, 2026 18:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Search and media indexing improvements#54

Search and media indexing improvements#54
ethanrous merged 13 commits intomainfrom
search-and-media-indexing-improvements

ethanrous commented Feb 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Feb 8, 2026

Uh oh!

Copilot AI Feb 8, 2026

Uh oh!

Copilot AI Feb 8, 2026

Uh oh!

Uh oh!

Copilot AI Feb 8, 2026

Uh oh!

Copilot AI Feb 8, 2026

Uh oh!

Copilot AI Feb 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

-					if an != bn {
-						// #nosec G40
-						return int(an - bn)
-					}
+					if an < bn {
+						return -1
+					}
+					if an > bn {
+						return 1
+					}

Conversation

ethanrous commented Feb 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant