Add end-to-end tests for GitHub Copilot plugin

AGENTS.md

@@ -0,0 +1,116 @@
+# Apache DevLake - AI Coding Agent Instructions
+
+## Project Overview
+Apache DevLake is a dev data platform that ingests data from DevOps tools (GitHub, GitLab, Jira, Jenkins, etc.), transforms it into standardized domain models, and enables metrics/dashboards via Grafana.
+
+## Architecture
+
+### Three-Layer Data Model
+1. **Raw Layer** (`_raw_*` tables): JSON data collected from APIs, stored for replay/debugging
+2. **Tool Layer** (`_tool_*` tables): Plugin-specific models extracted from raw data
+3. **Domain Layer** (standardized tables): Normalized models in [backend/core/models/domainlayer/](backend/core/models/domainlayer/) - CODE, TICKET, CICD, CODEREVIEW, CODEQUALITY, CROSS
+
+### Key Components
+- **backend/**: Go server + plugins (main codebase)
+- **backend/python/**: Python plugin framework via RPC
+- **config-ui/**: React frontend (TypeScript, Vite, Ant Design)
+- **grafana/**: Dashboard definitions
+
+## Plugin Development (Go)
+
+### Plugin Structure
+Each plugin in `backend/plugins/<name>/` follows this layout:
+```
+api/         # REST endpoints (connections, scopes, scope-configs)
+impl/        # Plugin implementation (implements core interfaces)
+models/      # Tool layer models + migrationscripts/
+tasks/       # Collectors, Extractors, Converters
+e2e/         # Integration tests with CSV fixtures
+```
+
+### Required Interfaces
+See [backend/plugins/gitlab/impl/impl.go](backend/plugins/gitlab/impl/impl.go) for reference:
+- `PluginMeta`: Name, Description, RootPkgPath
+- `PluginTask`: SubTaskMetas(), PrepareTaskData()
+- `PluginModel`: GetTablesInfo() - **must list all models or CI fails**
+- `PluginMigration`: MigrationScripts() for DB schema evolution
+- `PluginSource`: Connection(), Scope(), ScopeConfig()
+
+### Subtask Pattern (Collector → Extractor → Converter)
+```go
+// 1. Register subtask in tasks/register.go via init()
+func init() {
+    RegisterSubtaskMeta(&CollectIssuesMeta)
+}
+
+// 2. Define dependencies for execution order
+var CollectIssuesMeta = plugin.SubTaskMeta{
+    Name:         "Collect Issues",
+    Dependencies: []*plugin.SubTaskMeta{}, // or reference other metas
+}
+```
+
+### API Collectors
+- Use `helper.NewStatefulApiCollector` for incremental collection with time-based bookmarking
+- See [backend/plugins/gitlab/tasks/issue_collector.go](backend/plugins/gitlab/tasks/issue_collector.go)
+
+### Migration Scripts
+- Located in `models/migrationscripts/`
+- Register all scripts in `register.go`'s `All()` function
+- Version format: `YYYYMMDD_description.go`
+
+## Build & Development Commands
+
+```bash
+# From repo root
+make dep              # Install Go + Python dependencies
+make build            # Build plugins + server
+make dev              # Build + run server
+make godev            # Go-only dev (no Python plugins)
+make unit-test        # Run all unit tests
+make e2e-test         # Run E2E tests
+
+# From backend/
+make swag             # Regenerate Swagger docs (required after API changes)
+make lint             # Run golangci-lint
+```
+
+### Running Locally
+```bash
+docker-compose -f docker-compose-dev.yml up mysql grafana  # Start deps
+make dev                                                     # Run server on :8080
+cd config-ui && yarn && yarn start                          # UI on :4000
+```
+
+## Testing
+
+### Unit Tests
+Place `*_test.go` files alongside source. Use mocks from `backend/mocks/`.
+
+### E2E Tests for Plugins
+Use CSV fixtures in `e2e/` directory. See [backend/test/helper/](backend/test/helper/) for the Go test client that can spin up an in-memory DevLake instance.
+
+### Integration Testing
+```go
+helper.ConnectLocalServer(t, &helper.LocalClientConfig{
+    ServerPort:   8080,
+    DbURL:        "mysql://merico:merico@127.0.0.1:3306/lake",
+    CreateServer: true,
+    Plugins:      []plugin.PluginMeta{gitlab.Gitlab{}},
+})
+```
+
+## Python Plugins
+Located in `backend/python/plugins/`. Use Poetry for dependencies. See [backend/python/README.md](backend/python/README.md).
+
+## Code Conventions
+- Tool model table names: `_tool_<plugin>_<entity>` (e.g., `_tool_gitlab_issues`)
+- Domain model IDs: Use `didgen.NewDomainIdGenerator` for consistent cross-plugin IDs
+- All plugins must be independent - no cross-plugin imports
+- Apache 2.0 license header required on all source files
+
+## Common Pitfalls
+- Forgetting to add models to `GetTablesInfo()` fails `plugins/table_info_test.go`
+- Migration scripts must be added to `All()` in `register.go`
+- API changes require running `make swag` to update Swagger docs
+- Python plugins require `libgit2` for gitextractor functionality

backend/plugins/q_dev/e2e/README.md

@@ -0,0 +1,35 @@
+# Q Developer Plugin E2E Tests
+
+This directory contains end-to-end tests for the Q Developer plugin.
+
+## Running the Tests
+
+E2E tests require a MySQL database to run. You can set it up using the docker-compose file in the project root:
+
+```bash
+# From the project root directory
+docker-compose -f docker-compose-dev.yml up mysql
+
+# Set the E2E_DB_URL environment variable
+export E2E_DB_URL="mysql://merico:merico@127.0.0.1:3306/lake_test?charset=utf8mb4&parseTime=True&loc=UTC"
+
+# Create the test database
+mysql -h 127.0.0.1 -P 3306 -u root -p -e "CREATE DATABASE IF NOT EXISTS lake_test; GRANT ALL ON lake_test.* TO 'merico'@'%';"
+
+# Run the tests from the plugin directory
+cd backend/plugins/q_dev
+go test -v ./e2e/...
+```
+
+## Test Structure
+
+The e2e tests follow the standard DevLake e2e pattern:
+
+- `raw_tables/` - Contains raw test data (CSV files simulating S3 data)
+- `snapshot_tables/` - Contains expected output data for verification
+- `*_test.go` - Test files that verify the data transformation
+
+## Test Coverage
+
+- **TestQDevS3FileMeta** - Tests the S3 file metadata model
+- **TestQDevUserData** - Tests the user data model and CSV parsing

backend/plugins/q_dev/e2e/raw_tables/test_user_data.csv

@@ -0,0 +1,4 @@
+UserId,Date,CodeReview_FindingsCount,Inline_AcceptanceCount,Inline_SuggestionsCount,Chat_AICodeLines,Chat_MessagesInteracted,Chat_MessagesSent,CodeFix_AcceptanceEventCount,CodeFix_AcceptedLines,CodeFix_GeneratedLines,CodeFix_GenerationEventCount,InlineChat_AcceptanceEventCount,InlineChat_AcceptedLineAdditions,InlineChat_TotalEventCount
+user-001,2025-06-23,5,10,20,100,15,20,3,50,75,5,8,120,25
+user-002,2025-06-23,3,8,15,80,10,12,2,30,45,3,5,90,18
+user-001,2025-06-24,7,12,25,120,18,22,4,60,85,6,10,150,30

backend/plugins/q_dev/e2e/snapshot_tables/_tool_q_dev_s3_file_meta.csv

@@ -0,0 +1,2 @@
+connection_id,file_name,s3_path,scope_id,processed
+1,test_user_data.csv,test/2025/06/test_user_data.csv,test/2025/06,0

backend/plugins/q_dev/e2e/snapshot_tables/_tool_q_dev_user_data.csv

@@ -0,0 +1,4 @@
+id,connection_id,user_id,date,display_name,scope_id,code_review_findings_count,code_review_succeeded_event_count,inline_chat_acceptance_event_count,inline_chat_accepted_line_additions,inline_chat_accepted_line_deletions,inline_chat_dismissal_event_count,inline_chat_dismissed_line_additions,inline_chat_dismissed_line_deletions,inline_chat_rejected_line_additions,inline_chat_rejected_line_deletions,inline_chat_rejection_event_count,inline_chat_total_event_count,inline_ai_code_lines,inline_acceptance_count,inline_suggestions_count,chat_ai_code_lines,chat_messages_interacted,chat_messages_sent,code_fix_acceptance_event_count,code_fix_accepted_lines,code_fix_generated_lines,code_fix_generation_event_count,code_review_failed_event_count,dev_acceptance_event_count,dev_accepted_lines,dev_generated_lines,dev_generation_event_count,doc_generation_accepted_file_updates,doc_generation_accepted_files_creations,doc_generation_accepted_line_additions,doc_generation_accepted_line_updates,doc_generation_event_count,doc_generation_rejected_file_creations,doc_generation_rejected_file_updates,doc_generation_rejected_line_additions,doc_generation_rejected_line_updates,test_generation_accepted_lines,test_generation_accepted_tests,test_generation_event_count,test_generation_generated_lines,test_generation_generated_tests,transformation_event_count,transformation_lines_generated,transformation_lines_ingested
+1,1,user-001,2025-06-23T00:00:00.000+00:00,user-001,test/2025/06,5,0,8,120,0,0,0,0,0,0,0,25,0,10,20,100,15,20,3,50,75,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+2,1,user-002,2025-06-23T00:00:00.000+00:00,user-002,test/2025/06,3,0,5,90,0,0,0,0,0,0,0,18,0,8,15,80,10,12,2,30,45,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+3,1,user-001,2025-06-24T00:00:00.000+00:00,user-001,test/2025/06,7,0,10,150,0,0,0,0,0,0,0,30,0,12,25,120,18,22,4,60,85,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

backend/plugins/q_dev/e2e/user_data_test.go

@@ -0,0 +1,72 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package e2e
+
+import (
+	"testing"
+
+	"github.com/apache/incubator-devlake/core/models/common"
+	"github.com/apache/incubator-devlake/helpers/e2ehelper"
+	"github.com/apache/incubator-devlake/plugins/q_dev/impl"
+	"github.com/apache/incubator-devlake/plugins/q_dev/models"
+)
+
+// TestQDevS3FileMeta tests the S3 file metadata model
+func TestQDevS3FileMeta(t *testing.T) {
+	var plugin impl.QDev
+	dataflowTester := e2ehelper.NewDataFlowTester(t, "q_dev", plugin)
+
+	// Flush tables before testing
+	dataflowTester.FlushTabler(&models.QDevS3FileMeta{})
+
+	// Import CSV data into the tool table
+	dataflowTester.ImportCsvIntoTabler("./snapshot_tables/_tool_q_dev_s3_file_meta.csv", &models.QDevS3FileMeta{})
+
+	// Verify the file meta data
+	dataflowTester.VerifyTableWithOptions(
+		models.QDevS3FileMeta{},
+		e2ehelper.TableOptions{
+			CSVRelPath:  "./snapshot_tables/_tool_q_dev_s3_file_meta.csv",
+			IgnoreTypes: []interface{}{common.NoPKModel{}},
+			IgnoreFields: []string{
+				"processed_time",
+			},
+		},
+	)
+}
+
+// TestQDevUserData tests the user data model
+func TestQDevUserData(t *testing.T) {
+	var plugin impl.QDev
+	dataflowTester := e2ehelper.NewDataFlowTester(t, "q_dev", plugin)
+
+	// Flush tables before testing
+	dataflowTester.FlushTabler(&models.QDevUserData{})
+
+	// Import CSV data into the tool table
+	dataflowTester.ImportCsvIntoTabler("./snapshot_tables/_tool_q_dev_user_data.csv", &models.QDevUserData{})
+
+	// Verify the user data - comparing only the fields that exist in our CSV
+	dataflowTester.VerifyTableWithOptions(
+		models.QDevUserData{},
+		e2ehelper.TableOptions{
+			CSVRelPath:  "./snapshot_tables/_tool_q_dev_user_data.csv",
+			IgnoreTypes: []interface{}{common.Model{}},
+		},
+	)
+}