CSC456 Operating Systems - Programming Project2 Author: John Akujobi Version: 1.0
A client-server network file system demonstrating inter-process communication (IPC) via TCP sockets, concurrent access control with reader-writer locks, and comprehensive file system operations.
# Create virtual environment (recommended on Linux)
python3 -m venv .venv
source .venv/bin/activate
# Install package
pip install -e .# Terminal 1
netfs server# Terminal 2
netfs client --user professorpwd # Show current directory
mkdir demo # Create directory
write demo/hello.txt "Hello!" # Create file with content
cat demo/hello.txt # Read file
ls demo # List directory
tree # Show directory tree
exit # Exit client- Quick Start
- Features Summary
- Installation Details
- Usage Guide
- Complete Command Reference
- System Architecture
- Technical Implementation
- Requirements Compliance
- Testing
- Demo Walkthrough
- Configuration Reference
- Project Structure
- Challenges and Solutions
- Troubleshooting
- Known Limitations
| Category | Count | Description |
|---|---|---|
| Commands | 18 | Full Linux-like file system commands |
| Unit Tests | 314 | Comprehensive component testing |
| Integration Tests | 25 | End-to-end client-server tests |
| Code Coverage | 83% | Exceeds 70% requirement |
- Multi-client support: Multiple concurrent connections via threaded server
- Reader-writer locks: Multiple simultaneous readers OR exclusive writer
- TCP/JSON protocol: Length-prefixed framing for reliable message boundaries
- Path security: Prevents directory traversal attacks
- Binary file support: Base64 encoding for non-text files
- Interactive shell: Command history, auto-completion, colored output
- Python 3.11 or higher
- pip (Python package manager)
- Linux recommended (tested on Ubuntu with Python 3.12.3)
Modern Linux distributions (Ubuntu 23.04+, Debian 12+, Fedora 38+) use "externally managed" Python, which blocks pip install to protect system packages. You'll see this error:
error: externally-managed-environment
This environment is externally managed
Solution: Always use a virtual environment. This is the recommended approach and avoids all system Python conflicts.
# 1. Navigate to project directory
cd NetFS
# 2. Create virtual environment
python3 -m venv .venv
# 3. Activate virtual environment
source .venv/bin/activate # Linux/macOS
# .venv\Scripts\activate # Windows (if needed)
# 4. Verify you're in the virtual environment
# Your prompt should show (.venv) at the beginning
which python # Should show: /path/to/NetFS/.venv/bin/python
# 5. Install NetFS package
pip install -e .
# 6. Verify installation
netfs --help| Action | Command |
|---|---|
| Create venv | python3 -m venv .venv |
| Activate (Linux/Mac) | source .venv/bin/activate |
| Activate (Windows) | .venv\Scripts\activate |
| Deactivate | deactivate |
| Check if active | Look for (.venv) in prompt |
| Install package | pip install -e . |
| Install with dev tools | pip install -e ".[dev]" |
# Note: Activate virtual environment first!
source .venv/bin/activate
make install # Install package
make install-dev # Install with development dependencies# Check netfs is available
netfs --help
# Check Python version
python --version # Should be 3.11+
# Check installed packages
pip list | grep -E "netfs|loguru|pydantic|click"# Default: listens on 0.0.0.0:9999
netfs server
# Custom port
netfs server --port 8888
# Custom config file
netfs server --config path/to/config.yamlServer Output:
2024-11-27 10:00:00 | INFO | NetFS Server Starting
2024-11-27 10:00:00 | INFO | Server address: 0.0.0.0:9999
2024-11-27 10:00:00 | INFO | Storage root: ./storage
# Connect to local server (default)
netfs client
# Connect with username
netfs client --user john
# Connect to remote server
netfs client --host 192.168.1.100 --port 9999 --user alice
# Disable colored output
netfs client --no-colorClient Prompt:
netfs:/$ _
The prompt shows your current directory. Use help to see all commands.
| Key | Action |
|---|---|
Tab |
Auto-complete commands and paths |
Up/Down |
Navigate command history |
Ctrl+C |
Cancel current input |
Ctrl+D |
Exit shell |
| Command | Syntax | Description | Example |
|---|---|---|---|
pwd |
pwd |
Print current working directory | pwd |
cd |
cd <path> |
Change directory | cd /documents |
help |
help |
Show available commands | help |
exit |
exit or quit |
Exit the shell | exit |
| Command | Syntax | Description | Example |
|---|---|---|---|
ls |
ls [path] |
List directory contents | ls or ls /docs |
mkdir |
mkdir <path> |
Create directory | mkdir projects |
rmdir |
rmdir [-r] <path> |
Remove directory | rmdir temp or rmdir -r old |
tree |
tree [path] [-d depth] |
Show directory tree | tree or tree / -d 3 |
find |
find <path> -name <pattern> |
Find files by pattern | find / -name *.txt |
| Command | Syntax | Description | Example |
|---|---|---|---|
cat |
cat <path> |
Display entire file | cat readme.txt |
head |
head [-n N] <path> |
Show first N lines (default: 10) | head -n 5 log.txt |
tail |
tail [-n N] <path> |
Show last N lines (default: 10) | tail -n 20 log.txt |
stat |
stat <path> |
Show file/directory metadata | stat data.json |
exists |
exists <path> |
Check if path exists | exists config.yaml |
| Command | Syntax | Description | Example |
|---|---|---|---|
write |
write <path> <content> |
Create/overwrite file | write note.txt Hello |
append |
append <path> <content> |
Append to file | append log.txt New entry |
touch |
touch <path> |
Create empty file or update timestamp | touch newfile.txt |
| Command | Syntax | Description | Example |
|---|---|---|---|
rm |
rm [-r] <path> |
Remove file or directory | rm old.txt or rm -r temp/ |
mv |
mv <src> <dst> |
Move or rename | mv old.txt new.txt |
cp |
cp [-r] <src> <dst> |
Copy file or directory | cp file.txt backup.txt |
┌─────────────────────────────────────────────────────────────────┐
│ NetFS Architecture │
└─────────────────────────────────────────────────────────────────┘
┌──────────────┐ ┌──────────────────┐
│ Client 1 │◄─────── TCP Socket ─────────►│ │
│ (Interactive │ (Port 9999) │ ServerListener │
│ Shell) │ │ │
└──────────────┘ │ ┌────────────┐ │
│ │ Handler │ │
┌──────────────┐ │ │ Threads │ │
│ Client 2 │◄─────── TCP Socket ─────────►│ │ (1/client) │ │
│ (Interactive │ │ └────────────┘ │
│ Shell) │ │ │
└──────────────┘ │ ┌────────────┐ │
│ │ Lock │ │
┌──────────────┐ │ │ Manager │ │
│ Client 3 │◄─────── TCP Socket ─────────►│ │ (RWLock) │ │
│ (Interactive │ │ └────────────┘ │
│ Shell) │ │ │
└──────────────┘ │ ┌────────────┐ │
│ │ Storage │ │
│ │ Manager │ │
│ └────────────┘ │
└────────┬─────────┘
│
▼
┌──────────────┐
│ ./storage/ │
│ (Filesystem) │
└──────────────┘
| Component | File | Responsibility |
|---|---|---|
| ServerListener | src/netfs/server/listener.py |
Accept connections, spawn handler threads |
| ConnectionHandler | src/netfs/server/handler.py |
Request/response loop for one client |
| CommandDispatcher | src/netfs/server/commands/__init__.py |
Route commands to handlers |
| StorageManager | src/netfs/server/storage.py |
File system operations with security |
| LockManager | src/netfs/server/locks.py |
Per-path reader-writer locks |
| ClientSession | src/netfs/server/session.py |
Track session state (cwd, locks) |
| ProtocolClient | src/netfs/client/protocol_client.py |
TCP connection and message framing |
| InteractiveShell | src/netfs/client/shell.py |
REPL with history and completion |
NetFS uses TCP sockets for reliable, ordered delivery of messages between clients and server.
Server Socket Setup (src/netfs/server/listener.py lines 110-132):
# Create TCP socket
self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Allow immediate address reuse (prevents "Address already in use")
self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# Bind and listen
self.server_socket.bind((self.config.host, self.config.port))
self.server_socket.listen(self.config.max_connections)Handling Partial Receives (src/netfs/common/protocol.py lines 76-100):
TCP is a stream protocol - recv() may return fewer bytes than requested. The recv_exactly() function handles this:
def recv_exactly(sock, num_bytes):
buffer = b""
while len(buffer) < num_bytes:
chunk = sock.recv(num_bytes - len(buffer))
if not chunk:
raise ConnectionError("Connection closed")
buffer += chunk
return bufferMessage Format:
┌────────────────┬─────────────────────────────────────┐
│ 4 bytes │ Variable length │
│ Length Prefix │ JSON Payload │
│ (Big-endian) │ │
└────────────────┴─────────────────────────────────────┘
Request Example:
{
"cmd": "write",
"username": "john",
"args": {
"path": "/documents/report.txt",
"content": "Hello, World!",
"binary": false
}
}Response Example:
{
"status": "success",
"message": "File written",
"data": {
"path": "/documents/report.txt",
"size": 13,
"created": true
},
"error_code": null
}Implementation: See src/netfs/common/protocol.py
Main Thread Handler Threads (daemon)
│ │
│ accept() │
│──────────►┌──────────────────┤
│ │ Thread 1 │
│ │ (Client A) │
│ └──────────────────┤
│ │
│ accept() │
│──────────►┌──────────────────┤
│ │ Thread 2 │
│ │ (Client B) │
│ └──────────────────┤
│ │
▼ ... more clients ▼
Each client connection spawns a daemon thread running ConnectionHandler.run(). Daemon threads automatically terminate when the main thread exits.
Implementation: See src/netfs/server/listener.py lines 159-174
NetFS implements custom reader-writer locks to allow:
- Multiple concurrent readers (e.g., multiple
catcommands on same file) - Exclusive writer access (e.g.,
writeblocks other operations)
Lock Behavior Visualization:
Time ────────────────────────────────────────────────────►
Reader 1: ████████████ (reading file)
Reader 2: ████████████ (reading same file - OK!)
Writer 1: ░░░░████████████ (waits, then writes)
Reader 3: ████ (waits for writer)
████ = Active ░░░░ = Waiting
RWLock Implementation (src/netfs/server/locks.py lines 12-97):
- Uses
threading.Conditionfor efficient wait/notify - Read-preferring strategy: readers proceed immediately if no writer
- Timeout support prevents indefinite blocking
LockManager (src/netfs/server/locks.py lines 99-306):
- Maintains one RWLock per unique file path
- Tracks which session holds which locks
- Automatic cleanup on disconnect via
release_session()
Deadlock Prevention:
When operations require multiple locks (e.g., mv needs source and destination):
# Sort paths to ensure consistent acquisition order
paths = sorted([source_path, dest_path])
locks.acquire_write(paths[0], session_id)
locks.acquire_write(paths[1], session_id)See src/netfs/server/commands/files.py lines 441-483
Path Traversal Prevention (src/netfs/server/storage.py lines 42-70):
All paths are validated to prevent attacks like ../../etc/passwd:
def resolve_path(self, path: str) -> Path:
resolved = (self.root / path.lstrip("/")).resolve()
# Security check: must stay within storage root
try:
resolved.relative_to(self.root)
except ValueError:
raise InvalidPathError(path)
return resolvedFile Operations:
- All file operations use
pathlib.Pathfor cross-platform compatibility - Binary files are automatically detected and base64-encoded for JSON transport
- File size limits enforced (default: 10MB)
Implementation: See src/netfs/server/storage.py
| Requirement | Implementation | Evidence |
|---|---|---|
| Create directories | mkdir command |
src/netfs/server/commands/dirs.py lines 95-138 |
| Store files | write, touch commands |
src/netfs/server/commands/files.py lines 155-212 |
| Move files to/from directory | mv, cp commands |
src/netfs/server/commands/files.py lines 397-605 |
| Read files | cat, head, tail |
src/netfs/server/commands/files.py lines 17-67 |
| Write files | write, append |
src/netfs/server/commands/files.py lines 155-326 |
| List directory | ls, tree commands |
src/netfs/server/commands/dirs.py lines 49-93 |
| Serve multiple users | Threaded server with per-client handlers | src/netfs/server/listener.py lines 134-178 |
| IPC demonstration | TCP sockets with JSON protocol | src/netfs/common/protocol.py |
| Process synchronization | Reader-writer locks | src/netfs/server/locks.py |
Run automated requirements check:
python scripts/verify_requirements.py
# Expected: 44/44 checks passing (100%)| Category | Tests | Coverage |
|---|---|---|
| Unit Tests | 314 | Core logic: 90-100% |
| Integration Tests | 25 | End-to-end workflows |
| Total | 339 | 83% overall |
# All tests with coverage
make test
# Or directly with pytest
pytest tests/ -v --cov=src/netfs --cov-report=term-missing
# Unit tests only
make test-unit
# Integration tests only
make test-integration| Test File | What It Tests |
|---|---|
tests/unit/test_protocol.py |
Message framing, encode/decode |
tests/unit/test_storage.py |
File operations, path security |
tests/unit/test_locks.py |
RWLock behavior, threading |
tests/integration/test_connection.py |
Full request/response flow |
tests/integration/test_concurrent.py |
Multi-client, lock contention |
This section provides copy-paste commands to demonstrate all NetFS functionality.
Terminal 1 - Start Server:
cd NetFS
source .venv/bin/activate
netfs serverTerminal 2 - Start Client:
cd NetFS
source .venv/bin/activate
netfs client --user professorRun these commands in order in the client shell:
| Step | Command | Expected Result |
|---|---|---|
| 1 | pwd |
Shows / (root directory) |
| 2 | ls |
Empty directory or existing files |
| 3 | mkdir project |
Creates /project directory |
| 4 | mkdir project/src |
Creates nested directory |
| 5 | mkdir project/docs |
Creates another directory |
| 6 | cd project |
Changes to /project |
| 7 | pwd |
Shows /project |
| 8 | write src/main.py print('hello') |
Creates file with content |
| 9 | write docs/readme.txt This is the readme file |
Creates another file |
| 10 | cat src/main.py |
Shows print('hello') |
| 11 | ls |
Shows src/ and docs/ |
| 12 | tree |
Shows directory tree structure |
| 13 | stat src/main.py |
Shows file metadata (size, modified) |
| 14 | head docs/readme.txt |
Shows first 10 lines |
| 15 | append docs/readme.txt More content here |
Appends to file |
| 16 | cat docs/readme.txt |
Shows original + appended content |
| 17 | touch docs/notes.txt |
Creates empty file |
| 18 | exists docs/notes.txt |
Shows file exists |
| 19 | exists docs/missing.txt |
Shows file does not exist |
| 20 | cp src/main.py src/backup.py |
Copies file |
| 21 | mv src/backup.py src/old.py |
Renames file |
| 22 | ls src |
Shows main.py and old.py |
| 23 | rm src/old.py |
Removes file |
| 24 | find / -name *.txt |
Finds all .txt files |
| 25 | cd / |
Returns to root |
| 26 | rmdir -r project |
Removes project and all contents |
| 27 | ls |
Shows project is gone |
| 28 | exit |
Exits shell |
Terminal 2 - Client A:
netfs client --user alice
mkdir shared
write shared/data.txt Important dataTerminal 3 - Client B (while Alice is connected):
netfs client --user bob
ls shared # Sees Alice's directory
cat shared/data.txt # Reads Alice's file
write shared/bob.txt My contributionTerminal 2 - Client A:
ls shared # Sees Bob's file too
exit# Runs all 18 commands automatically
bash scripts/demo_full_auto.sh
# Or Python version with detailed output
python scripts/demo_full.pyConfiguration file: config/config.yaml
server:
host: "0.0.0.0" # Listen on all interfaces
port: 9999 # Default port
max_connections: 100 # Maximum concurrent clients
connection_timeout_seconds: 300
socket_timeout_seconds: 1.0
buffer_size: 8192
reuse_address: true # Allow immediate port reusestorage:
root_path: "./storage" # Where files are stored
create_if_missing: true # Auto-create on startup
max_file_size_mb: 10 # Maximum file size
max_path_length: 4096protocol:
encoding: "utf-8"
file_encoding: "base64" # For binary files
max_message_size_mb: 16
length_prefix_bytes: 4
byte_order: "big"logging:
console:
enabled: true
level: "INFO"
colorize: true
file:
enabled: true
level: "DEBUG"
path: "./logs/netfs.log"
rotation: "10 MB"
retention: "7 days"NetFS/
├── config/
│ └── config.yaml # Default configuration
├── docs/
│ └── planning/ # Design documents
│ ├── architecture.md # Detailed architecture
│ ├── requirements.md # Command specifications
│ └── testing.md # Testing strategy
├── logs/ # Log files (generated)
├── scripts/
│ ├── demo_basic.sh # Interactive demo
│ ├── demo_full_auto.sh # Automated verification
│ ├── demo_full.py # Python demo script
│ └── verify_requirements.py # Requirements checker
├── src/netfs/
│ ├── common/ # Shared utilities
│ │ ├── config.py # Configuration management
│ │ ├── errors.py # Error codes and exceptions
│ │ ├── logging.py # Loguru setup
│ │ ├── models.py # Request/Response models
│ │ └── protocol.py # Message framing
│ ├── server/ # Server implementation
│ │ ├── commands/ # Command handlers
│ │ │ ├── __init__.py # CommandDispatcher
│ │ │ ├── dirs.py # Directory commands
│ │ │ ├── files.py # File commands
│ │ │ └── meta.py # pwd, cd commands
│ │ ├── handler.py # Connection handler
│ │ ├── listener.py # Server socket
│ │ ├── locks.py # Reader-writer locks
│ │ ├── main.py # Server CLI entry
│ │ ├── session.py # Client session
│ │ └── storage.py # File operations
│ ├── client/ # Client implementation
│ │ ├── error_helpers.py # Smart error messages
│ │ ├── formatter.py # Output formatting
│ │ ├── main.py # Client CLI entry
│ │ ├── protocol_client.py # Server communication
│ │ └── shell.py # Interactive shell
│ └── cli.py # Main entry point
├── storage/ # File storage (generated)
├── tests/
│ ├── unit/ # Unit tests (314)
│ └── integration/ # Integration tests (25)
├── Makefile # Build automation
├── pyproject.toml # Package configuration
├── CHANGELOG.md # Version history
├── JOURNAL.md # Development log
└── README.md # This file
Problem: TCP is a stream protocol. A single recv() call may return partial data or multiple messages combined.
Solution: Length-prefix framing. Every message starts with a 4-byte big-endian integer indicating payload size.
Implementation: src/netfs/common/protocol.py - recv_exactly() function
Problem: When mv or cp requires locks on both source and destination, two concurrent operations could deadlock:
- Client A: locks
/a, waits for/b - Client B: locks
/b, waits for/a
Solution: Always acquire locks in sorted path order.
Implementation: src/netfs/server/commands/files.py lines 441-450
Problem: Server blocks indefinitely on accept(), preventing clean shutdown.
Solution: Socket timeout with periodic shutdown flag check.
Implementation: src/netfs/server/listener.py lines 179-203
Problem: Malicious paths like ../../etc/passwd could access files outside storage.
Solution: Resolve all paths with pathlib and verify they remain within storage root using is_relative_to().
Implementation: src/netfs/server/storage.py lines 42-70
Problem: During ls, files could be deleted by another client between iterdir() and stat().
Solution: Catch FileNotFoundError during listing and skip deleted files.
Implementation: src/netfs/server/storage.py lines 200-216
| Error Code | Description | When It Occurs |
|---|---|---|
FILE_NOT_FOUND |
Path does not exist | cat, rm, cd on missing path |
PATH_ALREADY_EXISTS |
Path exists when it shouldn't | mkdir on existing directory |
DIRECTORY_NOT_EMPTY |
Cannot remove non-empty directory | rmdir without -r flag |
FILE_TOO_LARGE |
File exceeds size limit | write with content > 10MB |
INVALID_PATH |
Path validation failed | Path traversal attempts |
LOCK_TIMEOUT |
Could not acquire lock | Concurrent access conflict |
INVALID_COMMAND |
Command not recognized | Typo in command name |
INTERNAL_ERROR |
Unexpected server error | Bug or system error |
make install # Install package
make install-dev # Install with dev dependencies
make test # Run all tests with coverage
make test-unit # Run unit tests only
make test-integration # Run integration tests only
make lint # Run linters (ruff, mypy)
make format # Format code (black, ruff)
make run-server # Start the server
make run-client # Start the client
make demo # Run interactive demo
make demo-full-auto # Run automated demo
make clean # Remove build artifacts
make clean-storage # Clear storage directory
make zip # Create submission zipRuntime Dependencies:
loguru- Logging with rotationpydantic/pydantic-settings- Configuration validationpyyaml- YAML config file parsingprompt-toolkit- Interactive shellclick- CLI framework
Development Dependencies:
pytest/pytest-cov- Testingblack/ruff- Code formattingmypy- Type checking
See pyproject.toml for version requirements.
Problem: Linux blocks pip install outside virtual environment.
error: externally-managed-environment
This environment is externally managed
Solution:
python3 -m venv .venv
source .venv/bin/activate
pip install -e .Problem: Port 9999 is already in use by another process.
# Find process using port 9999
lsof -i :9999
# Kill it if needed
kill <PID>
# Or use a different port
netfs server --port 8888Checklist:
- Is the server running? Check Terminal 1 for server output.
- Are host and port correct?
netfs client --host 127.0.0.1 --port 9999
- If connecting remotely, check firewall settings.
Problem: Virtual environment not activated or package not installed.
# 1. Activate virtual environment
source .venv/bin/activate
# 2. Verify netfs is installed
pip list | grep netfs
# 3. If not installed, install it
pip install -e .# Reinstall in development mode
source .venv/bin/activate
pip install -e ".[dev]"Check the log file for detailed error information:
cat logs/netfs.log | tail -50| Limitation | Description | Workaround |
|---|---|---|
| No authentication | Any user can connect; username is for logging only | N/A - by design for v1 |
| No file permissions | All files readable/writable by all users | N/A - by design for v1 |
| No symbolic links | ln command not implemented |
Use cp instead |
| No file locking persistence | Locks are lost if server restarts | Restart clients after server restart |
find pattern syntax |
Do not quote glob patterns | Use find / -name *.txt not "*.txt" |
| Single server only | No replication or distributed storage | N/A |
| Memory for large files | Entire file loaded into memory | Keep files under 10MB (configurable) |
No cd to home |
cd requires a path argument |
Use cd / for root |
No ls -l flag |
Detailed listing not available | Use stat <file> for details |
John Akujobi CSC456 - Operating Systems Programming Assignment 2: Network File System
- VSCode: Integrated Development Environment
- Perplexity: Research and documentation searches
- Claude: Project exploration and consultation
- Linux Mint OS: Server deployment environment
- HP Machine: Physical server hardware
- Windows 11: Development environment
- Tailscale: Private VPN for remote server access
- LMStudio + Continue.dev + Qwen: Local LLM inference for coding assistance
- Obsidian: Markdown editing and PDF conversion
- Git: Version control
Educational use only - CSC456 Programming Assignment 2