Skip to content

jakujobi/NetFS

Repository files navigation

NetFS - Network File System

CSC456 Operating Systems - Programming Project2 Author: John Akujobi Version: 1.0

A client-server network file system demonstrating inter-process communication (IPC) via TCP sockets, concurrent access control with reader-writer locks, and comprehensive file system operations.


Quick Start

Installation (30 seconds)

# Create virtual environment (recommended on Linux)
python3 -m venv .venv
source .venv/bin/activate

# Install package
pip install -e .

Run Server

# Terminal 1
netfs server

Run Client

# Terminal 2
netfs client --user professor

Try These Commands

pwd                              # Show current directory
mkdir demo                       # Create directory
write demo/hello.txt "Hello!"    # Create file with content
cat demo/hello.txt               # Read file
ls demo                          # List directory
tree                             # Show directory tree
exit                             # Exit client

Table of Contents


Features Summary

Category Count Description
Commands 18 Full Linux-like file system commands
Unit Tests 314 Comprehensive component testing
Integration Tests 25 End-to-end client-server tests
Code Coverage 83% Exceeds 70% requirement

Core Capabilities

  • Multi-client support: Multiple concurrent connections via threaded server
  • Reader-writer locks: Multiple simultaneous readers OR exclusive writer
  • TCP/JSON protocol: Length-prefixed framing for reliable message boundaries
  • Path security: Prevents directory traversal attacks
  • Binary file support: Base64 encoding for non-text files
  • Interactive shell: Command history, auto-completion, colored output

Installation Details

Prerequisites

  • Python 3.11 or higher
  • pip (Python package manager)
  • Linux recommended (tested on Ubuntu with Python 3.12.3)

Why Use a Virtual Environment?

Modern Linux distributions (Ubuntu 23.04+, Debian 12+, Fedora 38+) use "externally managed" Python, which blocks pip install to protect system packages. You'll see this error:

error: externally-managed-environment
This environment is externally managed

Solution: Always use a virtual environment. This is the recommended approach and avoids all system Python conflicts.

Step-by-Step Installation

# 1. Navigate to project directory
cd NetFS

# 2. Create virtual environment
python3 -m venv .venv

# 3. Activate virtual environment
source .venv/bin/activate    # Linux/macOS
# .venv\Scripts\activate     # Windows (if needed)

# 4. Verify you're in the virtual environment
# Your prompt should show (.venv) at the beginning
which python    # Should show: /path/to/NetFS/.venv/bin/python

# 5. Install NetFS package
pip install -e .

# 6. Verify installation
netfs --help

Virtual Environment Quick Reference

Action Command
Create venv python3 -m venv .venv
Activate (Linux/Mac) source .venv/bin/activate
Activate (Windows) .venv\Scripts\activate
Deactivate deactivate
Check if active Look for (.venv) in prompt
Install package pip install -e .
Install with dev tools pip install -e ".[dev]"

Alternative: Using Makefile

# Note: Activate virtual environment first!
source .venv/bin/activate

make install        # Install package
make install-dev    # Install with development dependencies

Verifying Installation

# Check netfs is available
netfs --help

# Check Python version
python --version    # Should be 3.11+

# Check installed packages
pip list | grep -E "netfs|loguru|pydantic|click"

Usage Guide

Starting the Server

# Default: listens on 0.0.0.0:9999
netfs server

# Custom port
netfs server --port 8888

# Custom config file
netfs server --config path/to/config.yaml

Server Output:

2024-11-27 10:00:00 | INFO | NetFS Server Starting
2024-11-27 10:00:00 | INFO | Server address: 0.0.0.0:9999
2024-11-27 10:00:00 | INFO | Storage root: ./storage

Starting the Client

# Connect to local server (default)
netfs client

# Connect with username
netfs client --user john

# Connect to remote server
netfs client --host 192.168.1.100 --port 9999 --user alice

# Disable colored output
netfs client --no-color

Client Prompt:

netfs:/$ _

The prompt shows your current directory. Use help to see all commands.

Keyboard Shortcuts

Key Action
Tab Auto-complete commands and paths
Up/Down Navigate command history
Ctrl+C Cancel current input
Ctrl+D Exit shell

Complete Command Reference

Meta Commands

Command Syntax Description Example
pwd pwd Print current working directory pwd
cd cd <path> Change directory cd /documents
help help Show available commands help
exit exit or quit Exit the shell exit

Directory Commands

Command Syntax Description Example
ls ls [path] List directory contents ls or ls /docs
mkdir mkdir <path> Create directory mkdir projects
rmdir rmdir [-r] <path> Remove directory rmdir temp or rmdir -r old
tree tree [path] [-d depth] Show directory tree tree or tree / -d 3
find find <path> -name <pattern> Find files by pattern find / -name *.txt

File Reading Commands

Command Syntax Description Example
cat cat <path> Display entire file cat readme.txt
head head [-n N] <path> Show first N lines (default: 10) head -n 5 log.txt
tail tail [-n N] <path> Show last N lines (default: 10) tail -n 20 log.txt
stat stat <path> Show file/directory metadata stat data.json
exists exists <path> Check if path exists exists config.yaml

File Writing Commands

Command Syntax Description Example
write write <path> <content> Create/overwrite file write note.txt Hello
append append <path> <content> Append to file append log.txt New entry
touch touch <path> Create empty file or update timestamp touch newfile.txt

File Management Commands

Command Syntax Description Example
rm rm [-r] <path> Remove file or directory rm old.txt or rm -r temp/
mv mv <src> <dst> Move or rename mv old.txt new.txt
cp cp [-r] <src> <dst> Copy file or directory cp file.txt backup.txt

System Architecture

High-Level Overview

┌─────────────────────────────────────────────────────────────────┐
│                        NetFS Architecture                       │
└─────────────────────────────────────────────────────────────────┘

  ┌──────────────┐                              ┌──────────────────┐
  │   Client 1   │◄─────── TCP Socket ─────────►│                  │
  │ (Interactive │       (Port 9999)            │  ServerListener  │
  │    Shell)    │                              │                  │
  └──────────────┘                              │  ┌────────────┐  │
                                                │  │  Handler   │  │
  ┌──────────────┐                              │  │  Threads   │  │
  │   Client 2   │◄─────── TCP Socket ─────────►│  │ (1/client) │  │
  │ (Interactive │                              │  └────────────┘  │
  │    Shell)    │                              │                  │
  └──────────────┘                              │  ┌────────────┐  │
                                                │  │   Lock     │  │
  ┌──────────────┐                              │  │  Manager   │  │
  │   Client 3   │◄─────── TCP Socket ─────────►│  │  (RWLock)  │  │
  │ (Interactive │                              │  └────────────┘  │
  │    Shell)    │                              │                  │
  └──────────────┘                              │  ┌────────────┐  │
                                                │  │  Storage   │  │
                                                │  │  Manager   │  │
                                                │  └────────────┘  │
                                                └────────┬─────────┘
                                                         │
                                                         ▼
                                                  ┌──────────────┐
                                                  │  ./storage/  │
                                                  │ (Filesystem) │
                                                  └──────────────┘

Component Responsibilities

Component File Responsibility
ServerListener src/netfs/server/listener.py Accept connections, spawn handler threads
ConnectionHandler src/netfs/server/handler.py Request/response loop for one client
CommandDispatcher src/netfs/server/commands/__init__.py Route commands to handlers
StorageManager src/netfs/server/storage.py File system operations with security
LockManager src/netfs/server/locks.py Per-path reader-writer locks
ClientSession src/netfs/server/session.py Track session state (cwd, locks)
ProtocolClient src/netfs/client/protocol_client.py TCP connection and message framing
InteractiveShell src/netfs/client/shell.py REPL with history and completion

Technical Implementation

TCP Socket Communication

NetFS uses TCP sockets for reliable, ordered delivery of messages between clients and server.

Server Socket Setup (src/netfs/server/listener.py lines 110-132):

# Create TCP socket
self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Allow immediate address reuse (prevents "Address already in use")
self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

# Bind and listen
self.server_socket.bind((self.config.host, self.config.port))
self.server_socket.listen(self.config.max_connections)

Handling Partial Receives (src/netfs/common/protocol.py lines 76-100):

TCP is a stream protocol - recv() may return fewer bytes than requested. The recv_exactly() function handles this:

def recv_exactly(sock, num_bytes):
    buffer = b""
    while len(buffer) < num_bytes:
        chunk = sock.recv(num_bytes - len(buffer))
        if not chunk:
            raise ConnectionError("Connection closed")
        buffer += chunk
    return buffer

Protocol Design

Message Format:

┌────────────────┬─────────────────────────────────────┐
│    4 bytes     │          Variable length            │
│ Length Prefix  │           JSON Payload              │
│  (Big-endian)  │                                     │
└────────────────┴─────────────────────────────────────┘

Request Example:

{
  "cmd": "write",
  "username": "john",
  "args": {
    "path": "/documents/report.txt",
    "content": "Hello, World!",
    "binary": false
  }
}

Response Example:

{
  "status": "success",
  "message": "File written",
  "data": {
    "path": "/documents/report.txt",
    "size": 13,
    "created": true
  },
  "error_code": null
}

Implementation: See src/netfs/common/protocol.py

Thread Model

Main Thread                    Handler Threads (daemon)
     │                              │
     │  accept()                    │
     │──────────►┌──────────────────┤
     │           │ Thread 1         │
     │           │ (Client A)       │
     │           └──────────────────┤
     │                              │
     │  accept()                    │
     │──────────►┌──────────────────┤
     │           │ Thread 2         │
     │           │ (Client B)       │
     │           └──────────────────┤
     │                              │
     ▼           ... more clients   ▼

Each client connection spawns a daemon thread running ConnectionHandler.run(). Daemon threads automatically terminate when the main thread exits.

Implementation: See src/netfs/server/listener.py lines 159-174

Reader-Writer Locks

NetFS implements custom reader-writer locks to allow:

  • Multiple concurrent readers (e.g., multiple cat commands on same file)
  • Exclusive writer access (e.g., write blocks other operations)

Lock Behavior Visualization:

Time ────────────────────────────────────────────────────►

Reader 1:  ████████████                      (reading file)
Reader 2:  ████████████                      (reading same file - OK!)
Writer 1:              ░░░░████████████      (waits, then writes)
Reader 3:                              ████  (waits for writer)

████ = Active    ░░░░ = Waiting

RWLock Implementation (src/netfs/server/locks.py lines 12-97):

  • Uses threading.Condition for efficient wait/notify
  • Read-preferring strategy: readers proceed immediately if no writer
  • Timeout support prevents indefinite blocking

LockManager (src/netfs/server/locks.py lines 99-306):

  • Maintains one RWLock per unique file path
  • Tracks which session holds which locks
  • Automatic cleanup on disconnect via release_session()

Deadlock Prevention: When operations require multiple locks (e.g., mv needs source and destination):

# Sort paths to ensure consistent acquisition order
paths = sorted([source_path, dest_path])
locks.acquire_write(paths[0], session_id)
locks.acquire_write(paths[1], session_id)

See src/netfs/server/commands/files.py lines 441-483

Storage and Security

Path Traversal Prevention (src/netfs/server/storage.py lines 42-70):

All paths are validated to prevent attacks like ../../etc/passwd:

def resolve_path(self, path: str) -> Path:
    resolved = (self.root / path.lstrip("/")).resolve()
  
    # Security check: must stay within storage root
    try:
        resolved.relative_to(self.root)
    except ValueError:
        raise InvalidPathError(path)
  
    return resolved

File Operations:

  • All file operations use pathlib.Path for cross-platform compatibility
  • Binary files are automatically detected and base64-encoded for JSON transport
  • File size limits enforced (default: 10MB)

Implementation: See src/netfs/server/storage.py


Requirements Compliance

Assignment Requirements Mapping

Requirement Implementation Evidence
Create directories mkdir command src/netfs/server/commands/dirs.py lines 95-138
Store files write, touch commands src/netfs/server/commands/files.py lines 155-212
Move files to/from directory mv, cp commands src/netfs/server/commands/files.py lines 397-605
Read files cat, head, tail src/netfs/server/commands/files.py lines 17-67
Write files write, append src/netfs/server/commands/files.py lines 155-326
List directory ls, tree commands src/netfs/server/commands/dirs.py lines 49-93
Serve multiple users Threaded server with per-client handlers src/netfs/server/listener.py lines 134-178
IPC demonstration TCP sockets with JSON protocol src/netfs/common/protocol.py
Process synchronization Reader-writer locks src/netfs/server/locks.py

Verification Script

Run automated requirements check:

python scripts/verify_requirements.py
# Expected: 44/44 checks passing (100%)

Testing

Test Summary

Category Tests Coverage
Unit Tests 314 Core logic: 90-100%
Integration Tests 25 End-to-end workflows
Total 339 83% overall

Running Tests

# All tests with coverage
make test

# Or directly with pytest
pytest tests/ -v --cov=src/netfs --cov-report=term-missing

# Unit tests only
make test-unit

# Integration tests only
make test-integration

Key Test Files

Test File What It Tests
tests/unit/test_protocol.py Message framing, encode/decode
tests/unit/test_storage.py File operations, path security
tests/unit/test_locks.py RWLock behavior, threading
tests/integration/test_connection.py Full request/response flow
tests/integration/test_concurrent.py Multi-client, lock contention

Demo Walkthrough

This section provides copy-paste commands to demonstrate all NetFS functionality.

Setup

Terminal 1 - Start Server:

cd NetFS
source .venv/bin/activate
netfs server

Terminal 2 - Start Client:

cd NetFS
source .venv/bin/activate
netfs client --user professor

Demo Commands Table

Run these commands in order in the client shell:

Step Command Expected Result
1 pwd Shows / (root directory)
2 ls Empty directory or existing files
3 mkdir project Creates /project directory
4 mkdir project/src Creates nested directory
5 mkdir project/docs Creates another directory
6 cd project Changes to /project
7 pwd Shows /project
8 write src/main.py print('hello') Creates file with content
9 write docs/readme.txt This is the readme file Creates another file
10 cat src/main.py Shows print('hello')
11 ls Shows src/ and docs/
12 tree Shows directory tree structure
13 stat src/main.py Shows file metadata (size, modified)
14 head docs/readme.txt Shows first 10 lines
15 append docs/readme.txt More content here Appends to file
16 cat docs/readme.txt Shows original + appended content
17 touch docs/notes.txt Creates empty file
18 exists docs/notes.txt Shows file exists
19 exists docs/missing.txt Shows file does not exist
20 cp src/main.py src/backup.py Copies file
21 mv src/backup.py src/old.py Renames file
22 ls src Shows main.py and old.py
23 rm src/old.py Removes file
24 find / -name *.txt Finds all .txt files
25 cd / Returns to root
26 rmdir -r project Removes project and all contents
27 ls Shows project is gone
28 exit Exits shell

Concurrent Access Demo

Terminal 2 - Client A:

netfs client --user alice
mkdir shared
write shared/data.txt Important data

Terminal 3 - Client B (while Alice is connected):

netfs client --user bob
ls shared              # Sees Alice's directory
cat shared/data.txt    # Reads Alice's file
write shared/bob.txt My contribution

Terminal 2 - Client A:

ls shared              # Sees Bob's file too
exit

Automated Demo

# Runs all 18 commands automatically
bash scripts/demo_full_auto.sh

# Or Python version with detailed output
python scripts/demo_full.py

Configuration Reference

Configuration file: config/config.yaml

Server Settings

server:
  host: "0.0.0.0"              # Listen on all interfaces
  port: 9999                    # Default port
  max_connections: 100          # Maximum concurrent clients
  connection_timeout_seconds: 300
  socket_timeout_seconds: 1.0
  buffer_size: 8192
  reuse_address: true           # Allow immediate port reuse

Storage Settings

storage:
  root_path: "./storage"        # Where files are stored
  create_if_missing: true       # Auto-create on startup
  max_file_size_mb: 10          # Maximum file size
  max_path_length: 4096

Protocol Settings

protocol:
  encoding: "utf-8"
  file_encoding: "base64"       # For binary files
  max_message_size_mb: 16
  length_prefix_bytes: 4
  byte_order: "big"

Logging Settings

logging:
  console:
    enabled: true
    level: "INFO"
    colorize: true
  file:
    enabled: true
    level: "DEBUG"
    path: "./logs/netfs.log"
    rotation: "10 MB"
    retention: "7 days"

Project Structure

NetFS/
├── config/
│   └── config.yaml                 # Default configuration
├── docs/
│   └── planning/                   # Design documents
│       ├── architecture.md         # Detailed architecture
│       ├── requirements.md         # Command specifications
│       └── testing.md              # Testing strategy
├── logs/                           # Log files (generated)
├── scripts/
│   ├── demo_basic.sh               # Interactive demo
│   ├── demo_full_auto.sh           # Automated verification
│   ├── demo_full.py                # Python demo script
│   └── verify_requirements.py      # Requirements checker
├── src/netfs/
│   ├── common/                     # Shared utilities
│   │   ├── config.py               # Configuration management
│   │   ├── errors.py               # Error codes and exceptions
│   │   ├── logging.py              # Loguru setup
│   │   ├── models.py               # Request/Response models
│   │   └── protocol.py             # Message framing
│   ├── server/                     # Server implementation
│   │   ├── commands/               # Command handlers
│   │   │   ├── __init__.py         # CommandDispatcher
│   │   │   ├── dirs.py             # Directory commands
│   │   │   ├── files.py            # File commands
│   │   │   └── meta.py             # pwd, cd commands
│   │   ├── handler.py              # Connection handler
│   │   ├── listener.py             # Server socket
│   │   ├── locks.py                # Reader-writer locks
│   │   ├── main.py                 # Server CLI entry
│   │   ├── session.py              # Client session
│   │   └── storage.py              # File operations
│   ├── client/                     # Client implementation
│   │   ├── error_helpers.py        # Smart error messages
│   │   ├── formatter.py            # Output formatting
│   │   ├── main.py                 # Client CLI entry
│   │   ├── protocol_client.py      # Server communication
│   │   └── shell.py                # Interactive shell
│   └── cli.py                      # Main entry point
├── storage/                        # File storage (generated)
├── tests/
│   ├── unit/                       # Unit tests (314)
│   └── integration/                # Integration tests (25)
├── Makefile                        # Build automation
├── pyproject.toml                  # Package configuration
├── CHANGELOG.md                    # Version history
├── JOURNAL.md                      # Development log
└── README.md                       # This file

Challenges and Solutions

Challenge 1: TCP Message Boundaries

Problem: TCP is a stream protocol. A single recv() call may return partial data or multiple messages combined.

Solution: Length-prefix framing. Every message starts with a 4-byte big-endian integer indicating payload size.

Implementation: src/netfs/common/protocol.py - recv_exactly() function

Challenge 2: Deadlock Prevention

Problem: When mv or cp requires locks on both source and destination, two concurrent operations could deadlock:

  • Client A: locks /a, waits for /b
  • Client B: locks /b, waits for /a

Solution: Always acquire locks in sorted path order.

Implementation: src/netfs/server/commands/files.py lines 441-450

Challenge 3: Graceful Shutdown

Problem: Server blocks indefinitely on accept(), preventing clean shutdown.

Solution: Socket timeout with periodic shutdown flag check.

Implementation: src/netfs/server/listener.py lines 179-203

Challenge 4: Path Traversal Attacks

Problem: Malicious paths like ../../etc/passwd could access files outside storage.

Solution: Resolve all paths with pathlib and verify they remain within storage root using is_relative_to().

Implementation: src/netfs/server/storage.py lines 42-70

Challenge 5: Concurrent File Listing

Problem: During ls, files could be deleted by another client between iterdir() and stat().

Solution: Catch FileNotFoundError during listing and skip deleted files.

Implementation: src/netfs/server/storage.py lines 200-216


Error Codes

Error Code Description When It Occurs
FILE_NOT_FOUND Path does not exist cat, rm, cd on missing path
PATH_ALREADY_EXISTS Path exists when it shouldn't mkdir on existing directory
DIRECTORY_NOT_EMPTY Cannot remove non-empty directory rmdir without -r flag
FILE_TOO_LARGE File exceeds size limit write with content > 10MB
INVALID_PATH Path validation failed Path traversal attempts
LOCK_TIMEOUT Could not acquire lock Concurrent access conflict
INVALID_COMMAND Command not recognized Typo in command name
INTERNAL_ERROR Unexpected server error Bug or system error

Makefile Commands

make install         # Install package
make install-dev     # Install with dev dependencies
make test            # Run all tests with coverage
make test-unit       # Run unit tests only
make test-integration # Run integration tests only
make lint            # Run linters (ruff, mypy)
make format          # Format code (black, ruff)
make run-server      # Start the server
make run-client      # Start the client
make demo            # Run interactive demo
make demo-full-auto  # Run automated demo
make clean           # Remove build artifacts
make clean-storage   # Clear storage directory
make zip             # Create submission zip

Dependencies

Runtime Dependencies:

  • loguru - Logging with rotation
  • pydantic / pydantic-settings - Configuration validation
  • pyyaml - YAML config file parsing
  • prompt-toolkit - Interactive shell
  • click - CLI framework

Development Dependencies:

  • pytest / pytest-cov - Testing
  • black / ruff - Code formatting
  • mypy - Type checking

See pyproject.toml for version requirements.


Troubleshooting

"externally-managed-environment" Error

Problem: Linux blocks pip install outside virtual environment.

error: externally-managed-environment
This environment is externally managed

Solution:

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Server won't start: "Address already in use"

Problem: Port 9999 is already in use by another process.

# Find process using port 9999
lsof -i :9999

# Kill it if needed
kill <PID>

# Or use a different port
netfs server --port 8888

Client can't connect

Checklist:

  1. Is the server running? Check Terminal 1 for server output.
  2. Are host and port correct?
    netfs client --host 127.0.0.1 --port 9999
  3. If connecting remotely, check firewall settings.

"Command not found: netfs"

Problem: Virtual environment not activated or package not installed.

# 1. Activate virtual environment
source .venv/bin/activate

# 2. Verify netfs is installed
pip list | grep netfs

# 3. If not installed, install it
pip install -e .

Tests fail with import errors

# Reinstall in development mode
source .venv/bin/activate
pip install -e ".[dev]"

Server logs show errors but client shows success

Check the log file for detailed error information:

cat logs/netfs.log | tail -50

Known Limitations

Limitation Description Workaround
No authentication Any user can connect; username is for logging only N/A - by design for v1
No file permissions All files readable/writable by all users N/A - by design for v1
No symbolic links ln command not implemented Use cp instead
No file locking persistence Locks are lost if server restarts Restart clients after server restart
find pattern syntax Do not quote glob patterns Use find / -name *.txt not "*.txt"
Single server only No replication or distributed storage N/A
Memory for large files Entire file loaded into memory Keep files under 10MB (configurable)
No cd to home cd requires a path argument Use cd / for root
No ls -l flag Detailed listing not available Use stat <file> for details

Author

John Akujobi CSC456 - Operating Systems Programming Assignment 2: Network File System


Tools Used

  • VSCode: Integrated Development Environment
  • Perplexity: Research and documentation searches
  • Claude: Project exploration and consultation
  • Linux Mint OS: Server deployment environment
  • HP Machine: Physical server hardware
  • Windows 11: Development environment
  • Tailscale: Private VPN for remote server access
  • LMStudio + Continue.dev + Qwen: Local LLM inference for coding assistance
  • Obsidian: Markdown editing and PDF conversion
  • Git: Version control

License

Educational use only - CSC456 Programming Assignment 2

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published