Fix: URL-encoded special characters in $filter and $orderby break OData parsing

[Copilot]

Why make this change?

HttpUtility.ParseQueryString() decodes query parameter values, causing double-decoding when DAB constructs OData filter strings. A URL like ?$filter=title eq 'A %26 B' becomes title eq 'A & B', where the literal & is interpreted as a query parameter separator, truncating the filter to title eq 'A and producing "unterminated string literal" errors.

What is this change?

Preserve URL encoding for OData parameters by extracting raw values before ParseQueryString() decodes them:

1. RestRequestContext: Added RawQueryString property to store original encoded query
2. RestService: Populate RawQueryString alongside ParsedQueryString
3. RequestParser: New ExtractRawQueryParameter() extracts encoded values by splitting on unencoded & separators
4. $filter/$orderby parsing: Use raw values instead of decoded values when constructing OData query strings

// Extract raw parameter preserving encoding
private static string? ExtractRawQueryParameter(string queryString, string parameterName)
{
    if (string.IsNullOrWhiteSpace(queryString)) return null;
    foreach (string param in queryString.TrimStart('?').Split('&'))
    {
        int idx = param.IndexOf('=');
        if (idx >= 0 && param.Substring(0, idx).Equals(parameterName, StringComparison.OrdinalIgnoreCase))
            return idx < param.Length - 1 ? param.Substring(idx + 1) : string.Empty;
    }
    return null;
}

Database-agnostic: operates at HTTP parsing layer before any DB-specific processing.

How was this tested?

• Integration Tests - Added test data and queries across all DB types (MsSql, PostgreSQL, MySQL, DwSql)
• Unit Tests - 14 tests validating encoding preservation in various scenarios

Sample Request(s)

REST - Before (fails):

GET /api/Book?$filter=title eq 'filter & test'
→ 400 Bad Request: "unterminated string literal at position 17 in 'title eq 'filter'."

REST - After (succeeds):

GET /api/Book?$filter=title eq 'filter & test'
→ 200 OK: [{"id": 22, "title": "filter & test", "publisher_id": 1234}]

Works with any URL-encoded special character: %26 (&), %3D (=), %3F (?), etc.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[Bug]: Special characters in filter clause</issue_title>
<issue_description>### What happened?

Tried filtering on the string "filter & test"

Original URL: https://localhost:5001/api/my_entity?$filter=region eq 'filter & test'
Encoded URL: https://localhost:5001/api/my_entity?$filter=region%20eq%20%27filter%20%26%20test%27

I get this response when using the encoded URL:
{
"error": {
"code": "BadRequest",
"message": "There is an unterminated string literal at position 17 in 'region eq 'filter'.",
"status": 400
}
}

Version

1.6.84

What database are you using?

Azure SQL

What hosting model are you using?

Container Apps

Which API approach are you accessing DAB through?

REST

Relevant log output

Azure.DataApiBuilder.Service.Exceptions.DataApiBuilderException: There is an unterminated string literal at position 17 in 'region eq 'filter'.
       ---> Microsoft.OData.ODataException: There is an unterminated string literal at position 17 in 'region eq 'filter'.
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseComparison()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseLogicalAnd()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseLogicalOr()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseExpression()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseExpressionText(String expressionText)
         at Microsoft.OData.UriParser.ODataQueryOptionParser.ParseFilterImplementation(String filter, ODataUriParserConfiguration configuration, ODataPathInfo odataPathInfo)
         at Azure.DataApiBuilder.Core.Parsers.ODataParser.GetFilterClause(String filterQueryString, String resourcePath, ODataUriResolver customResolver) in /_/src/Core/Parsers/FilterParser.cs:line 64
         --- End of inner exception stack trace ---
         at Azure.DataApiBuilder.Core.Parsers.ODataParser.GetFilterClause(String filterQueryString, String resourcePath, ODataUriResolver customResolver) in /_/src/Core/Parsers/FilterParser.cs:line 68
         at Azure.DataApiBuilder.Core.Parsers.RequestParser.ParseQueryString(RestRequestContext context, ISqlMetadataProvider sqlMetadataProvider) in /_/src/Core/Parsers/RequestParser.cs:line 99
         at Azure.DataApiBuilder.Core.Services.RestService.ExecuteAsync(String entityName, EntityActionOperation operationType, String primaryKeyRoute) in /_/src/Core/Services/RestService.cs:line 178
         at Azure.DataApiBuilder.Service.Controllers.RestController.HandleOperation(String route, EntityActionOperation operationType) in /_/src/Service/Controllers/RestController.cs:line 239

Code of Conduct

• I agree to follow this project's Code of Conduct</issue_description>

<agent_instructions>Introduce as little new code as necessary. Ensure this and variants of this problem are resolved by creating as generic of a solution as possible. </agent_instructions>

Comments on the Issue (you are @copilot in this section)

@JerryNixon This is a DAB bug, not a user error.

What's Happening

The user correctly URL-encoded their filter value:

• & → %26
• spaces → %20

However, DAB (or the underlying framework) is double-decoding the URL. The %26 gets decoded back to &, which then causes the query string parser to split on that & — treating everything after it as a separate query parameter.

Result: The filter string gets truncated to region eq 'filter (note the unterminated quote), hence the error.

Root Cause

This is a known class of issue with URL encoding in ASP.NET / OData scenarios. The %26 is being decoded at the framework level before the OData filter parser receives it, so the filter parser never sees the encoded value — it sees a literal & which breaks query string parsing.

Workaround

Double-encode the ampersand: %2526 (encodes the % in %26)

Fix for DAB

The OData filter parser needs to receive the raw/encoded query string, or DAB needs to handle this edge case explicitly when parsing $filter values containing special characters.</comment_new>

• Fixes [Bug]: Special characters in filter clause #3039

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

Pull request overview

This PR fixes a critical bug where URL-encoded special characters in $filter and $orderby query parameters were being double-decoded, causing OData parsing failures. The issue occurred because HttpUtility.ParseQueryString() decodes parameter values before they reach the OData parser, which also expects to decode them.

Changes:

• Added RawQueryString property to preserve URL-encoded query strings before decoding
• Modified $filter and $orderby parsing to use raw (URL-encoded) values instead of decoded values
• Added comprehensive unit and integration tests across all supported database types

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/Core/Models/RestRequestContexts/RestRequestContext.cs	Added RawQueryString property to store the URL-encoded query string
src/Core/Services/RestService.cs	Populated RawQueryString before parsing query parameters
src/Core/Parsers/RequestParser.cs	Added ExtractRawQueryParameter() method and modified $filter/$orderby parsing to use raw values
src/Service.Tests/UnitTests/RequestParserUnitTests.cs	Added 14 unit tests validating URL encoding preservation in various scenarios
src/Service.Tests/SqlTests/RestApiTests/Find/*.cs	Added integration tests for all database types (MsSql, PostgreSQL, MySQL, DwSql)
src/Service.Tests/DatabaseSchema-*.sql	Added test data with special characters for integration tests

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.