# Cognica - Full Documentation
> A transactional, PostgreSQL-compatible database engine that unifies SQL, full-text search, vector search, and graph queries in a single system.
Full text of Cognica's product documentation, concatenated for LLM consumption. For a concise, curated index see https://www.cognica.io/llms.txt
# SQL Reference
## Preface: Welcome to Cognica SQL
Welcome to the Cognica SQL Reference Manual. This document is designed to be your comprehensive guide to writing SQL queries in Cognica Database. Whether you're a seasoned database administrator or just starting your journey with SQL, this manual will help you understand not just the syntax, but the underlying concepts that make your queries work efficiently.
### PostgreSQL Compatibility: Use What You Already Know
**Cognica speaks PostgreSQL.** If you know PostgreSQL, you already know Cognica SQL.
Cognica implements the PostgreSQL wire protocol, which means:
- **Your existing tools just work.** Connect with psql, pgAdmin, DBeaver, DataGrip, or any PostgreSQL-compatible client. No special drivers or adapters needed.
- **Your existing code just works.** Applications using libpq, psycopg2, node-postgres, JDBC, or any PostgreSQL driver can connect to Cognica without modification.
- **Your existing queries just work.** The vast majority of PostgreSQL SQL syntax is supported. SELECT, INSERT, UPDATE, DELETE, JOIN, subqueries, CTEs, window functions, transactions - they all work as expected.
```python
# Python example - same code works for both PostgreSQL and Cognica
import psycopg2
# Just change the connection string
conn = psycopg2.connect("host=localhost port=5432 dbname=mydb")
cursor = conn.cursor()
cursor.execute("SELECT * FROM users WHERE created_at > NOW() - INTERVAL '7 days'")
```
```javascript
// Node.js example - identical API
const { Client } = require('pg');
const client = new Client({ host: 'localhost', port: 5432, database: 'mydb' });
await client.connect();
const result = await client.query('SELECT * FROM orders WHERE status = $1', ['pending']);
```
**What This Means for You:**
| If you're coming from... | Your experience with Cognica |
|--------------------------|------------------------------|
| PostgreSQL | Immediate productivity. Your SQL knowledge transfers directly. |
| MySQL/MariaDB | Familiar SQL with PostgreSQL-style syntax. Minor adjustments needed. |
| SQL Server | Standard SQL works. Replace T-SQL specific features with PostgreSQL equivalents. |
| Oracle | Standard SQL works. PL/SQL procedures need conversion to PL/Python. |
**Cognica-Specific Extensions:**
While Cognica maintains PostgreSQL compatibility, it also provides extensions for modern workloads:
- **Vector Search**: `SELECT * FROM docs WHERE _all @@ 'embedding:[[0.1, 0.2, ...]]' ORDER BY _meta.score DESC`
- **Full-Text Search**: `SELECT * FROM articles WHERE _all @@ 'content:(database performance)'`
- **Hybrid Search**: Combine keyword precision with semantic understanding
- **Document Operations**: Native JSON/JSONB support with rich query operators
- **Data Federation**: Query external databases and files as if they were local tables
**Hybrid Search - The Best of Both Worlds:**
Modern search applications need both keyword precision and semantic understanding. Cognica's unified query syntax makes this elegant:
```sql
-- Hybrid search: keyword filter + vector ranking in one query
SELECT id, title, _meta.score
FROM articles
WHERE _all @@ 'content:Python AND content_embedding:[0.1, 0.2, ...]'
ORDER BY _meta.score DESC
LIMIT 10;
```
```sql
-- SQL filters + vector search for precise RAG retrieval
SELECT content, source, _meta.score AS relevance
FROM knowledge_base
WHERE published_date > '2024-01-01'
AND source_type IN ('official_docs', 'peer_reviewed')
AND _all @@ 'content_embedding:[query_embedding...]'
ORDER BY _meta.score DESC
LIMIT 5;
```
This unified syntax is essential for RAG (Retrieval-Augmented Generation) applications - no complex score merging or multiple queries needed.
These extensions use PostgreSQL-compatible syntax wherever possible, so they feel natural to PostgreSQL users.
### Who This Manual Is For
This manual is written for:
- **Application Developers** who need to write queries for their applications and want to understand how to get the best performance
- **Data Analysts** who want to extract insights from data and need to master complex queries
- **Database Administrators** who need deep understanding of how queries execute and how to optimize them
- **Anyone Learning SQL** who wants a thorough, friendly guide that explains concepts clearly
### How to Use This Manual
Each chapter builds on previous concepts, but you can also use this as a reference to look up specific topics. Every section includes:
- Clear explanations of what each feature does and why it exists
- Practical examples you can run immediately
- Common mistakes to avoid
- Performance considerations where relevant
- Tips from real-world usage
Let's begin your journey into Cognica SQL!
---
---
## Chapter 1: Understanding Cognica's Architecture
Before writing your first query, it helps tremendously to understand what happens behind the scenes when you execute SQL. This knowledge will help you write better queries and troubleshoot performance issues.
### 1.1 The Journey of a SQL Query
When you type a SQL query and press Enter, that simple text goes through a remarkable transformation before returning your results. Let's trace this journey step by step.
```mermaid
flowchart LR
subgraph Input
SQL["SQL Query"]
end
subgraph Processing["Query Processing"]
Parser["1. Parser
(libpg_query)"]
Analyzer["2. Semantic
Analyzer"]
Planner["3. Query
Planner"]
Executor["4. Executor
(CVM)"]
end
subgraph Output
Results["Results"]
end
SQL --> Parser
Parser --> Analyzer
Analyzer --> Planner
Planner --> Executor
Executor --> Results
Parser -.->|"Syntax Error"| SQL
Analyzer -.->|"Table/Column
Not Found"| SQL
```
**Step 1: Parsing - Understanding Your Intent**
The first thing Cognica does is read your SQL text and figure out what you're asking for. This is called parsing, and Cognica uses the exact same parser that PostgreSQL uses (through a library called libpg_query). This means if your query works in PostgreSQL, it will almost certainly parse correctly in Cognica.
For example, when you write:
```sql
SELECT name, email FROM users WHERE active = true ORDER BY name
```
The parser breaks this down into a structured representation:
- "This is a SELECT statement"
- "The user wants two columns: name and email"
- "The data comes from the users table"
- "Only include rows where active equals true"
- "Sort the results by name"
**Why This Matters to You**: If you ever get a syntax error, it's coming from this parsing stage. The error message will tell you exactly where the parser got confused. Common parsing errors include missing commas, unmatched parentheses, or misspelled keywords.
**Pro Tip**: You can see exactly how Cognica parses any query using the `--show-tree` command:
```bash
bin/cognica db query sql "SELECT name FROM users WHERE id = 1" --show-tree
```
This shows you the parse tree in JSON format - incredibly useful for debugging complex queries or understanding how SQL syntax is interpreted.
You can also see the compiled CVM bytecode using the `--show-bytecode` command:
```bash
bin/cognica db query sql "SELECT name FROM users WHERE id = 1" --show-bytecode
```
This displays the physical execution plan and disassembled bytecode that the CVM (Cognica Virtual Machine) will execute:
```
---------- Physical Plan ----------
Seq Scan on users (cost=0.00..110.00 rows=1000 width=32)
Filter: (id = 1)
---------- CVM Bytecode Disassembly ----------
Query: SELECT name FROM users WHERE id = 1
; CVM Bytecode Disassembly
; ========================
;
; Version: 1.0
; Entry Point: 0x0000
; Code Size: 56 bytes
; Max GPRs: 5
; Max FPRs: 0
; Stack Depth: 0
.constants:
[ 0] string: "users"
[ 1] field_ref: "id"
[ 2] int64: 1
[ 3] field_ref: "name"
.code:
0000: F0000000 CURSOR_OPEN slot[0], [0] ; open cursor on "users"
0004: 7300000B JUMP_NULL R0, @0030 ; offset 11
0008: 90200001 GET_FIELD R2, R0, [1] ; get field "id"
000C: 07300002 LOAD_CONSTANT R3, [2] ; int64: 1
0010: 5E423000 CMP_EQ_POLY R4, R2, R3 ; equal
0014: 72400005 JUMP_FALSE R4, @0028 ; offset 5
0018: FE0B1000 DOCUMENT_NEW R1 ; create empty document
001C: 90200003 GET_FIELD R2, R0, [3] ; get field "name"
0020: 92120003 SET_FIELD R1, R2, [3] ; set field "name"
0024: F5010000 EMIT_ROW R1 ; emit row to output
0028: F1000000 CURSOR_NEXT R0, slot[0] ; fetch next row
002C: 7000FFF6 JUMP @0004 ; offset -10
0030: F2000000 CURSOR_CLOSE slot[0] ; close cursor
0034: 79000000 HALT
---------- Compilation Info ----------
Output columns: [name]
Cursor slots used: 1
Registers used: 5
Compilation time: 259 us
```
This is useful for understanding execution flow, debugging performance issues, or learning how SQL queries translate to low-level operations.
**Step 2: Semantic Analysis - Checking Your Work**
After parsing, Cognica needs to verify that your query actually makes sense. This is like a teacher checking your homework:
- Does the `users` table actually exist?
- Does that table have columns called `name` and `email`?
- Is `active` really a boolean column that can be compared to `true`?
- Do you have permission to read from this table?
If anything is wrong, you'll get a clear error message. For example:
```sql
SELECT naem FROM users; -- Typo in column name
-- Error: column "naem" does not exist
-- Hint: Perhaps you meant "name"?
```
**Step 3: Query Planning - Finding the Best Path**
This is where Cognica's intelligence really shines. There are often many ways to execute the same query, and the query planner's job is to find the fastest way.
Consider this query:
```sql
SELECT * FROM orders WHERE customer_id = 123 AND status = 'shipped'
```
The planner might consider:
- Should we scan the entire table and check each row?
- Is there an index on customer_id we could use?
- Is there an index on status?
- Should we use both indexes and combine the results?
Cognica's planner is specially optimized for LSM-tree storage. Cognica uses a customized version of RocksDB, modified and optimized specifically for Cognica's workloads. This is important because LSM-trees have different performance characteristics than traditional B-tree databases. What's fast in MySQL might not be fast in Cognica, and vice versa. The planner knows these differences and chooses accordingly.
**Step 4: Execution - Getting Your Results**
Finally, Cognica executes the plan. The execution engine (called CVM - Cognica Virtual Machine) reads data from storage, applies your filters, performs any calculations, sorts the results, and sends them back to you.
### 1.2 Understanding Storage: Why It Affects Your Queries
Cognica stores data using an LSM-tree (Log-Structured Merge-tree) architecture through a customized version of RocksDB. This is not vanilla RocksDB - Cognica maintains its own fork with modifications optimized for database workloads, including custom compaction strategies and memory management. You don't need to understand all the details, but knowing the basics helps you write better queries.
**Writes Are Fast**: When you INSERT data, it goes into memory first and is written to disk later in batches. This makes INSERT operations very fast - much faster than traditional databases that must update on-disk structures immediately.
**Reads May Check Multiple Places**: When you SELECT data, Cognica might need to check several places: the in-memory buffer, recent disk files, and older compacted files. The query planner accounts for this when deciding how to execute your query.
**Indexes Help a Lot**: Like all databases, Cognica performs much better when it can use indexes. But because of the LSM-tree architecture, the benefit of indexes can be even more pronounced in Cognica.
**Practical Implication**: If you notice a query is slow, the first thing to check is whether there's an appropriate index. We'll cover indexes in detail in Chapter 10.
---
## Chapter 2: Data Types - Choosing the Right Container for Your Data
Data types are the foundation of your database. Choosing the right type for each column affects storage efficiency, query performance, and data integrity. This chapter explains each type in detail, with guidance on when to use each one.
### The Philosophy of Type Selection
Choosing data types is one of the most consequential decisions you make when designing a database. Unlike application code that can be easily refactored, changing a column's data type in a table with millions of rows is expensive and risky. Getting it right the first time matters.
**The Tension: Flexibility vs. Integrity**
- More permissive types (TEXT, JSONB) accept almost anything but provide less validation
- More restrictive types (INTEGER, TIMESTAMP) reject invalid data but require you to know the format upfront
There is no universally correct choice. A startup prototyping rapidly might favor flexible types that can evolve. A financial system handling regulatory data might favor strict types that prevent any invalid entry.
**Common Type Selection Mistakes**
1. **VARCHAR everywhere**: Using VARCHAR(255) for everything ignores the semantic meaning of data. An email address, a UUID, and a product description are fundamentally different; their types should reflect this.
2. **Oversized integers**: Using BIGINT for a column that will never exceed a few thousand values wastes storage and memory. INTEGER handles 2 billion values; SMALLINT handles 32,767. Use the smallest type that fits your domain.
3. **Floating-point for money**: FLOAT and DOUBLE introduce rounding errors. Currency must use NUMERIC/DECIMAL with explicit precision, or be stored as integer cents.
4. **Timestamps without timezone awareness**: A timestamp of "2024-12-25 10:00:00" is ambiguous without knowing the timezone. Use TIMESTAMPTZ (timestamp with time zone) unless you have a specific reason not to.
5. **Stringly-typed data**: Storing structured data as strings (comma-separated values, dates as "MM/DD/YYYY" text) loses all the benefits of proper types: validation, indexing, comparison operators.
**The Type Selection Process**
For each column, ask:
1. What values are valid? (This determines the type)
2. What values are invalid? (This determines constraints)
3. How will this column be queried? (This affects indexing)
4. Will the type ever need to change? (This affects flexibility)
### 2.1 Numbers: Integers and Decimals
#### Integer Types: Counting Things
Integers are whole numbers without decimal points. Cognica provides three sizes:
| Type | Storage | Range | Best For |
|------|---------|-------|----------|
| SMALLINT | 2 bytes | -32,768 to 32,767 | Age, small quantities, ratings (1-5) |
| INTEGER | 4 bytes | About -2.1 billion to 2.1 billion | Most things: IDs, counts, quantities |
| BIGINT | 8 bytes | About -9.2 quintillion to 9.2 quintillion | Very large numbers, timestamps as integers |
**How to Choose the Right Integer Type**
The most common mistake is using BIGINT for everything "just to be safe." While this works, it wastes storage and can slow down queries because the database has to move more data around.
Here's a practical guide:
**Use SMALLINT when:**
- The value will definitely stay small (age, rating, month number)
- You're storing many millions of rows and want to save space
- Example: A 5-star rating system only needs values 1-5
```sql
CREATE TABLE reviews (
id INTEGER PRIMARY KEY,
rating SMALLINT CHECK (rating BETWEEN 1 AND 5),
review_text TEXT
);
```
**Use INTEGER when:**
- You're not sure how large values might get
- You need a primary key for a table that won't exceed 2 billion rows
- This is your default choice for most integer columns
```sql
CREATE TABLE products (
id INTEGER PRIMARY KEY, -- Will we have 2 billion products? Unlikely.
stock_quantity INTEGER, -- Could be thousands, but not billions
category_id INTEGER -- References another table
);
```
**Use BIGINT when:**
- You know values will exceed 2 billion
- You're storing Unix timestamps in milliseconds
- You're storing file sizes in bytes (files can be huge)
- You're integrating with systems that use 64-bit IDs
```sql
CREATE TABLE events (
id BIGINT PRIMARY KEY, -- High-volume event stream
timestamp_ms BIGINT, -- Unix timestamp in milliseconds
file_size_bytes BIGINT -- File could be gigabytes
);
```
#### Auto-Incrementing Primary Keys
Most tables need a unique identifier for each row. Cognica provides two ways to create auto-incrementing IDs:
**SERIAL (Traditional PostgreSQL Style)**
```sql
CREATE TABLE users (
id SERIAL PRIMARY KEY,
name VARCHAR(100)
);
```
When you use SERIAL:
- Cognica automatically creates a sequence (a counter)
- Each INSERT gets the next number from the sequence
- The column is automatically NOT NULL
**GENERATED AS IDENTITY (Modern SQL Standard)**
```sql
CREATE TABLE users (
id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
name VARCHAR(100)
);
```
There are two variants:
- `GENERATED ALWAYS`: You cannot manually specify a value (safer)
- `GENERATED BY DEFAULT`: Auto-generates unless you provide a value
**Which Should You Use?**
For new projects, we recommend `GENERATED ALWAYS AS IDENTITY`:
- It's the SQL standard (more portable)
- It explicitly prevents you from accidentally providing your own values
- The intent is clearer when reading the schema
For compatibility with existing PostgreSQL code, SERIAL works perfectly fine.
#### Decimal Numbers: When Precision Matters
For numbers with decimal points, you have two fundamentally different choices:
**Floating-Point Types (REAL and DOUBLE PRECISION)**
These store approximate values using binary representation. They're fast because they use hardware floating-point operations.
```sql
CREATE TABLE measurements (
temperature REAL, -- About 6 decimal digits of precision
precise_value DOUBLE PRECISION -- About 15 decimal digits of precision
);
```
**The Critical Warning About Floating-Point**
Floating-point numbers have a well-known quirk that surprises many developers:
```sql
SELECT 0.1 + 0.2;
-- Result: 0.30000000000000004 (not exactly 0.3!)
```
This isn't a bug - it's how binary floating-point works. The number 0.1 cannot be represented exactly in binary, just like 1/3 cannot be represented exactly in decimal.
**When This Matters:**
- Financial calculations: If you're dealing with money, NEVER use floating-point
- Equality comparisons: `WHERE price = 10.5` might not find rows you expect
- Accumulating many values: Errors can compound
**When Floating-Point Is Fine:**
- Scientific measurements (which have inherent uncertainty anyway)
- Percentages where small errors don't matter
- Graphics and game coordinates
- Machine learning features
**NUMERIC/DECIMAL: Exact Decimal Arithmetic**
For financial calculations or any situation where you need exact decimal representation, use NUMERIC:
```sql
CREATE TABLE transactions (
id SERIAL PRIMARY KEY,
amount NUMERIC(15, 2), -- Up to 15 total digits, 2 after decimal
tax_rate NUMERIC(5, 4), -- Like 0.0725 (7.25%)
exchange_rate NUMERIC(20, 10) -- Very precise exchange rate
);
```
The syntax is `NUMERIC(precision, scale)`:
- **Precision**: Total number of significant digits
- **Scale**: Number of digits after the decimal point
**Practical Examples:**
```sql
-- For US dollars and cents
amount NUMERIC(15, 2) -- Handles up to $9,999,999,999,999.99
-- For cryptocurrency with 8 decimal places
btc_amount NUMERIC(20, 8) -- Like Bitcoin's satoshi precision
-- For percentages stored as decimals
percentage NUMERIC(5, 4) -- 0.0000 to 9.9999 (0% to 999.99%)
```
**Performance Note**: NUMERIC is slower than floating-point because it can't use hardware acceleration. For most applications, this doesn't matter. If you're doing heavy numerical computations on millions of values, consider whether you really need exact precision.
### 2.2 Text: Storing Words and Documents
Cognica provides several text types, but the choice is simpler than you might think.
#### VARCHAR(n): Variable-Length with a Limit
VARCHAR stores text up to a specified maximum length:
```sql
CREATE TABLE users (
username VARCHAR(50), -- Max 50 characters
email VARCHAR(255), -- Standard email length limit
country_code VARCHAR(2) -- Always 2 characters, but VARCHAR works fine
);
```
**Why Specify a Length?**
The length limit serves as data validation. If someone tries to insert a 100-character username, they'll get an error instead of bad data entering your database:
```sql
INSERT INTO users (username) VALUES ('this_is_a_very_long_username_that_exceeds_fifty_characters');
-- Error: value too long for type character varying(50)
```
**Common Length Choices:**
| Use Case | Suggested Length | Reasoning |
|----------|------------------|-----------|
| Username | 50 | Most usernames are short; prevents abuse |
| Email | 255 | Technical maximum for email addresses |
| Name (first/last) | 100 | Accommodates long names from any culture |
| Phone number | 20 | Includes country code and formatting |
| URL | 2048 | Common browser limit |
| Short description | 500 | A reasonable paragraph |
#### TEXT: Unlimited Length
TEXT stores text of any length - from a single character to gigabytes:
```sql
CREATE TABLE articles (
title VARCHAR(200),
content TEXT, -- Could be a whole book
raw_html TEXT -- Might be very large
);
```
**VARCHAR vs TEXT: Which to Use?**
Here's a secret: in Cognica (and PostgreSQL), VARCHAR and TEXT have identical storage performance. The only difference is whether you want to enforce a length limit.
**Use VARCHAR(n) when:**
- You have a meaningful maximum length
- You want the database to reject data that's too long
- Example: Email addresses have a technical limit
**Use TEXT when:**
- The length is genuinely unbounded
- You don't know how long the content might be
- Example: Blog post content, user comments
#### CHAR(n): Fixed-Length Text
CHAR always stores exactly n characters, padding with spaces if necessary:
```sql
CREATE TABLE countries (
code CHAR(2) PRIMARY KEY, -- 'US', 'JP', 'GB'
name VARCHAR(100)
);
INSERT INTO countries VALUES ('US', 'United States');
INSERT INTO countries VALUES ('A', 'Invalid'); -- Stored as 'A ' (with space)
```
**Rarely Needed**: In practice, VARCHAR is almost always better. CHAR is occasionally useful for fixed-format codes where you want consistent storage, but VARCHAR works fine for these too.
### 2.3 Dates and Times: Tracking When Things Happen
Date/time handling is notoriously tricky in programming. Cognica provides comprehensive support that, once understood, makes working with dates straightforward.
#### DATE: Just the Calendar Day
DATE stores a date without any time information:
```sql
CREATE TABLE employees (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
birth_date DATE,
hire_date DATE
);
INSERT INTO employees (name, birth_date, hire_date)
VALUES ('Alice', '1990-06-15', '2020-01-10');
```
**Date Formats**: Cognica accepts various formats, but ISO 8601 (YYYY-MM-DD) is strongly recommended for clarity:
```sql
-- All of these work, but use the first one
'2024-12-25' -- ISO 8601 - RECOMMENDED
'December 25, 2024'
'25-Dec-2024'
```
**Getting the Current Date**:
```sql
SELECT CURRENT_DATE; -- Today's date
SELECT CURRENT_DATE - 7; -- 7 days ago (returns DATE)
```
#### TIME: Just the Clock Time
TIME stores a time of day without any date:
```sql
CREATE TABLE business_hours (
day_of_week INTEGER,
open_time TIME,
close_time TIME
);
INSERT INTO business_hours VALUES (1, '09:00:00', '17:30:00'); -- Monday
```
**Time Precision**: You can include fractional seconds:
```sql
'14:30:00' -- 2:30 PM
'14:30:00.123' -- With milliseconds
'14:30:00.123456' -- With microseconds
```
**TIME WITH TIME ZONE**: Exists but is rarely useful. A time without a date has ambiguous timezone meaning (is 3pm PST the same as 3pm PDT? Depends on the date!). If you need timezone-aware times, use TIMESTAMP WITH TIME ZONE.
#### TIMESTAMP: Date and Time Together
TIMESTAMP combines date and time. This is what you'll use most often for recording when things happen.
**Without Timezone (TIMESTAMP)**:
```sql
CREATE TABLE logs (
id SERIAL PRIMARY KEY,
message TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
When you store '2024-12-25 14:30:00', exactly that value is stored and retrieved. No timezone conversion happens.
**With Timezone (TIMESTAMP WITH TIME ZONE or TIMESTAMPTZ)**:
```sql
CREATE TABLE events (
id SERIAL PRIMARY KEY,
event_name VARCHAR(200),
event_time TIMESTAMPTZ
);
```
TIMESTAMPTZ is the most robust choice for recording when things happen:
- Values are stored internally as UTC
- When displayed, they're converted to your session timezone
- You never lose track of the actual instant in time
**Which Should You Use?**
This is a common source of confusion. Here's a clear guideline:
**Use TIMESTAMPTZ (with timezone) when:**
- Recording when something actually happened (audit logs, events)
- Users are in different timezones
- You care about the actual instant in time
- **This should be your default choice**
**Use TIMESTAMP (without timezone) when:**
- Storing "wall clock time" that shouldn't be converted
- Example: "This task is due at 9:00 AM local time" (not a specific instant)
- The timezone is stored separately or always known
**Example of the Difference**:
```sql
-- A user in Seoul schedules a meeting
INSERT INTO meetings (title, meeting_time_tz, meeting_time_local)
VALUES ('Team Sync',
'2024-12-25 10:00:00+09:00', -- TIMESTAMPTZ: specific instant
'2024-12-25 10:00:00'); -- TIMESTAMP: "10 AM"
-- Later, a user in New York views the meeting:
-- TIMESTAMPTZ shows: 2024-12-24 20:00:00-05:00 (correct instant, their timezone)
-- TIMESTAMP shows: 2024-12-25 10:00:00 (same value, potentially confusing)
```
#### INTERVAL: Durations and Differences
INTERVAL represents a span of time:
```sql
-- Various ways to write intervals
INTERVAL '1 day'
INTERVAL '2 hours 30 minutes'
INTERVAL '1 year 6 months'
INTERVAL '1 week'
INTERVAL '-5 days' -- Negative intervals work too
```
**Practical Uses**:
```sql
-- Find events in the next week
SELECT * FROM events WHERE event_time < NOW() + INTERVAL '7 days';
-- Find users who haven't logged in for 30 days
SELECT * FROM users WHERE last_login < NOW() - INTERVAL '30 days';
-- Add business days (simplistic example)
SELECT order_date + INTERVAL '5 days' AS estimated_delivery FROM orders;
-- Calculate age
SELECT name, AGE(birth_date) AS age FROM employees;
-- Returns something like: '34 years 6 months 12 days'
```
### 2.4 Boolean: True, False, and Unknown
The BOOLEAN type stores true/false values with support for NULL (unknown):
```sql
CREATE TABLE tasks (
id SERIAL PRIMARY KEY,
title VARCHAR(200),
is_completed BOOLEAN DEFAULT FALSE,
is_urgent BOOLEAN
);
```
**Accepted Values**:
```sql
-- These all mean TRUE
TRUE, 't', 'true', 'yes', 'on', '1'
-- These all mean FALSE
FALSE, 'f', 'false', 'no', 'off', '0'
-- This means unknown
NULL
```
**The Three-Valued Logic Trap**
SQL uses three-valued logic: TRUE, FALSE, and NULL (unknown). This is one of the most common sources of bugs:
```sql
-- Suppose is_urgent can be NULL
SELECT * FROM tasks WHERE is_urgent = TRUE; -- Only TRUE rows
SELECT * FROM tasks WHERE is_urgent = FALSE; -- Only FALSE rows
SELECT * FROM tasks WHERE is_urgent IS NULL; -- Only NULL rows
-- THIS DOES NOT RETURN ALL ROWS:
SELECT * FROM tasks WHERE is_urgent = TRUE OR is_urgent = FALSE;
-- Rows where is_urgent IS NULL are NOT included!
-- To get all rows including NULL:
SELECT * FROM tasks WHERE is_urgent IS NOT NULL;
SELECT * FROM tasks; -- Or just don't filter
```
**Boolean Expressions in WHERE**:
```sql
-- These are equivalent (and idiomatic)
SELECT * FROM tasks WHERE is_completed = TRUE;
SELECT * FROM tasks WHERE is_completed; -- Preferred
-- These are equivalent
SELECT * FROM tasks WHERE is_completed = FALSE;
SELECT * FROM tasks WHERE NOT is_completed; -- Preferred
-- Be careful: this excludes NULL values!
SELECT * FROM tasks WHERE NOT is_completed;
-- If is_completed is NULL, NOT NULL is still NULL (not TRUE), so row is excluded
```
### 2.5 JSON: Flexible Document Storage
JSON types allow you to store semi-structured data. This is one of Cognica's strengths, bridging SQL and document databases.
#### JSON vs JSONB: Understanding the Difference
Cognica provides two JSON types with different trade-offs:
**JSON**: Stores the exact text you provide
```sql
CREATE TABLE raw_events (
id SERIAL PRIMARY KEY,
data JSON
);
INSERT INTO raw_events (data) VALUES ('{"name": "Alice", "name": "Bob"}');
-- Duplicate keys preserved! When read, you get both.
```
**JSONB**: Stores a processed binary representation
```sql
CREATE TABLE events (
id SERIAL PRIMARY KEY,
data JSONB
);
INSERT INTO events (data) VALUES ('{"name": "Alice", "name": "Bob"}');
-- Duplicate keys resolved! Only "Bob" is kept (last value wins).
```
**Detailed Comparison**:
| Aspect | JSON | JSONB |
|--------|------|-------|
| Storage | Exact text preserved | Binary, decomposed |
| Duplicate keys | Preserved | Last value wins |
| Key order | Preserved | Not preserved (sorted) |
| Whitespace | Preserved | Normalized |
| Indexing | Not supported | GIN indexes supported |
| Query operators | Basic only | Full set including @>, ?, etc. |
| Input speed | Slightly faster | Slightly slower (parsing) |
| Query speed | Slower (must parse) | Much faster |
**When to Use Each**:
**Use JSON when:**
- You need to preserve exact formatting (rare)
- You're storing JSON just to pass it through unchanged
- Duplicate keys have meaning in your application (very rare)
**Use JSONB when:**
- You want to query JSON contents (most common)
- You want to use indexes for JSON queries
- You want containment operators (@>, <@, ?)
- **This should be your default choice**
#### Working with JSON Data
**Inserting JSON**:
```sql
INSERT INTO events (data) VALUES ('{
"user": {
"id": 123,
"name": "Alice",
"email": "alice@example.com"
},
"action": "login",
"timestamp": "2024-12-25T10:30:00Z",
"metadata": {
"ip": "192.168.1.1",
"device": "mobile"
}
}');
```
**Querying JSON** (see [Section 4.5 JSON Functions and Operators](#45-json-functions-and-operators) for complete reference):
```sql
-- Extract a field as JSON
SELECT data->'user' FROM events;
-- Extract a field as text
SELECT data->>'action' FROM events;
-- Extract nested field
SELECT data->'user'->>'name' FROM events;
-- Filter by JSON content
SELECT * FROM events WHERE data->>'action' = 'login';
-- Filter using containment (much faster with index)
SELECT * FROM events WHERE data @> '{"action": "login"}';
```
**Quick Operator Reference**:
| Operator | Description |
|----------|-------------|
| `->` | Extract element as JSON |
| `->>` | Extract element as text |
| `@>` | Contains |
| `?` | Key exists |
| `||` | Concatenate |
| `-` | Delete key/element |
For the complete list of operators, functions, and JSONPath support, see [Section 4.5 JSON Functions and Operators](#45-json-functions-and-operators).
### 2.6 Arrays: Multiple Values in One Column
Arrays let you store multiple values of the same type in a single column:
```sql
CREATE TABLE products (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
tags TEXT[],
prices_history NUMERIC(10,2)[]
);
INSERT INTO products (name, tags, prices_history)
VALUES ('Widget', ARRAY['electronics', 'sale', 'featured'], ARRAY[29.99, 24.99, 19.99]);
```
#### Working with Arrays
**Array Indexing** (1-based, not 0-based!):
```sql
SELECT tags[1] FROM products; -- First element: 'electronics'
SELECT tags[2] FROM products; -- Second element: 'sale'
SELECT tags[0] FROM products; -- NULL (no element 0)
```
**Array Slicing**:
```sql
SELECT tags[1:2] FROM products; -- First two elements
SELECT tags[2:] FROM products; -- From second to end
SELECT prices_history[:2] FROM products; -- First two prices
```
**Checking Array Contents**:
```sql
-- Does the array contain this element?
SELECT * FROM products WHERE 'sale' = ANY(tags);
-- Does the array contain all these elements?
SELECT * FROM products WHERE tags @> ARRAY['sale', 'featured'];
-- Do the arrays have any common elements?
SELECT * FROM products WHERE tags && ARRAY['clearance', 'sale'];
```
#### When to Use Arrays
**Good Use Cases**:
- Tags or categories (small, frequently accessed together)
- A fixed small set of values (RGB colors, coordinates)
- Denormalizing for read performance (careful!)
**Bad Use Cases**:
- Large collections (use a separate table instead)
- Data that needs individual querying or indexing
- Many-to-many relationships (use a junction table)
**Example of When NOT to Use Arrays**:
```sql
-- BAD: Storing order items as an array
CREATE TABLE orders_bad (
id SERIAL PRIMARY KEY,
items TEXT[] -- Hard to query, update, or extend
);
-- GOOD: Separate table for order items
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER
);
CREATE TABLE order_items (
id SERIAL PRIMARY KEY,
order_id INTEGER REFERENCES orders(id),
product_id INTEGER,
quantity INTEGER,
price NUMERIC(10,2)
);
```
### 2.7 UUID: Universally Unique Identifiers
UUIDs are 128-bit identifiers that are unique across all systems without coordination:
```sql
CREATE TABLE distributed_events (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
event_type VARCHAR(50),
payload JSONB
);
-- Generated IDs look like: 'a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11'
```
#### UUID vs Serial: Making the Choice
**Use SERIAL/BIGSERIAL when:**
- Single database server
- Sequential IDs are acceptable or desired
- You want smaller, human-readable IDs
- Optimal index performance matters
**Use UUID when:**
- Distributed systems generating IDs independently
- IDs need to be generated before INSERT (client-side)
- Merging data from multiple databases
- Security (sequential IDs can leak information)
### 2.8 ENUM Types: Custom Value Sets
ENUM types let you define a column with a fixed set of allowed values. This is useful for status fields, categories, and other constrained value sets:
```sql
-- Define an enum type
CREATE TYPE order_status AS ENUM ('pending', 'processing', 'shipped', 'delivered', 'cancelled');
-- Use it in a table
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER NOT NULL,
status order_status NOT NULL DEFAULT 'pending',
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Insert with enum values
INSERT INTO orders (customer_id, status) VALUES (1, 'processing');
-- Invalid values are rejected
INSERT INTO orders (customer_id, status) VALUES (2, 'unknown');
-- Error: invalid input value for enum order_status: "unknown"
```
#### ENUM Ordering
ENUM values have a natural ordering based on their position in the type definition:
```sql
-- 'pending' < 'processing' < 'shipped' < 'delivered' < 'cancelled'
SELECT * FROM orders WHERE status > 'processing'
ORDER BY status; -- Returns 'shipped', 'delivered', 'cancelled'
```
#### Managing ENUM Types
```sql
-- List existing enum types and their values via pg_enum catalog
SELECT enumlabel FROM pg_enum
WHERE enumtypid = (SELECT oid FROM pg_type WHERE typname = 'order_status')
ORDER BY enumsortorder;
-- Drop an enum type
DROP TYPE order_status;
DROP TYPE IF EXISTS order_status;
```
**Limitations vs PostgreSQL:**
- `ALTER TYPE ... ADD VALUE` is not supported (you must recreate the type)
- No `CASCADE` / `RESTRICT` on `DROP TYPE`
### 2.9 Range Types: Representing Intervals of Values
Range types represent a range of values of some element type. They are useful for scheduling, temporal data, and numeric intervals:
```sql
-- Integer range
SELECT int4range(1, 10); -- [1,10) - includes 1, excludes 10
SELECT int4range(1, 10, '[]'); -- [1,10] - includes both endpoints
SELECT int4range(1, 10, '()'); -- (1,10) - excludes both endpoints
-- Date range
SELECT daterange('2024-01-01', '2024-12-31', '[]'); -- Full year 2024
-- Timestamp range
SELECT tsrange('2024-01-01 09:00', '2024-01-01 17:00'); -- Business hours
```
#### Available Range Types
| Range Type | Element Type | Example |
|-----------|-------------|---------|
| `int4range` | INTEGER | `int4range(1, 100)` |
| `int8range` | BIGINT | `int8range(1, 1000000000)` |
| `numrange` | NUMERIC | `numrange(1.5, 9.9)` |
| `tsrange` | TIMESTAMP | `tsrange('2024-01-01', '2024-12-31')` |
| `tstzrange` | TIMESTAMPTZ | `tstzrange('2024-01-01 00:00+00', '2024-12-31 23:59+00')` |
| `daterange` | DATE | `daterange('2024-01-01', '2024-12-31')` |
#### Bound Notation
The third argument to range constructors controls inclusivity:
| Notation | Meaning | Example |
|----------|---------|---------|
| `'[)'` | Include lower, exclude upper (default) | `int4range(1, 5, '[)')` = [1,5) = {1,2,3,4} |
| `'[]'` | Include both | `int4range(1, 5, '[]')` = [1,5] = {1,2,3,4,5} |
| `'()'` | Exclude both | `int4range(1, 5, '()')` = (1,5) = {2,3,4} |
| `'(]'` | Exclude lower, include upper | `int4range(1, 5, '(]')` = (1,5] = {2,3,4,5} |
#### Range Operators
```sql
-- Overlap: do two ranges share any values?
SELECT int4range(1, 10) && int4range(5, 15); -- true (overlap at 5-9)
-- Contains: does the range contain a value or another range?
SELECT int4range(1, 10) @> 5; -- true
SELECT int4range(1, 10) @> int4range(3, 7); -- true
-- Contained by
SELECT 5 <@ int4range(1, 10); -- true
-- Strictly left/right of
SELECT int4range(1, 5) << int4range(10, 20); -- true (strictly left)
SELECT int4range(10, 20) >> int4range(1, 5); -- true (strictly right)
-- Adjacent (ranges touch but don't overlap)
SELECT int4range(1, 5) -|- int4range(5, 10); -- true
```
#### Range Functions
```sql
-- Check if range is empty
SELECT isempty(int4range(1, 1)); -- true (empty range)
SELECT isempty(int4range(1, 5)); -- false
-- Check boundary inclusivity
SELECT lower_inc(int4range(1, 10)); -- true (lower bound included)
SELECT upper_inc(int4range(1, 10)); -- false (upper bound excluded by default)
-- Check for infinite bounds
SELECT lower_inf(int4range(NULL, 10)); -- true (no lower bound)
SELECT upper_inf(int4range(1, NULL)); -- true (no upper bound)
-- Merge two ranges
SELECT range_merge(int4range(1, 5), int4range(3, 10)); -- [1,10)
```
#### Practical Examples
**Room Booking System:**
```sql
CREATE TABLE room_bookings (
id SERIAL PRIMARY KEY,
room_id INTEGER NOT NULL,
booking_period tsrange NOT NULL,
booked_by TEXT NOT NULL
);
-- Find overlapping bookings (double-booking detection)
SELECT a.id, b.id
FROM room_bookings a, room_bookings b
WHERE a.room_id = b.room_id
AND a.id < b.id
AND a.booking_period && b.booking_period;
```
**Price History with Effective Dates:**
```sql
CREATE TABLE price_history (
product_id INTEGER NOT NULL,
price NUMERIC(10,2) NOT NULL,
effective_period daterange NOT NULL
);
-- Find the current price
SELECT price FROM price_history
WHERE product_id = 42
AND effective_period @> CURRENT_DATE;
```
### 2.10 Type Casting: Converting Between Types
You'll often need to convert values from one type to another.
**PostgreSQL-Style Cast (::)**:
```sql
SELECT '42'::INTEGER; -- String to integer
SELECT 3.7::INTEGER; -- Decimal to integer (truncates to 3)
SELECT 123::TEXT; -- Number to string
SELECT '2024-12-25'::DATE; -- String to date
SELECT NOW()::DATE; -- Timestamp to date (drops time)
```
**SQL Standard CAST**:
```sql
SELECT CAST('42' AS INTEGER);
SELECT CAST(3.7 AS INTEGER);
SELECT CAST(123 AS TEXT);
```
Both forms are equivalent. Use whichever you find more readable.
**Implicit vs Explicit Casting**:
Cognica automatically casts types when safe and unambiguous:
```sql
SELECT 1 + 2.5; -- Integer automatically becomes numeric
SELECT 'Hello ' || 123; -- Integer automatically becomes text
SELECT 10 > 5.5; -- Integer automatically becomes numeric for comparison
```
Explicit casting is required when:
- The conversion might lose data
- The conversion is ambiguous
```sql
SELECT 3.99::INTEGER; -- Must be explicit (loses .99)
SELECT '123'::INTEGER; -- String to number needs explicit cast
```
---
## Chapter 3: Operators - The Building Blocks of Expressions
Operators are the verbs of SQL expressions - they specify what action to perform on your data. While functions transform data through named procedures (like `UPPER()` or `SUM()`), operators provide a more concise symbolic notation for common operations. The expression `price * quantity` is more readable than a hypothetical `MULTIPLY(price, quantity)` function would be.
Understanding operators deeply is crucial because they appear everywhere in SQL: in SELECT clauses for calculations, in WHERE clauses for filtering, in JOIN conditions for matching rows, and in ORDER BY clauses for sorting logic. A solid grasp of operator behavior - especially the subtle edge cases around NULL values and type coercion - will prevent countless debugging sessions.
### The Type System and Operator Behavior
Operators in SQL are type-aware. The `+` operator behaves differently depending on what types you give it: with integers, it performs integer addition; with timestamps and intervals, it adds time. When operands have different types, Cognica follows PostgreSQL's type coercion rules to determine the result type. Understanding these rules helps you predict what type your expressions will produce and avoid subtle bugs.
The most common surprise involves integer division: when both operands are integers, division produces an integer by truncating toward zero. This behavior is mathematically consistent but often unexpected by those coming from languages where division always produces floating-point results.
### NULL: The Value That Changes Everything
Perhaps the most important conceptual point about SQL operators is how they interact with NULL. In standard SQL, NULL represents "unknown" - not zero, not empty, not false, but truly unknown. This seemingly simple concept has profound implications: any arithmetic operation involving NULL produces NULL, any comparison with NULL produces NULL (not true or false), and most logical operations propagate NULL in ways that can surprise the unwary.
This NULL propagation is mathematically sound - if you do not know a value, you cannot know what happens when you add 5 to it or compare it to something. However, it means that seemingly equivalent expressions can produce different results when NULLs are involved. The expression `WHERE status <> 'deleted'` does NOT return rows where status is NULL, because NULL <> 'deleted' evaluates to NULL, not true.
### 3.1 Arithmetic Operators: Doing Math
**The Basic Four (Plus Modulo)**:
| Operator | Name | Example | Result |
|----------|------|---------|--------|
| `+` | Addition | `5 + 3` | `8` |
| `-` | Subtraction | `10 - 4` | `6` |
| `*` | Multiplication | `6 * 7` | `42` |
| `/` | Division | `20 / 4` | `5` |
| `%` | Modulo (remainder) | `17 % 5` | `2` |
**The Integer Division Gotcha**:
This trips up almost everyone at first:
```sql
SELECT 10 / 3; -- Result: 3 (not 3.333!)
SELECT 10 / 4; -- Result: 2 (not 2.5!)
```
When both operands are integers, division returns an integer by truncating toward zero. To get decimal results:
```sql
SELECT 10.0 / 3; -- Result: 3.333... (one operand is decimal)
SELECT 10 / 3.0; -- Result: 3.333... (one operand is decimal)
SELECT 10::NUMERIC / 3; -- Result: 3.333... (explicit cast)
SELECT CAST(10 AS NUMERIC) / 3; -- Same thing
```
**Practical Use of Modulo**:
The modulo operator returns the remainder after division. It's surprisingly useful:
```sql
-- Check if a number is even or odd
SELECT
n,
CASE WHEN n % 2 = 0 THEN 'even' ELSE 'odd' END
FROM generate_series(1, 10) AS n;
-- Distribute items into buckets
SELECT id, id % 4 AS bucket FROM items; -- Buckets 0, 1, 2, 3
-- Check if year is leap year (simplified)
SELECT year, (year % 4 = 0 AND (year % 100 <> 0 OR year % 400 = 0)) AS is_leap
FROM generate_series(2020, 2030) AS year;
```
### 3.2 Comparison Operators: Testing Relationships
Comparison operators return TRUE, FALSE, or NULL.
**The Basics**:
| Operator | Meaning | Example |
|----------|---------|---------|
| `=` | Equal to | `status = 'active'` |
| `<>` or `!=` | Not equal to | `status <> 'deleted'` |
| `<` | Less than | `age < 18` |
| `>` | Greater than | `salary > 50000` |
| `<=` | Less than or equal | `quantity <= 0` |
| `>=` | Greater than or equal | `rating >= 4.0` |
**BETWEEN: Testing Ranges**
BETWEEN checks if a value falls within a range, inclusive on both ends:
```sql
-- These are exactly equivalent
SELECT * FROM products WHERE price BETWEEN 10 AND 50;
SELECT * FROM products WHERE price >= 10 AND price <= 50;
```
**Important**: BETWEEN is inclusive. Both boundary values are included. If you need exclusive boundaries:
```sql
-- Exclusive on upper bound
SELECT * FROM products WHERE price >= 10 AND price < 50;
-- Exclusive on both bounds
SELECT * FROM products WHERE price > 10 AND price < 50;
```
**NOT BETWEEN**:
```sql
SELECT * FROM products WHERE price NOT BETWEEN 10 AND 50;
-- Equivalent to: price < 10 OR price > 50
```
**Working with NULLs: The Most Important Thing to Understand**
NULL represents "unknown" or "missing" data. It's not zero, not empty string, not false - it's the absence of a value. This has profound implications:
```sql
-- ALL of these return NULL (unknown), not TRUE or FALSE:
SELECT NULL = NULL; -- NULL (we don't know if two unknowns are equal)
SELECT NULL <> NULL; -- NULL
SELECT NULL < 10; -- NULL
SELECT NULL > 10; -- NULL
SELECT 5 = NULL; -- NULL
SELECT 5 <> NULL; -- NULL
```
**Testing for NULL**:
```sql
-- WRONG: This finds nothing (NULL = NULL is NULL, not TRUE)
SELECT * FROM users WHERE middle_name = NULL;
-- RIGHT: Use IS NULL
SELECT * FROM users WHERE middle_name IS NULL;
-- RIGHT: Use IS NOT NULL
SELECT * FROM users WHERE email IS NOT NULL;
```
**IS DISTINCT FROM: NULL-Safe Comparison**
Sometimes you want NULL to be treated as a comparable value:
```sql
-- Regular comparison (has problems with NULL)
SELECT * FROM audit WHERE old_value <> new_value;
-- Misses rows where one value is NULL!
-- NULL-safe comparison
SELECT * FROM audit WHERE old_value IS DISTINCT FROM new_value;
-- Includes rows where one is NULL and one isn't
-- Detailed behavior:
SELECT 1 IS DISTINCT FROM 1; -- FALSE (same value)
SELECT 1 IS DISTINCT FROM 2; -- TRUE (different values)
SELECT 1 IS DISTINCT FROM NULL; -- TRUE (value vs unknown)
SELECT NULL IS DISTINCT FROM NULL; -- FALSE (both unknown = same)
```
### 3.3 Logical Operators: Combining Conditions
**AND, OR, NOT**:
```sql
-- AND: Both conditions must be true
SELECT * FROM products WHERE price < 50 AND category = 'Electronics';
-- OR: At least one condition must be true
SELECT * FROM users WHERE role = 'admin' OR role = 'moderator';
-- NOT: Inverts the condition
SELECT * FROM tasks WHERE NOT is_completed;
```
**Three-Valued Logic in Practice**:
Because NULL means "unknown," logical operations with NULL can be surprising:
```sql
-- AND with NULL
TRUE AND NULL = NULL -- Might be true if NULL were true, might be false
FALSE AND NULL = FALSE -- Definitely false regardless of NULL
NULL AND NULL = NULL -- Unknown
-- OR with NULL
TRUE OR NULL = TRUE -- Definitely true regardless of NULL
FALSE OR NULL = NULL -- Might be true if NULL were true
NULL OR NULL = NULL -- Unknown
```
**Practical Implication**:
```sql
-- Find incomplete tasks
SELECT * FROM tasks WHERE NOT is_completed;
-- If is_completed is NULL, NOT NULL = NULL, and the row is NOT included
-- This might not be what you want!
-- To include NULLs:
SELECT * FROM tasks WHERE NOT is_completed OR is_completed IS NULL;
SELECT * FROM tasks WHERE is_completed IS NOT TRUE; -- Cleaner
```
**Operator Precedence**:
NOT binds tighter than AND, which binds tighter than OR:
```sql
-- This might not do what you expect:
SELECT * FROM users WHERE status = 'active' OR role = 'admin' AND verified;
-- Interpreted as: status = 'active' OR (role = 'admin' AND verified)
-- Use parentheses to be clear:
SELECT * FROM users WHERE (status = 'active' OR role = 'admin') AND verified;
```
**Best Practice**: Always use parentheses when mixing AND and OR. It makes your intent clear and prevents bugs.
### 3.4 Pattern Matching: Finding Text Patterns
#### LIKE: Simple Pattern Matching
LIKE uses two wildcards:
- `%` matches any sequence of characters (including empty)
- `_` matches exactly one character
```sql
-- Names starting with 'John'
SELECT * FROM users WHERE name LIKE 'John%';
-- Matches: John, Johnny, John Smith, Johnathan
-- Email addresses from gmail
SELECT * FROM users WHERE email LIKE '%@gmail.com';
-- Matches: alice@gmail.com, bob.smith@gmail.com
-- Three-character codes
SELECT * FROM products WHERE code LIKE '___';
-- Matches: ABC, X12, a1b (exactly 3 characters)
-- Specific pattern: letter-digit-digit-digit
SELECT * FROM products WHERE sku LIKE 'A___';
-- Matches: A123, A999, A000 (A followed by exactly 3 characters)
```
**Escaping Wildcards**:
What if you need to search for literal % or _?
```sql
-- Find products with '50%' in the name
SELECT * FROM products WHERE name LIKE '%50\%%' ESCAPE '\';
-- The ESCAPE clause tells Cognica that \ is the escape character
-- \% means literal %, not the wildcard
```
#### ILIKE: Case-Insensitive LIKE
```sql
SELECT * FROM users WHERE name ILIKE 'john%';
-- Matches: John, JOHN, john, JoHn, etc.
```
#### Performance Note
LIKE patterns starting with `%` cannot use indexes efficiently:
```sql
-- CAN use an index on name (if one exists)
SELECT * FROM users WHERE name LIKE 'John%';
-- CANNOT use an index (must scan all rows)
SELECT * FROM users WHERE name LIKE '%John%';
```
For frequent substring searches, consider Full-Text Search (Chapter 11).
#### Regular Expressions: Full Pattern Power
For complex patterns, use POSIX regular expressions:
| Operator | Case Sensitive | Meaning |
|----------|----------------|---------|
| `~` | Yes | Matches regex |
| `!~` | Yes | Does not match regex |
| `~*` | No | Matches regex (case-insensitive) |
| `!~*` | No | Does not match regex (case-insensitive) |
```sql
-- Validate email format
SELECT * FROM users
WHERE email ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$';
-- Find phone numbers (various formats)
SELECT * FROM contacts
WHERE phone ~ '^\d{3}[-.\s]?\d{3}[-.\s]?\d{4}$';
-- Case-insensitive search for variations of "color"/"colour"
SELECT * FROM products WHERE description ~* 'colou?r';
```
#### SIMILAR TO: SQL Standard Pattern Matching
SIMILAR TO provides SQL standard pattern matching, combining LIKE's `%` and `_` wildcards with regex-style character classes and alternation. It sits between LIKE (simple wildcards only) and POSIX regex (full regex power).
| Feature | LIKE | SIMILAR TO | POSIX Regex (~) |
|---------|------|------------|-----------------|
| `%` wildcard | Yes | Yes | No (use `.*`) |
| `_` wildcard | Yes | Yes | No (use `.`) |
| Character classes `[...]` | No | Yes | Yes |
| Alternation `\|` | No | Yes | Yes |
| Quantifiers `+`, `*`, `?` | No | Yes | Yes |
| Case-insensitive variant | ILIKE | No | `~*` |
```sql
-- Match product codes: letter followed by 3 digits
SELECT * FROM products WHERE sku SIMILAR TO '[A-Z][0-9]{3}';
-- Matches: A123, B456, Z999
-- Match multiple patterns with alternation
SELECT * FROM users WHERE role SIMILAR TO '(admin|manager|supervisor)';
-- Matches: admin, manager, supervisor
-- Combine wildcards with character classes
SELECT * FROM files WHERE name SIMILAR TO '%\.(jpg|png|gif)';
-- Matches: photo.jpg, logo.png, icon.gif
-- Quantifiers: one or more digits followed by optional letter
SELECT * FROM codes WHERE value SIMILAR TO '[0-9]+[A-Z]?';
-- Matches: 123, 456A, 7B
```
**How SIMILAR TO Differs from LIKE and Regex:**
```sql
-- These three queries find names starting with 'A' or 'B':
-- LIKE: requires two conditions
SELECT * FROM users WHERE name LIKE 'A%' OR name LIKE 'B%';
-- SIMILAR TO: uses alternation
SELECT * FROM users WHERE name SIMILAR TO '(A|B)%';
-- POSIX regex: full regex syntax
SELECT * FROM users WHERE name ~ '^[AB]';
```
**Performance Note**: Internally, SIMILAR TO patterns are compiled into POSIX regular expressions. Performance is equivalent to the `~` operator. For simple prefix matching, LIKE with an index is faster.
### 3.5 The IN Operator: Testing Set Membership
IN checks if a value matches any value in a list:
```sql
-- Simple list
SELECT * FROM products WHERE category IN ('Electronics', 'Books', 'Toys');
-- Equivalent to:
SELECT * FROM products
WHERE category = 'Electronics'
OR category = 'Books'
OR category = 'Toys';
-- NOT IN
SELECT * FROM products WHERE category NOT IN ('Clearance', 'Discontinued');
```
**IN with Subquery**:
```sql
-- Find users who have placed orders
SELECT * FROM users WHERE id IN (SELECT DISTINCT user_id FROM orders);
-- Find products that have never been ordered
SELECT * FROM products WHERE id NOT IN (SELECT DISTINCT product_id FROM order_items);
```
**NULL Warning with NOT IN**:
This is a subtle but dangerous gotcha:
```sql
-- Suppose some orders have NULL user_id
SELECT * FROM users WHERE id NOT IN (SELECT user_id FROM orders);
-- If ANY user_id is NULL, this returns ZERO rows!
-- Why? Because "id NOT IN (1, 2, NULL)" means:
-- id <> 1 AND id <> 2 AND id <> NULL
-- And "id <> NULL" is NULL (unknown), making the whole thing NULL
-- Solution: Filter out NULLs
SELECT * FROM users
WHERE id NOT IN (SELECT user_id FROM orders WHERE user_id IS NOT NULL);
-- Or use NOT EXISTS (generally safer):
SELECT * FROM users u
WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id);
```
### 3.6 The Power Operator
The `^` operator computes exponentiation:
```sql
SELECT 2 ^ 10; -- 1024 (2 to the power of 10)
SELECT 3 ^ 3; -- 27
SELECT 9 ^ 0.5; -- 3.0 (square root)
SELECT 2 ^ -1; -- 0.5
-- Practical: compound interest calculation
SELECT principal * (1 + rate) ^ years AS future_value
FROM investments;
```
The `^` operator is equivalent to the `power()` function:
```sql
SELECT power(2, 10); -- 1024 (same as 2 ^ 10)
```
### 3.7 Bitwise Operators
Bitwise operators work on integer values at the bit level:
| Operator | Name | Example | Result |
|----------|------|---------|--------|
| `&` | Bitwise AND | `91 & 15` | `11` |
| `\|` | Bitwise OR | `32 \| 3` | `35` |
| `#` | Bitwise XOR | `17 # 5` | `20` |
| `~` | Bitwise NOT | `~1` | `-2` |
| `<<` | Left Shift | `1 << 4` | `16` |
| `>>` | Right Shift | `8 >> 2` | `2` |
```sql
-- Check if a specific permission bit is set
SELECT user_id, permissions
FROM users
WHERE permissions & 4 = 4; -- Check bit 2 (read permission)
-- Set a permission bit
UPDATE users SET permissions = permissions | 8 WHERE user_id = 1; -- Set bit 3
-- Clear a permission bit
UPDATE users SET permissions = permissions & ~8 WHERE user_id = 1; -- Clear bit 3
-- Toggle a permission bit
UPDATE users SET permissions = permissions # 8 WHERE user_id = 1; -- Toggle bit 3
```
### 3.8 Range Operators
Range operators compare and test range values (see [Section 2.9](#29-range-types-representing-intervals-of-values) for range type details):
| Operator | Name | Example | Result |
|----------|------|---------|--------|
| `&&` | Overlap | `int4range(1,5) && int4range(3,8)` | `true` |
| `@>` | Contains | `int4range(1,10) @> 5` | `true` |
| `<@` | Contained by | `5 <@ int4range(1,10)` | `true` |
| `<<` | Strictly left of | `int4range(1,5) << int4range(10,20)` | `true` |
| `>>` | Strictly right of | `int4range(10,20) >> int4range(1,5)` | `true` |
| `&<` | Does not extend right of | `int4range(1,5) &< int4range(3,10)` | `true` |
| `&>` | Does not extend left of | `int4range(3,10) &> int4range(1,5)` | `true` |
| `-\|-` | Adjacent to | `int4range(1,5) -\|- int4range(5,10)` | `true` |
```sql
-- Find all events that overlap with a given time window
SELECT * FROM events
WHERE event_period && tsrange('2024-06-01', '2024-06-30');
-- Check if a value falls within a range
SELECT * FROM price_ranges
WHERE numrange(min_price, max_price, '[]') @> 29.99;
```
### 3.9 Array Operators
Array operators perform set-like operations on arrays:
| Operator | Name | Example | Result |
|----------|------|---------|--------|
| `@>` | Contains | `ARRAY[1,2,3] @> ARRAY[2,3]` | `true` |
| `<@` | Contained by | `ARRAY[2,3] <@ ARRAY[1,2,3]` | `true` |
| `&&` | Overlap (any common elements) | `ARRAY[1,2] && ARRAY[2,3]` | `true` |
| `\|\|` | Concatenation | `ARRAY[1,2] \|\| ARRAY[3,4]` | `{1,2,3,4}` |
```sql
-- Find products with any of these tags
SELECT * FROM products WHERE tags && ARRAY['sale', 'clearance'];
-- Find products that have ALL required tags
SELECT * FROM products WHERE tags @> ARRAY['electronics', 'featured'];
-- ANY/ALL with arrays
SELECT * FROM products WHERE 'sale' = ANY(tags); -- tag exists
SELECT * FROM products WHERE 'sale' <> ALL(tags); -- tag does not exist
```
---
## Chapter 4: Functions - Transforming Your Data
Functions are the workhorses of SQL expressions. While operators combine values using symbolic notation (`+`, `=`, `||`), functions provide named operations that transform, calculate, format, and extract information from your data. Understanding how functions behave is essential for writing correct and efficient queries.
### The Anatomy of a SQL Function
Every SQL function has three key characteristics that affect how you use it:
**1. Determinism**: A deterministic function always returns the same output for the same inputs. `UPPER('hello')` will always return `'HELLO'`. Most functions in this chapter are deterministic. Non-deterministic functions like `NOW()` or `RANDOM()` return different values on each call, which has implications for query optimization and indexing. You cannot create an expression index on a non-deterministic function because the index would become stale immediately.
**2. NULL Propagation**: The vast majority of SQL functions propagate NULL. If any argument is NULL, the result is NULL. This is not a bug but a fundamental design principle: NULL represents "unknown," and an operation on unknown data produces an unknown result. For example, `LENGTH(NULL)` returns NULL, not 0 or an error. Functions that explicitly handle NULL (like `COALESCE`, `NULLIF`, `CONCAT`) are the exceptions, and this chapter notes them.
**3. Type Sensitivity**: Functions care about data types. `SUBSTRING` works on strings, `ROUND` works on numbers, `DATE_TRUNC` works on timestamps. Passing the wrong type usually results in an error, though implicit casts may occur in some cases. Understanding your data types helps you choose the right function.
### Categories of Functions
This chapter covers **scalar functions**, which operate on individual values and return a single value. For each row processed, a scalar function is evaluated once per invocation in the SELECT list.
Other function categories exist but are covered in dedicated chapters:
- **Aggregate functions** (Chapter 5): Operate on sets of rows, returning one result per group
- **Window functions** (Chapter 6): Operate across rows related to the current row without collapsing them
### Performance Considerations
Functions in the SELECT list execute once per row in the result set. In a WHERE clause, they may execute many times during filtering. Complex functions on large tables can become performance bottlenecks. When performance matters:
- Avoid functions on columns in WHERE clauses when possible (they prevent index usage)
- Consider expression indexes for frequently-filtered function expressions
- Pre-compute expensive transformations and store them in columns
With these principles in mind, let us explore the most useful functions organized by the type of data they work with.
### 4.1 String Functions: Working with Text
#### Getting String Information
**LENGTH** - Count characters:
```sql
SELECT LENGTH('Hello'); -- 5
SELECT LENGTH(''); -- 0
SELECT LENGTH('Hello World'); -- 11 (space counts)
SELECT LENGTH(NULL); -- NULL (not 0!)
```
**CHAR_LENGTH** / **CHARACTER_LENGTH** - Count characters (alias for LENGTH):
```sql
SELECT CHAR_LENGTH('Hello'); -- 5
SELECT CHARACTER_LENGTH('Hello'); -- 5
```
**BIT_LENGTH** - Count bits:
```sql
SELECT BIT_LENGTH('Hello'); -- 40 (5 characters x 8 bits)
```
**OCTET_LENGTH** - Count bytes:
```sql
SELECT OCTET_LENGTH('Hello'); -- 5 (ASCII: 1 byte per character)
SELECT OCTET_LENGTH('Hello'); -- More if using special characters
```
These three length functions distinguish between characters, bits, and bytes — a distinction that matters for multibyte character encodings like UTF-8.
**POSITION** and **STRPOS** - Find substring location:
```sql
SELECT POSITION('world' IN 'Hello world'); -- 7 (1-based position)
SELECT STRPOS('Hello world', 'world'); -- 7 (same, different syntax)
SELECT POSITION('xyz' IN 'Hello world'); -- 0 (not found)
```
#### Modifying Strings
**UPPER, LOWER, INITCAP** - Change case:
```sql
SELECT UPPER('hello World'); -- 'HELLO WORLD'
SELECT LOWER('Hello WORLD'); -- 'hello world'
SELECT INITCAP('hello world'); -- 'Hello World' (capitalize each word)
SELECT INITCAP('HELLO WORLD'); -- 'Hello World' (lowercases rest)
```
**TRIM, LTRIM, RTRIM, BTRIM** - Remove characters:
```sql
SELECT TRIM(' hello '); -- 'hello' (remove spaces from both ends)
SELECT LTRIM(' hello '); -- 'hello ' (left only)
SELECT RTRIM(' hello '); -- ' hello' (right only)
SELECT BTRIM('xxhelloxx', 'x'); -- 'hello' (remove 'x' from both ends)
SELECT TRIM(BOTH 'x' FROM 'xxhelloxx'); -- 'hello' (SQL standard syntax)
```
**LPAD, RPAD** - Pad strings to a length:
```sql
SELECT LPAD('42', 5, '0'); -- '00042' (left pad with zeros)
SELECT RPAD('42', 5, '*'); -- '42***' (right pad with asterisks)
SELECT LPAD('12345', 3, '0'); -- '12345' (no truncation if already longer)
```
Common use: formatting numbers for display, creating fixed-width output.
**REVERSE** - Reverse a string:
```sql
SELECT REVERSE('Hello'); -- 'olleH'
```
**REPEAT** - Repeat a string:
```sql
SELECT REPEAT('ab', 3); -- 'ababab'
SELECT REPEAT('*', 10); -- '**********'
```
**OVERLAY** - Replace a substring at a given position:
```sql
SELECT OVERLAY('Txxxxas' PLACING 'hom' FROM 2 FOR 4); -- 'Thomas'
SELECT OVERLAY('abcdef' PLACING '123' FROM 3); -- 'ab123f'
```
**TRANSLATE** - Character-by-character replacement:
```sql
SELECT TRANSLATE('12345', '143', 'ax'); -- 'a2x5' (1->a, 4->x, 3->removed)
```
Unlike REPLACE which substitutes entire substrings, TRANSLATE maps individual characters.
**REPLACE** - Replace all occurrences:
```sql
SELECT REPLACE('Hello World', 'World', 'Cognica'); -- 'Hello Cognica'
SELECT REPLACE('banana', 'a', 'o'); -- 'bonono' (all a's replaced)
```
**SUBSTRING** - Extract part of a string:
```sql
SELECT SUBSTRING('Hello World' FROM 1 FOR 5); -- 'Hello'
SELECT SUBSTRING('Hello World', 7, 5); -- 'World' (shorter syntax)
SELECT SUBSTRING('Hello World', 7); -- 'World' (to end)
SELECT LEFT('Hello World', 5); -- 'Hello' (first N chars)
SELECT RIGHT('Hello World', 5); -- 'World' (last N chars)
```
#### Building Strings
**Concatenation with ||**:
```sql
SELECT 'Hello' || ' ' || 'World'; -- 'Hello World'
SELECT first_name || ' ' || last_name AS full_name FROM users;
```
**Warning**: NULL concatenated with anything is NULL:
```sql
SELECT 'Hello' || NULL; -- NULL (not 'Hello'!)
```
**CONCAT** - NULL-safe concatenation:
```sql
SELECT CONCAT('Hello', NULL, 'World'); -- 'HelloWorld' (NULL ignored)
SELECT CONCAT(first_name, ' ', last_name) FROM users; -- Safe if names might be NULL
```
**CONCAT_WS** - Concatenate with separator:
```sql
SELECT CONCAT_WS(', ', 'Alice', 'Bob', 'Charlie'); -- 'Alice, Bob, Charlie'
SELECT CONCAT_WS(', ', 'Alice', NULL, 'Charlie'); -- 'Alice, Charlie' (NULL skipped)
```
**FORMAT** - Build strings with placeholders:
```sql
SELECT FORMAT('Hello, %s!', 'World'); -- 'Hello, World!'
SELECT FORMAT('User %s has %s orders', name, order_count) FROM users;
SELECT FORMAT('%-10s', 'test'); -- 'test ' (left-aligned, 10 chars)
```
#### Character Functions
**ASCII** - Get the numeric code of the first character:
```sql
SELECT ASCII('A'); -- 65
SELECT ASCII('a'); -- 97
```
**CHR** - Get the character for a numeric code:
```sql
SELECT CHR(65); -- 'A'
SELECT CHR(97); -- 'a'
```
#### Hashing and Encoding
**MD5** - Compute the MD5 hash of a string:
```sql
SELECT MD5('hello'); -- '5d41402abc4b2a76b9719d911017c592'
```
**TO_HEX** - Convert an integer to its hexadecimal representation:
```sql
SELECT TO_HEX(255); -- 'ff'
SELECT TO_HEX(4096); -- '1000'
```
**ENCODE** / **DECODE** - Encode or decode binary data:
```sql
SELECT ENCODE('hello'::bytea, 'base64'); -- 'aGVsbG8='
SELECT DECODE('aGVsbG8=', 'base64'); -- '\x68656c6c6f'
SELECT ENCODE('hello'::bytea, 'hex'); -- '68656c6c6f'
```
#### Prefix and Suffix Testing
**STARTS_WITH** - Test if a string starts with a prefix:
```sql
SELECT STARTS_WITH('Hello World', 'Hello'); -- true
SELECT STARTS_WITH('Hello World', 'World'); -- false
```
**ENDS_WITH** - Test if a string ends with a suffix:
```sql
SELECT ENDS_WITH('Hello World', 'World'); -- true
SELECT ENDS_WITH('Hello World', 'Hello'); -- false
```
#### Quoting Functions
These functions are useful for constructing dynamic SQL safely:
**QUOTE_IDENT** - Quote an identifier if needed:
```sql
SELECT QUOTE_IDENT('my_table'); -- 'my_table' (no quoting needed)
SELECT QUOTE_IDENT('user name'); -- '"user name"' (quoting needed for space)
```
**QUOTE_LITERAL** - Quote a value as a SQL literal:
```sql
SELECT QUOTE_LITERAL('hello'); -- '''hello'''
SELECT QUOTE_LITERAL('it''s a test'); -- '''it''s a test'''
```
**QUOTE_NULLABLE** - Like QUOTE_LITERAL, but returns the string 'NULL' for NULL input:
```sql
SELECT QUOTE_NULLABLE('hello'); -- '''hello'''
SELECT QUOTE_NULLABLE(NULL); -- 'NULL'
```
#### Regular Expressions
**REGEXP_MATCH** - Return the first match of a POSIX regular expression:
```sql
SELECT REGEXP_MATCH('foobarbaz', 'b(.)r');
-- Result: {a} (captured group)
```
**REGEXP_REPLACE** - Replace substrings matching a regular expression:
```sql
SELECT REGEXP_REPLACE('Hello World 123', '[0-9]+', 'NUM');
-- Result: 'Hello World NUM'
SELECT REGEXP_REPLACE('abc123def456', '[0-9]+', '#', 'g');
-- Result: 'abc#def#' (global replacement with 'g' flag)
```
**REGEXP_SPLIT_TO_ARRAY** - Split a string using a regular expression:
```sql
SELECT REGEXP_SPLIT_TO_ARRAY('one-two--three', '-+');
-- Result: {one,two,three}
```
#### Splitting Strings
**SPLIT_PART** - Get one part of a delimited string:
```sql
SELECT SPLIT_PART('a,b,c,d', ',', 1); -- 'a' (first part)
SELECT SPLIT_PART('a,b,c,d', ',', 2); -- 'b' (second part)
SELECT SPLIT_PART('a,b,c,d', ',', 5); -- '' (empty if not enough parts)
```
**STRING_TO_ARRAY** - Split into array:
```sql
SELECT STRING_TO_ARRAY('a,b,c', ','); -- {a,b,c}
SELECT STRING_TO_ARRAY('a|b|c', '|'); -- {a,b,c}
```
**ARRAY_TO_STRING** - Join array into string:
```sql
SELECT ARRAY_TO_STRING(ARRAY['a', 'b', 'c'], ', '); -- 'a, b, c'
```
### 4.2 Mathematical Functions
#### Rounding Functions
```sql
SELECT ROUND(4.567); -- 5 (round to nearest integer)
SELECT ROUND(4.5); -- 5 (rounds up at .5)
SELECT ROUND(4.567, 2); -- 4.57 (round to 2 decimal places)
SELECT ROUND(4.565, 2); -- 4.57 (banker's rounding at exactly .5)
SELECT TRUNC(4.567); -- 4 (truncate toward zero)
SELECT TRUNC(-4.567); -- -4 (toward zero, not toward negative infinity)
SELECT TRUNC(4.567, 2); -- 4.56 (truncate to 2 decimal places)
SELECT CEIL(4.1); -- 5 (smallest integer >= value)
SELECT CEIL(-4.1); -- -4
SELECT FLOOR(4.9); -- 4 (largest integer <= value)
SELECT FLOOR(-4.9); -- -5
```
**Which to Use?**
- **ROUND**: When you want the closest value
- **TRUNC**: When you want to simply remove decimal places
- **CEIL**: When you need to round up (e.g., number of pages needed)
- **FLOOR**: When you need to round down (e.g., completed whole units)
#### Other Mathematical Functions
```sql
SELECT ABS(-42); -- 42 (absolute value)
SELECT SIGN(-42); -- -1 (returns -1, 0, or 1)
SELECT POWER(2, 10); -- 1024 (2^10)
SELECT SQRT(16); -- 4 (square root)
SELECT CBRT(27); -- 3 (cube root)
SELECT MOD(17, 5); -- 2 (same as 17 % 5)
SELECT DIV(17, 5); -- 3 (integer quotient, truncated)
SELECT FACTORIAL(5); -- 120 (5! = 5*4*3*2*1)
SELECT GCD(12, 8); -- 4 (greatest common divisor)
SELECT LCM(12, 8); -- 24 (least common multiple)
SELECT GREATEST(1, 5, 3, 9, 2); -- 9 (maximum of list)
SELECT LEAST(1, 5, 3, 9, 2); -- 1 (minimum of list)
SELECT RANDOM(); -- Random number between 0 and 1
SELECT PI(); -- 3.14159265358979...
```
#### Logarithmic and Exponential Functions
```sql
SELECT LN(2.718281828); -- 1.0 (natural logarithm)
SELECT LOG(100); -- 2.0 (base-10 logarithm, 1 arg)
SELECT LOG(2, 8); -- 3.0 (logarithm with custom base)
SELECT LOG10(1000); -- 3.0 (base-10 logarithm, explicit)
SELECT EXP(1); -- 2.718281828... (e^1)
```
#### Trigonometric Functions
All trigonometric functions operate in radians:
```sql
SELECT SIN(0); -- 0 (sine)
SELECT COS(0); -- 1 (cosine)
SELECT TAN(0); -- 0 (tangent)
SELECT COT(1); -- 0.6420926... (cotangent)
SELECT ASIN(1); -- 1.5707963... (arc sine)
SELECT ACOS(0); -- 1.5707963... (arc cosine)
SELECT ATAN(1); -- 0.7853981... (arc tangent)
SELECT ATAN2(1, 1); -- 0.7853981... (arc tangent of y/x)
```
**Degree-based trigonometric functions** accept and return values in degrees:
```sql
SELECT SIND(90); -- 1 (sine of 90 degrees)
SELECT COSD(0); -- 1 (cosine of 0 degrees)
SELECT TAND(45); -- 1 (tangent of 45 degrees)
SELECT COTD(45); -- 1 (cotangent of 45 degrees)
SELECT ASIND(1); -- 90 (arc sine, result in degrees)
SELECT ACOSD(0); -- 90 (arc cosine, result in degrees)
SELECT ATAND(1); -- 45 (arc tangent, result in degrees)
SELECT ATAN2D(1, 1); -- 45 (arc tangent of y/x, result in degrees)
```
**Angle conversion** between radians and degrees:
```sql
SELECT DEGREES(PI()); -- 180 (radians to degrees)
SELECT RADIANS(180); -- 3.14159... (degrees to radians)
```
#### Hyperbolic Functions
```sql
SELECT SINH(1); -- 1.1752011... (hyperbolic sine)
SELECT COSH(1); -- 1.5430806... (hyperbolic cosine)
SELECT TANH(1); -- 0.7615941... (hyperbolic tangent)
SELECT ASINH(1); -- 0.8813735... (inverse hyperbolic sine)
SELECT ACOSH(2); -- 1.3169578... (inverse hyperbolic cosine)
SELECT ATANH(0.5); -- 0.5493061... (inverse hyperbolic tangent)
```
#### WIDTH_BUCKET
Assigns a value to a bucket in an equi-width histogram:
```sql
SELECT WIDTH_BUCKET(5.5, 0, 10, 5); -- 3 (bucket for 5.5 in 5 buckets from 0 to 10)
SELECT WIDTH_BUCKET(15, 0, 10, 5); -- 6 (above range)
SELECT WIDTH_BUCKET(-1, 0, 10, 5); -- 0 (below range)
```
### 4.3 Date and Time Functions
#### Getting Current Date/Time
```sql
SELECT CURRENT_DATE; -- Today's date (e.g., 2024-12-25)
SELECT CURRENT_TIME; -- Current time with timezone
SELECT CURRENT_TIMESTAMP; -- Current date and time with timezone
SELECT NOW(); -- Same as CURRENT_TIMESTAMP
SELECT LOCALTIME; -- Current time without timezone
SELECT LOCALTIMESTAMP; -- Current timestamp without timezone
```
#### Extracting Components
**EXTRACT** (SQL standard):
```sql
SELECT EXTRACT(YEAR FROM TIMESTAMP '2024-12-25 14:30:00'); -- 2024
SELECT EXTRACT(MONTH FROM TIMESTAMP '2024-12-25 14:30:00'); -- 12
SELECT EXTRACT(DAY FROM TIMESTAMP '2024-12-25 14:30:00'); -- 25
SELECT EXTRACT(HOUR FROM TIMESTAMP '2024-12-25 14:30:00'); -- 14
SELECT EXTRACT(MINUTE FROM TIMESTAMP '2024-12-25 14:30:00'); -- 30
SELECT EXTRACT(SECOND FROM TIMESTAMP '2024-12-25 14:30:00'); -- 0
SELECT EXTRACT(DOW FROM TIMESTAMP '2024-12-25'); -- 3 (Wednesday, 0=Sunday)
SELECT EXTRACT(DOY FROM TIMESTAMP '2024-12-25'); -- 360 (day of year)
SELECT EXTRACT(WEEK FROM TIMESTAMP '2024-12-25'); -- 52 (week number)
SELECT EXTRACT(QUARTER FROM TIMESTAMP '2024-12-25'); -- 4
```
**DATE_PART** (PostgreSQL function form):
```sql
SELECT DATE_PART('year', TIMESTAMP '2024-12-25'); -- 2024
SELECT DATE_PART('month', TIMESTAMP '2024-12-25'); -- 12
-- Same fields as EXTRACT
```
#### Truncating Dates
DATE_TRUNC rounds down to the specified precision:
```sql
SELECT DATE_TRUNC('year', TIMESTAMP '2024-12-25 14:30:00');
-- Result: 2024-01-01 00:00:00
SELECT DATE_TRUNC('month', TIMESTAMP '2024-12-25 14:30:00');
-- Result: 2024-12-01 00:00:00
SELECT DATE_TRUNC('day', TIMESTAMP '2024-12-25 14:30:00');
-- Result: 2024-12-25 00:00:00
SELECT DATE_TRUNC('hour', TIMESTAMP '2024-12-25 14:30:45');
-- Result: 2024-12-25 14:00:00
```
**Common Use Case: Grouping by Time Period**:
```sql
-- Daily sales totals
SELECT DATE_TRUNC('day', created_at) AS day, SUM(amount)
FROM orders
GROUP BY DATE_TRUNC('day', created_at)
ORDER BY day;
-- Monthly revenue
SELECT DATE_TRUNC('month', created_at) AS month, SUM(amount)
FROM orders
GROUP BY DATE_TRUNC('month', created_at)
ORDER BY month;
```
#### Date Arithmetic
```sql
-- Add days to date
SELECT DATE '2024-12-25' + 7; -- 2025-01-01
SELECT DATE '2024-12-25' + INTEGER '7'; -- 2025-01-01
-- Add intervals
SELECT TIMESTAMP '2024-12-25 10:00:00' + INTERVAL '2 hours';
-- Result: 2024-12-25 12:00:00
SELECT NOW() + INTERVAL '1 day'; -- Tomorrow same time
SELECT NOW() - INTERVAL '1 week'; -- A week ago
SELECT NOW() + INTERVAL '1 month'; -- Same day next month
-- Difference between dates
SELECT DATE '2024-12-31' - DATE '2024-01-01'; -- 365 (days)
-- Age calculation
SELECT AGE(TIMESTAMP '2024-12-25', TIMESTAMP '1990-06-15');
-- Result: 34 years 6 mons 10 days
SELECT AGE(TIMESTAMP '1990-06-15'); -- Age from today
```
#### Formatting Dates
**TO_CHAR** - Convert date/time to formatted string:
```sql
SELECT TO_CHAR(NOW(), 'YYYY-MM-DD'); -- '2024-12-25'
SELECT TO_CHAR(NOW(), 'Month DD, YYYY'); -- 'December 25, 2024'
SELECT TO_CHAR(NOW(), 'FMMonth DD, YYYY'); -- 'December 25, 2024' (FM removes padding)
SELECT TO_CHAR(NOW(), 'Day'); -- 'Wednesday'
SELECT TO_CHAR(NOW(), 'HH24:MI:SS'); -- '14:30:00'
SELECT TO_CHAR(NOW(), 'HH12:MI AM'); -- '02:30 PM'
```
**Common Format Codes**:
| Code | Meaning | Example |
|------|---------|---------|
| YYYY | 4-digit year | 2024 |
| MM | Month number | 12 |
| Month | Full month name | December |
| Mon | Abbreviated month | Dec |
| DD | Day of month | 25 |
| Day | Full day name | Wednesday |
| Dy | Abbreviated day | Wed |
| HH24 | Hour (00-23) | 14 |
| HH12 | Hour (01-12) | 02 |
| MI | Minutes | 30 |
| SS | Seconds | 00 |
| AM/PM | Meridiem | PM |
**TO_DATE and TO_TIMESTAMP** - Parse strings:
```sql
SELECT TO_DATE('25-12-2024', 'DD-MM-YYYY');
-- Result: 2024-12-25
SELECT TO_TIMESTAMP('2024-12-25 14:30', 'YYYY-MM-DD HH24:MI');
-- Result: 2024-12-25 14:30:00
```
#### Timestamp Precision Functions
Cognica provides several functions that return the current date/time at different levels of precision, matching PostgreSQL semantics:
**CLOCK_TIMESTAMP** — Returns the actual wall-clock time. Unlike NOW(), this changes during statement execution:
```sql
SELECT CLOCK_TIMESTAMP(); -- Actual current time (changes per call)
```
**STATEMENT_TIMESTAMP** — Returns the time at the start of the current statement:
```sql
SELECT STATEMENT_TIMESTAMP(); -- Same value throughout one statement
```
**TRANSACTION_TIMESTAMP** — Returns the time at the start of the current transaction (identical to NOW()):
```sql
SELECT TRANSACTION_TIMESTAMP(); -- Same value throughout the transaction
```
**TIMEOFDAY** — Returns the current date and time as a formatted text string:
```sql
SELECT TIMEOFDAY();
-- Result: 'Thu Dec 25 14:30:00.123456 2024 UTC'
```
#### Date/Time Construction Functions
Build date, time, and timestamp values from individual components:
**MAKE_DATE** — Construct a DATE from year, month, day:
```sql
SELECT MAKE_DATE(2024, 12, 25); -- 2024-12-25
```
**MAKE_TIME** — Construct a TIME from hour, minute, second:
```sql
SELECT MAKE_TIME(14, 30, 0); -- 14:30:00
```
**MAKE_TIMESTAMP** — Construct a TIMESTAMP from components:
```sql
SELECT MAKE_TIMESTAMP(2024, 12, 25, 14, 30, 0);
-- Result: 2024-12-25 14:30:00
```
**MAKE_TIMESTAMPTZ** — Construct a TIMESTAMPTZ (with optional timezone):
```sql
SELECT MAKE_TIMESTAMPTZ(2024, 12, 25, 14, 30, 0);
-- Result: 2024-12-25 14:30:00+00 (uses session timezone)
SELECT MAKE_TIMESTAMPTZ(2024, 12, 25, 14, 30, 0, 'America/New_York');
-- Result: 2024-12-25 14:30:00-05
```
**MAKE_INTERVAL** — Construct an INTERVAL from components (all optional, default 0):
```sql
SELECT MAKE_INTERVAL(years => 1, months => 6);
-- Result: 1 year 6 mons
SELECT MAKE_INTERVAL(days => 10, hours => 3);
-- Result: 10 days 03:00:00
```
#### Interval Adjustment Functions
**JUSTIFY_DAYS** — Convert 30-day periods to months:
```sql
SELECT JUSTIFY_DAYS(INTERVAL '60 days'); -- '2 mons'
```
**JUSTIFY_HOURS** — Convert 24-hour periods to days:
```sql
SELECT JUSTIFY_HOURS(INTERVAL '48 hours'); -- '2 days'
```
**JUSTIFY_INTERVAL** — Adjust both days and hours:
```sql
SELECT JUSTIFY_INTERVAL(INTERVAL '1 month -1 hour');
-- Result: '29 days 23:00:00'
```
#### Date/Time Validation and Binning
**ISFINITE** — Test whether a date, timestamp, or interval is finite:
```sql
SELECT ISFINITE(DATE '2024-12-25'); -- true
SELECT ISFINITE(TIMESTAMP 'infinity'); -- false
```
**DATE_BIN** — Bin timestamps into regular intervals aligned to a specified origin:
```sql
SELECT DATE_BIN(
INTERVAL '15 minutes',
TIMESTAMP '2024-12-25 14:37:00',
TIMESTAMP '2024-12-25 00:00:00'
);
-- Result: 2024-12-25 14:30:00
```
DATE_BIN is useful for aggregating time-series data into fixed-width buckets.
#### AT TIME ZONE
The AT TIME ZONE construct converts timestamps between time zones:
```sql
SELECT TIMESTAMP '2024-12-25 14:30:00' AT TIME ZONE 'America/New_York';
-- Converts a TIMESTAMP to TIMESTAMPTZ, treating the input as New York time
SELECT NOW() AT TIME ZONE 'Asia/Tokyo';
-- Converts TIMESTAMPTZ to the specified timezone
```
#### OVERLAPS
Tests whether two time periods overlap:
```sql
SELECT (DATE '2024-01-01', DATE '2024-06-30')
OVERLAPS
(DATE '2024-03-01', DATE '2024-09-30');
-- Result: true (the periods share March through June)
SELECT (DATE '2024-01-01', DATE '2024-03-01')
OVERLAPS
(DATE '2024-06-01', DATE '2024-09-01');
-- Result: false (no overlap)
```
### 4.4 Conditional Functions
#### COALESCE - First Non-NULL Value
COALESCE returns the first argument that is not NULL:
```sql
SELECT COALESCE(NULL, 'default'); -- 'default'
SELECT COALESCE(NULL, NULL, 'fallback'); -- 'fallback'
SELECT COALESCE(email, phone, 'No contact') FROM users;
```
**Common Use Cases**:
```sql
-- Provide default values
SELECT COALESCE(nickname, first_name) AS display_name FROM users;
-- Handle missing data in calculations
SELECT SUM(COALESCE(quantity, 0) * price) FROM order_items;
-- Build complete addresses
SELECT CONCAT_WS(', ',
address_line1,
COALESCE(address_line2, ''),
city,
state
) AS full_address FROM addresses;
```
#### NULLIF - Return NULL on Equality
NULLIF returns NULL if both arguments are equal:
```sql
SELECT NULLIF(5, 5); -- NULL
SELECT NULLIF(5, 10); -- 5
SELECT NULLIF('', ''); -- NULL
```
**Primary Use Case - Preventing Division by Zero**:
```sql
-- Without NULLIF: error when count is 0
SELECT total / count FROM stats; -- ERROR: division by zero
-- With NULLIF: returns NULL instead of error
SELECT total / NULLIF(count, 0) FROM stats;
```
#### CASE - Conditional Logic
**Simple CASE** (comparing to specific values):
```sql
SELECT
order_id,
CASE status
WHEN 'P' THEN 'Pending'
WHEN 'S' THEN 'Shipped'
WHEN 'D' THEN 'Delivered'
WHEN 'C' THEN 'Cancelled'
ELSE 'Unknown'
END AS status_text
FROM orders;
```
**Searched CASE** (evaluating conditions):
```sql
SELECT
product_name,
price,
CASE
WHEN price < 10 THEN 'Budget'
WHEN price < 50 THEN 'Standard'
WHEN price < 100 THEN 'Premium'
ELSE 'Luxury'
END AS price_tier
FROM products;
```
**CASE in ORDER BY**:
```sql
-- Custom sort order
SELECT * FROM tasks
ORDER BY
CASE priority
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
WHEN 'low' THEN 4
ELSE 5
END;
```
**CASE in Aggregations**:
```sql
SELECT
COUNT(*) AS total_orders,
COUNT(CASE WHEN status = 'completed' THEN 1 END) AS completed,
COUNT(CASE WHEN status = 'pending' THEN 1 END) AS pending,
COUNT(CASE WHEN status = 'cancelled' THEN 1 END) AS cancelled
FROM orders;
```
### 4.5 JSON Functions and Operators
Cognica provides comprehensive JSON support compatible with PostgreSQL, enabling you to store, query, and manipulate semi-structured data alongside traditional relational data. This section covers all JSON operators and functions available in Cognica.
#### JSON Operators Overview
JSON operators provide concise syntax for common operations. Here's a complete reference:
| Operator | Description | Example | Result |
|----------|-------------|---------|--------|
| `->` | Extract JSON element by key or index | `'{"a":1}'::jsonb -> 'a'` | `1` |
| `->>` | Extract JSON element as text | `'{"a":"hello"}'::jsonb ->> 'a'` | `hello` |
| `#>` | Extract JSON at path | `'{"a":{"b":1}}'::jsonb #> '{a,b}'` | `1` |
| `#>>` | Extract JSON at path as text | `'{"a":{"b":"x"}}'::jsonb #>> '{a,b}'` | `x` |
| `@>` | Contains (left contains right) | `'{"a":1,"b":2}'::jsonb @> '{"a":1}'` | `true` |
| `<@` | Contained by (left contained in right) | `'{"a":1}'::jsonb <@ '{"a":1,"b":2}'` | `true` |
| `?` | Key exists | `'{"a":1}'::jsonb ? 'a'` | `true` |
| `?|` | Any key exists | `'{"a":1}'::jsonb ?| array['a','b']` | `true` |
| `?&` | All keys exist | `'{"a":1,"b":2}'::jsonb ?& array['a','b']` | `true` |
| `||` | Concatenate JSON values | `'{"a":1}'::jsonb || '{"b":2}'` | `{"a":1,"b":2}` |
| `-` | Delete key or array element | `'{"a":1,"b":2}'::jsonb - 'a'` | `{"b":2}` |
| `#-` | Delete at path | `'{"a":{"b":1}}'::jsonb #- '{a,b}'` | `{"a":{}}` |
| `@?` | JSONPath exists | `'{"a":1}'::jsonb @? '$.a'` | `true` |
| `@@` | JSONPath match | `'{"a":true}'::jsonb @@ '$.a'` | `true` |
#### Extraction Operators
**The `->` Operator: Extract as JSON**
The `->` operator extracts a JSON element and returns it as JSON:
```sql
-- Extract by key from object
SELECT '{"name": "Alice", "age": 30}'::jsonb -> 'name';
-- Result: "Alice" (as JSON string)
-- Extract by index from array
SELECT '["a", "b", "c"]'::jsonb -> 1;
-- Result: "b"
-- Chain for nested access
SELECT '{"user": {"profile": {"name": "Bob"}}}'::jsonb -> 'user' -> 'profile' -> 'name';
-- Result: "Bob"
```
**The `->>` Operator: Extract as Text**
The `->>` operator extracts a JSON element and returns it as plain text:
```sql
-- Extract as text (no quotes)
SELECT '{"name": "Alice"}'::jsonb ->> 'name';
-- Result: Alice (plain text, no quotes)
-- Useful for comparisons
SELECT * FROM users
WHERE profile ->> 'status' = 'active';
-- Extract numbers as text
SELECT '{"count": 42}'::jsonb ->> 'count';
-- Result: 42 (as text string)
```
**Path Extraction with `#>` and `#>>`**
For deep nested access, use path operators:
```sql
-- Extract nested value as JSON
SELECT '{"a": {"b": {"c": "deep"}}}'::jsonb #> '{a,b,c}';
-- Result: "deep"
-- Extract nested value as text
SELECT '{"a": {"b": {"c": "deep"}}}'::jsonb #>> '{a,b,c}';
-- Result: deep
-- Access array elements in path
SELECT '{"users": [{"name": "Alice"}, {"name": "Bob"}]}'::jsonb #> '{users,0,name}';
-- Result: "Alice"
-- Practical example: Extract nested config
SELECT
config #>> '{database,host}' AS db_host,
config #>> '{database,port}' AS db_port
FROM application_settings;
```
#### Containment Operators
**The `@>` Operator: Contains**
Tests if the left JSON value contains the right JSON value:
```sql
-- Object containment
SELECT '{"name": "Alice", "age": 30, "city": "NYC"}'::jsonb
@> '{"name": "Alice"}';
-- Result: true
-- Array containment (all elements must exist)
SELECT '[1, 2, 3, 4, 5]'::jsonb @> '[2, 4]';
-- Result: true
-- Nested containment
SELECT '{"user": {"role": "admin", "active": true}}'::jsonb
@> '{"user": {"role": "admin"}}';
-- Result: true
-- Finding documents by nested properties (very efficient with GIN index)
SELECT * FROM events
WHERE data @> '{"type": "purchase", "status": "completed"}';
```
**The `<@` Operator: Contained By**
Tests if the left JSON value is contained within the right JSON value:
```sql
SELECT '{"a": 1}'::jsonb <@ '{"a": 1, "b": 2}';
-- Result: true
-- Useful for checking if a value is a subset
SELECT '["admin"]'::jsonb <@ '["admin", "user", "guest"]';
-- Result: true
```
#### Key Existence Operators
**The `?` Operator: Single Key Exists**
```sql
-- Check if key exists
SELECT '{"name": "Alice", "email": "alice@example.com"}'::jsonb ? 'email';
-- Result: true
SELECT '{"name": "Alice"}'::jsonb ? 'email';
-- Result: false
-- Find users with optional fields
SELECT * FROM users WHERE profile ? 'phone_number';
```
**The `?|` Operator: Any Key Exists**
```sql
-- Check if any of the keys exist
SELECT '{"a": 1, "b": 2}'::jsonb ?| array['b', 'c', 'd'];
-- Result: true (because 'b' exists)
-- Find products with any discount field
SELECT * FROM products
WHERE attributes ?| array['discount', 'sale_price', 'promo_code'];
```
**The `?&` Operator: All Keys Exist**
```sql
-- Check if all keys exist
SELECT '{"a": 1, "b": 2, "c": 3}'::jsonb ?& array['a', 'b'];
-- Result: true
SELECT '{"a": 1, "b": 2}'::jsonb ?& array['a', 'b', 'c'];
-- Result: false (missing 'c')
-- Validate required fields
SELECT * FROM orders
WHERE shipping_info ?& array['street', 'city', 'zip'];
```
#### Modification Operators
**The `||` Operator: Concatenation**
```sql
-- Merge objects (right side wins on conflicts)
SELECT '{"a": 1}'::jsonb || '{"b": 2}'::jsonb;
-- Result: {"a": 1, "b": 2}
SELECT '{"a": 1}'::jsonb || '{"a": 2}'::jsonb;
-- Result: {"a": 2}
-- Concatenate arrays
SELECT '[1, 2]'::jsonb || '[3, 4]'::jsonb;
-- Result: [1, 2, 3, 4]
-- Add element to array
SELECT '[1, 2]'::jsonb || '3'::jsonb;
-- Result: [1, 2, 3]
-- Update a document
UPDATE users
SET profile = profile || '{"verified": true}'::jsonb
WHERE id = 123;
```
**The `-` Operator: Deletion**
```sql
-- Delete key from object
SELECT '{"a": 1, "b": 2, "c": 3}'::jsonb - 'b';
-- Result: {"a": 1, "c": 3}
-- Delete by index from array
SELECT '[1, 2, 3, 4]'::jsonb - 1;
-- Result: [1, 3, 4]
-- Delete with negative index (from end)
SELECT '[1, 2, 3, 4]'::jsonb - (-1);
-- Result: [1, 2, 3]
-- Remove a field
UPDATE products
SET metadata = metadata - 'deprecated_field'
WHERE category = 'electronics';
```
**The `#-` Operator: Delete at Path**
```sql
-- Delete nested key
SELECT '{"a": {"b": 1, "c": 2}}'::jsonb #- '{a,b}';
-- Result: {"a": {"c": 2}}
-- Delete array element at path
SELECT '{"items": [1, 2, 3]}'::jsonb #- '{items,1}';
-- Result: {"items": [1, 3]}
```
#### JSON Extraction Functions
**json_extract_path / jsonb_extract_path**
Extract value at a specified path:
```sql
SELECT jsonb_extract_path('{"a": {"b": {"c": 1}}}'::jsonb, 'a', 'b', 'c');
-- Result: 1
-- Equivalent to #> operator
SELECT '{"a": {"b": {"c": 1}}}'::jsonb #> '{a,b,c}';
-- Result: 1
```
**json_extract_path_text / jsonb_extract_path_text**
Extract value at path as text:
```sql
SELECT jsonb_extract_path_text('{"user": {"name": "Alice"}}'::jsonb, 'user', 'name');
-- Result: Alice (text, not JSON)
-- Equivalent to #>> operator
SELECT '{"user": {"name": "Alice"}}'::jsonb #>> '{user,name}';
-- Result: Alice
```
#### JSON Construction Functions
**json_build_object / jsonb_build_object**
Build a JSON object from key-value pairs:
```sql
SELECT jsonb_build_object(
'name', 'Alice',
'age', 30,
'active', true
);
-- Result: {"name": "Alice", "age": 30, "active": true}
-- Dynamic object construction in queries
SELECT jsonb_build_object(
'id', u.id,
'full_name', u.first_name || ' ' || u.last_name,
'email', u.email
) AS user_json
FROM users u;
```
**json_build_array / jsonb_build_array**
Build a JSON array from values:
```sql
SELECT jsonb_build_array(1, 2, 'three', true, null);
-- Result: [1, 2, "three", true, null]
-- Combine with queries
SELECT jsonb_build_array(
jsonb_build_object('type', 'home', 'number', home_phone),
jsonb_build_object('type', 'work', 'number', work_phone)
) AS phone_numbers
FROM contacts;
```
**json_object / jsonb_object**
Alternative object construction:
```sql
-- From key-value arrays
SELECT jsonb_object('{name, age}', '{Alice, 30}');
-- Result: {"name": "Alice", "age": "30"}
-- From single array of alternating keys and values
SELECT jsonb_object('{name, Alice, age, 30}');
-- Result: {"name": "Alice", "age": "30"}
```
**to_json / to_jsonb**
Convert SQL values to JSON:
```sql
SELECT to_jsonb(ROW('Alice', 30, true));
-- Result: {"f1": "Alice", "f2": 30, "f3": true}
-- Convert array
SELECT to_jsonb(ARRAY[1, 2, 3]);
-- Result: [1, 2, 3]
```
#### JSON Modification Functions
**jsonb_set**
Set a value at a path, optionally creating missing keys:
```sql
-- Update existing value
SELECT jsonb_set('{"a": 1, "b": 2}'::jsonb, '{a}', '100');
-- Result: {"a": 100, "b": 2}
-- Set nested value
SELECT jsonb_set('{"user": {"name": "Alice"}}'::jsonb, '{user,email}', '"alice@example.com"');
-- Result: {"user": {"name": "Alice", "email": "alice@example.com"}}
-- Set array element
SELECT jsonb_set('{"items": [1, 2, 3]}'::jsonb, '{items,1}', '99');
-- Result: {"items": [1, 99, 3]}
-- create_if_missing parameter (default true)
SELECT jsonb_set('{"a": 1}'::jsonb, '{b}', '2', true);
-- Result: {"a": 1, "b": 2}
SELECT jsonb_set('{"a": 1}'::jsonb, '{b}', '2', false);
-- Result: {"a": 1} (b not created because create_if_missing is false)
```
**jsonb_insert**
Insert a value at a path:
```sql
-- Insert before array element
SELECT jsonb_insert('{"items": [1, 2, 3]}'::jsonb, '{items,1}', '99');
-- Result: {"items": [1, 99, 2, 3]}
-- Insert after array element
SELECT jsonb_insert('{"items": [1, 2, 3]}'::jsonb, '{items,1}', '99', true);
-- Result: {"items": [1, 2, 99, 3]}
-- Insert new key into object
SELECT jsonb_insert('{"a": 1}'::jsonb, '{b}', '2');
-- Result: {"a": 1, "b": 2}
```
**jsonb_strip_nulls**
Remove all object fields with null values (recursively):
```sql
SELECT jsonb_strip_nulls('{"a": 1, "b": null, "c": {"d": null, "e": 2}}'::jsonb);
-- Result: {"a": 1, "c": {"e": 2}}
-- Note: null array elements are NOT removed
SELECT jsonb_strip_nulls('[1, null, 2]'::jsonb);
-- Result: [1, null, 2]
```
**jsonb_pretty**
Format JSON with indentation for readability:
```sql
SELECT jsonb_pretty('{"name":"Alice","address":{"city":"NYC","zip":"10001"}}'::jsonb);
-- Result:
-- {
-- "name": "Alice",
-- "address": {
-- "city": "NYC",
-- "zip": "10001"
-- }
-- }
```
#### JSON Inspection Functions
**json_typeof / jsonb_typeof**
Return the type of a JSON value:
```sql
SELECT jsonb_typeof('{"a": 1}'::jsonb); -- "object"
SELECT jsonb_typeof('[1, 2, 3]'::jsonb); -- "array"
SELECT jsonb_typeof('"hello"'::jsonb); -- "string"
SELECT jsonb_typeof('42'::jsonb); -- "number"
SELECT jsonb_typeof('true'::jsonb); -- "boolean"
SELECT jsonb_typeof('null'::jsonb); -- "null"
```
**json_array_length / jsonb_array_length**
Return the number of elements in a JSON array:
```sql
SELECT jsonb_array_length('[1, 2, 3, 4, 5]'::jsonb);
-- Result: 5
SELECT jsonb_array_length('[]'::jsonb);
-- Result: 0
-- Returns NULL for non-arrays
SELECT jsonb_array_length('{"a": 1}'::jsonb);
-- Result: NULL
```
**json_object_keys / jsonb_object_keys**
Return the set of keys in a JSON object:
```sql
SELECT jsonb_object_keys('{"a": 1, "b": 2, "c": 3}'::jsonb);
-- Returns: a, b, c (as separate rows)
-- Use with array_agg to get as array
SELECT array_agg(key) FROM jsonb_object_keys('{"a": 1, "b": 2}'::jsonb) AS key;
-- Result: {a, b}
```
#### JSONPath Support
Cognica supports the SQL/JSON path language (JSONPath) for powerful JSON querying. JSONPath provides a standardized way to navigate and filter JSON data.
**Basic JSONPath Syntax**
| Element | Description | Example |
|---------|-------------|---------|
| `$` | Root element | `$` |
| `.key` | Object member access | `$.name` |
| `[n]` | Array element access | `$[0]` |
| `[*]` | All array elements | `$[*]` |
| `.*` | All object members | `$.*` |
| `..` | Recursive descent | `$..name` |
| `[start:end]` | Array slice | `$[1:3]` |
| `?(expr)` | Filter expression | `$[?(@.age > 18)]` |
**JSONPath Operators**
**The `@?` Operator: Path Exists**
Check if a JSONPath returns any values:
```sql
SELECT '{"user": {"name": "Alice"}}'::jsonb @? '$.user.name';
-- Result: true
SELECT '{"user": {"name": "Alice"}}'::jsonb @? '$.user.email';
-- Result: false
-- With filter
SELECT '[{"age": 15}, {"age": 25}]'::jsonb @? '$[?(@.age >= 18)]';
-- Result: true
```
**The `@@` Operator: Path Match**
Evaluate a JSONPath predicate and return the boolean result:
```sql
SELECT '{"active": true}'::jsonb @@ '$.active';
-- Result: true
SELECT '{"active": false}'::jsonb @@ '$.active';
-- Result: false
```
**jsonb_path_exists**
Function form of the `@?` operator:
```sql
SELECT jsonb_path_exists('{"a": {"b": 1}}'::jsonb, '$.a.b');
-- Result: true
-- With filter expression
SELECT jsonb_path_exists(
'[{"name": "Alice", "age": 25}, {"name": "Bob", "age": 35}]'::jsonb,
'$[?(@.age > 30)]'
);
-- Result: true
```
**jsonb_path_match**
Function form of the `@@` operator:
```sql
SELECT jsonb_path_match('{"verified": true}'::jsonb, '$.verified');
-- Result: true
```
**jsonb_path_query_array**
Returns all JSONPath matches as a JSON array:
```sql
SELECT jsonb_path_query_array(
'[{"name": "Alice", "age": 25}, {"name": "Bob", "age": 35}]'::jsonb,
'$[?(@.age > 30)]'
);
-- Result: [{"name": "Bob", "age": 35}]
```
**jsonb_path_query_first**
Returns the first JSONPath match (or NULL if no match):
```sql
SELECT jsonb_path_query_first(
'[{"name": "Alice"}, {"name": "Bob"}]'::jsonb,
'$[*].name'
);
-- Result: "Alice"
```
**JSONPath Filter Expressions**
Filter expressions use `@` to reference the current element:
```sql
-- Simple comparison
SELECT '{"items": [{"price": 10}, {"price": 50}, {"price": 100}]}'::jsonb
@? '$.items[?(@.price > 30)]';
-- Result: true
-- String comparison
SELECT '[{"name": "Alice"}, {"name": "Bob"}]'::jsonb
@? '$[?(@.name == "Alice")]';
-- Result: true
-- Logical operators
SELECT '[{"age": 25, "active": true}, {"age": 30, "active": false}]'::jsonb
@? '$[?(@.age > 20 && @.active == true)]';
-- Result: true
-- Existence check
SELECT '[{"email": "a@b.com"}, {"name": "Bob"}]'::jsonb
@? '$[?(@.email)]';
-- Result: true
-- Negation
SELECT '[{"deleted": false}, {"deleted": true}]'::jsonb
@? '$[?(!@.deleted)]';
-- Result: true
```
**JSONPath Methods**
```sql
-- .type() - Get the type of a value
SELECT jsonb_path_exists('{"arr": [1,2,3]}'::jsonb, '$.arr.type()');
-- Can be used in filters
-- .size() - Get array length
SELECT jsonb_path_exists('{"items": [1,2,3,4,5]}'::jsonb, '$[?(@.items.size() > 3)]');
-- Result: true
-- .double() - Convert string to number
SELECT jsonb_path_exists('{"price": "19.99"}'::jsonb, '$[?(@.price.double() > 15)]');
-- Result: true
```
**JSONPath Mode: Lax vs Strict**
JSONPath supports two modes:
```sql
-- Lax mode (default): Tolerant of structural errors
SELECT jsonb_path_exists('{"a": 1}'::jsonb, 'lax $.b');
-- Result: false (no error, just returns false)
-- Strict mode: Errors on missing paths
SELECT jsonb_path_exists('{"a": 1}'::jsonb, 'strict $.b');
-- Result: error or false depending on implementation
```
#### JSON Aggregate Functions
**json_agg / jsonb_agg**
Aggregate values into a JSON array:
```sql
SELECT jsonb_agg(name) FROM users WHERE department = 'Engineering';
-- Result: ["Alice", "Bob", "Charlie"]
-- Aggregate complex values
SELECT jsonb_agg(jsonb_build_object('id', id, 'name', name))
FROM users
WHERE active = true;
-- Result: [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
-- With ORDER BY
SELECT jsonb_agg(name ORDER BY created_at DESC)
FROM users;
```
**json_agg_strict / jsonb_agg_strict**
Like json_agg but excludes NULL values:
```sql
SELECT jsonb_agg_strict(optional_field) FROM records;
-- NULLs are excluded from the result array
```
**json_object_agg / jsonb_object_agg**
Aggregate key-value pairs into a JSON object:
```sql
SELECT jsonb_object_agg(setting_name, setting_value)
FROM user_settings
WHERE user_id = 123;
-- Result: {"theme": "dark", "language": "en", "notifications": true}
-- Create lookup table
SELECT jsonb_object_agg(code, name) FROM countries;
-- Result: {"US": "United States", "UK": "United Kingdom", ...}
```
**json_object_agg_strict / jsonb_object_agg_strict**
Exclude rows where the value is NULL:
```sql
SELECT jsonb_object_agg_strict(key, value) FROM settings;
-- Rows with NULL values are excluded
```
**json_object_agg_unique / jsonb_object_agg_unique**
Error on duplicate keys:
```sql
SELECT jsonb_object_agg_unique(key, value) FROM data;
-- Throws error if duplicate keys are encountered
```
#### GIN Indexing for JSON
For efficient JSON querying, create GIN indexes:
```sql
-- Index for containment and existence operators
CREATE INDEX idx_data_gin ON events USING GIN (data);
-- Now these queries use the index:
SELECT * FROM events WHERE data @> '{"type": "purchase"}';
SELECT * FROM events WHERE data ? 'user_id';
SELECT * FROM events WHERE data ?& array['type', 'timestamp'];
-- Index specific JSON path (expression index)
CREATE INDEX idx_user_email ON users USING BTREE ((profile ->> 'email'));
-- Now direct field access is fast:
SELECT * FROM users WHERE profile ->> 'email' = 'alice@example.com';
```
#### Practical Examples
**Event Logging System**
```sql
CREATE TABLE events (
id SERIAL PRIMARY KEY,
data JSONB NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_events_gin ON events USING GIN (data);
-- Insert events
INSERT INTO events (data) VALUES
('{"type": "page_view", "page": "/home", "user_id": 123}'),
('{"type": "purchase", "product_id": 456, "amount": 99.99, "user_id": 123}'),
('{"type": "login", "user_id": 123, "method": "oauth"}');
-- Query by event type
SELECT * FROM events WHERE data @> '{"type": "purchase"}';
-- Query with multiple conditions
SELECT * FROM events
WHERE data @> '{"type": "purchase"}'
AND (data ->> 'amount')::numeric > 50;
-- Extract fields
SELECT
data ->> 'type' AS event_type,
data ->> 'user_id' AS user_id,
created_at
FROM events
WHERE data ? 'user_id';
```
**User Preferences**
```sql
CREATE TABLE users (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
preferences JSONB DEFAULT '{}'::jsonb
);
-- Set preferences
UPDATE users
SET preferences = preferences || jsonb_build_object('theme', 'dark', 'language', 'en')
WHERE id = 1;
-- Update single preference
UPDATE users
SET preferences = jsonb_set(preferences, '{notifications,email}', 'true')
WHERE id = 1;
-- Remove preference
UPDATE users
SET preferences = preferences - 'deprecated_setting'
WHERE id = 1;
-- Query users with specific preferences
SELECT * FROM users
WHERE preferences @> '{"theme": "dark"}';
```
**Product Attributes**
```sql
CREATE TABLE products (
id SERIAL PRIMARY KEY,
name VARCHAR(200),
attributes JSONB
);
CREATE INDEX idx_product_attrs ON products USING GIN (attributes);
-- Insert products with varied attributes
INSERT INTO products (name, attributes) VALUES
('Laptop', '{"brand": "Dell", "ram": 16, "storage": "512GB SSD", "color": "silver"}'),
('Phone', '{"brand": "Apple", "storage": "128GB", "color": "black", "5g": true}'),
('Tablet', '{"brand": "Samsung", "storage": "64GB", "color": "white"}');
-- Find products by attribute
SELECT * FROM products WHERE attributes @> '{"brand": "Apple"}';
-- Find products with specific features
SELECT * FROM products WHERE attributes ? '5g';
-- Filter by numeric attribute
SELECT * FROM products
WHERE (attributes ->> 'ram')::int >= 16;
-- Search products with any of multiple colors
SELECT * FROM products
WHERE attributes ->> 'color' IN ('black', 'silver');
```
### 4.6 Array Functions
Beyond the array operators (`@>`, `<@`, `&&`) covered in Chapter 3, Cognica provides functions for manipulating arrays:
#### Array Information
```sql
-- Get array length
SELECT array_length(ARRAY[1, 2, 3, 4, 5], 1); -- 5 (dimension 1)
SELECT cardinality(ARRAY[1, 2, 3]); -- 3
-- Find element position
SELECT array_position(ARRAY['a', 'b', 'c'], 'b'); -- 2
SELECT array_positions(ARRAY[1, 2, 1, 3, 1], 1); -- {1,3,5}
-- Array bounds
SELECT array_upper(ARRAY[10, 20, 30], 1); -- 3
SELECT array_lower(ARRAY[10, 20, 30], 1); -- 1
SELECT array_ndims(ARRAY[[1, 2], [3, 4]]); -- 2
SELECT array_dims(ARRAY[[1, 2], [3, 4]]); -- '[1:2][1:2]'
```
#### Array Manipulation
```sql
-- Append/prepend elements
SELECT array_append(ARRAY[1, 2, 3], 4); -- {1,2,3,4}
SELECT array_prepend(0, ARRAY[1, 2, 3]); -- {0,1,2,3}
-- Concatenate arrays
SELECT array_cat(ARRAY[1, 2], ARRAY[3, 4]); -- {1,2,3,4}
SELECT ARRAY[1, 2] || ARRAY[3, 4]; -- {1,2,3,4} (operator form)
-- Remove elements
SELECT array_remove(ARRAY[1, 2, 3, 2, 1], 2); -- {1,3,1}
-- Convert array to rows
SELECT unnest(ARRAY['a', 'b', 'c']);
-- Returns 3 rows: 'a', 'b', 'c'
-- Convert text to array
SELECT string_to_array('one,two,three', ','); -- {one,two,three}
-- Convert array to text
SELECT array_to_string(ARRAY[1, 2, 3], ', '); -- '1, 2, 3'
```
### 4.7 Range Functions
Range functions construct and inspect range values (see [Section 2.9](#29-range-types-representing-intervals-of-values) for range type details):
#### Range Constructors
```sql
-- Integer ranges
SELECT int4range(1, 10); -- [1,10)
SELECT int4range(1, 10, '[]'); -- [1,11) (canonicalized to include 10)
SELECT int8range(1, 1000000000); -- [1,1000000000)
-- Numeric range (continuous, no canonicalization)
SELECT numrange(1.5, 9.9); -- [1.5,9.9)
SELECT numrange(1.5, 9.9, '[]'); -- [1.5,9.9]
-- Date and timestamp ranges
SELECT daterange('2024-01-01', '2024-12-31', '[]'); -- [2024-01-01,2025-01-01)
SELECT tsrange('2024-01-01 09:00', '2024-01-01 17:00');
SELECT tstzrange('2024-01-01 00:00+00', '2024-12-31 23:59+00');
```
#### Range Accessors
```sql
-- Check if empty
SELECT isempty(int4range(1, 1)); -- true (empty: no values in [1,1))
SELECT isempty(int4range(1, 2)); -- false
-- Check bound inclusivity
SELECT lower_inc(int4range(1, 10)); -- true (lower bound is inclusive)
SELECT upper_inc(int4range(1, 10)); -- false (upper bound is exclusive)
-- Check for infinite bounds
SELECT lower_inf(int4range(NULL, 10)); -- true (unbounded below)
SELECT upper_inf(int4range(1, NULL)); -- true (unbounded above)
-- Merge overlapping or adjacent ranges
SELECT range_merge(int4range(1, 5), int4range(3, 10)); -- [1,10)
```
### 4.8 Sequence Functions
Sequence functions manage auto-incrementing counters:
```sql
-- Get next value (advances the sequence)
SELECT nextval('orders_id_seq'); -- 1, 2, 3, ... (each call increments)
-- Get current value (does not advance; error if nextval not yet called in session)
SELECT currval('orders_id_seq');
-- Get last value returned by nextval in this session (any sequence)
SELECT lastval();
-- Set sequence value
SELECT setval('orders_id_seq', 1000); -- Next nextval returns 1001
SELECT setval('orders_id_seq', 1000, false); -- Next nextval returns 1000
```
Sequences are automatically created for `SERIAL` and `BIGSERIAL` columns with the naming convention `{table}_{column}_seq`.
---
## Chapter 5: Aggregate Functions - Summarizing Data
Aggregate functions represent one of the most powerful concepts in SQL, transforming the way you think about data from individual rows to meaningful summaries. While most SQL operations work row-by-row (selecting columns, filtering with WHERE, joining tables), aggregate functions break this pattern by consuming multiple rows and producing a single result value.
This distinction is fundamental to understanding SQL. A regular function like `UPPER(name)` operates on one value at a time - give it 'alice' and it returns 'ALICE'. An aggregate function like `COUNT(*)` operates on an entire set of rows - give it a table of 1,000 users and it returns a single number: 1000. This "many-to-one" transformation is what makes aggregates essential for reporting, analytics, and any situation where you need to answer questions about groups of data rather than individual records.
### The Mental Model: From Rows to Summaries
Think of aggregate functions as answering questions that begin with "What is the total...", "How many...", "What is the average...", or "What is the highest...". These questions cannot be answered by looking at individual rows - they require examining a collection of rows and distilling that collection into a single answer.
Consider a sales table with 10,000 orders. Asking "What is order #5432?" is a row-level question - you filter to that specific row and read its columns. But asking "What is our total revenue?" is an aggregate question - you must examine every order, sum up all the amounts, and produce one number. This is the domain of aggregate functions.
### NULL Handling: The Silent Complication
One of the most important behavioral characteristics of aggregate functions is how they handle NULL values. With the sole exception of `COUNT(*)`, all aggregate functions ignore NULL values entirely. This is not merely a convenience - it fundamentally affects your calculations.
When you compute `AVG(salary)` over employees where some salaries are NULL, those NULL values are excluded from both the sum and the count used to compute the average. If you have three employees with salaries of 50000, NULL, and 70000, the average is (50000 + 70000) / 2 = 60000, not (50000 + 0 + 70000) / 3 = 40000. The NULL employee is invisible to the calculation.
This behavior is usually desirable - NULL represents "unknown", and including unknowns in a calculation would produce a misleading result. However, if NULL in your data actually means "zero" (a common data quality issue), you must explicitly convert NULLs using `COALESCE(column, 0)` before aggregating.
### The Relationship Between Aggregates and GROUP BY
Aggregate functions and GROUP BY are deeply intertwined. Without GROUP BY, an aggregate function treats the entire result set as a single group and produces one row of output. With GROUP BY, the result set is partitioned into groups based on the specified columns, and the aggregate function produces one result per group.
This relationship creates a strict rule that often confuses newcomers: in a query with GROUP BY, every column in the SELECT list must either appear in the GROUP BY clause or be wrapped in an aggregate function. There is no middle ground. This rule exists because each output row represents an entire group - if you ask for a non-grouped column, which of the potentially many different values in that group should be displayed?
### 5.1 The Fundamentals
#### How Aggregates Work
Aggregate functions process all rows in a group and return one result. Without GROUP BY, all rows form a single implicit group:
```sql
SELECT COUNT(*) FROM users; -- One result: count of all users
SELECT AVG(price) FROM products; -- One result: average of all prices
```
With GROUP BY, each group produces one result:
```sql
SELECT category, COUNT(*) FROM products GROUP BY category;
-- One row per category
```
#### The Basic Aggregates
**COUNT** - Counting rows:
```sql
SELECT COUNT(*) FROM users; -- Count ALL rows (including NULLs)
SELECT COUNT(email) FROM users; -- Count rows where email is NOT NULL
SELECT COUNT(DISTINCT country) FROM users; -- Count unique countries
```
**Important**: COUNT(*) counts all rows. COUNT(column) counts non-NULL values. These are different!
```sql
-- Users table: 100 rows, but 5 have NULL email
SELECT COUNT(*) FROM users; -- 100
SELECT COUNT(email) FROM users; -- 95
```
**SUM** - Adding values:
```sql
SELECT SUM(amount) FROM orders;
SELECT SUM(quantity * price) AS total_revenue FROM order_items;
```
**AVG** - Calculating averages:
```sql
SELECT AVG(price) FROM products;
SELECT AVG(rating) FROM reviews WHERE product_id = 123;
```
**Critical Point**: AVG ignores NULL values:
```sql
-- Values: 10, NULL, 30
SELECT AVG(value) FROM data; -- Result: 20 (not 13.33!)
-- AVG = (10 + 30) / 2 = 20
-- If you want NULLs treated as 0:
SELECT AVG(COALESCE(value, 0)) FROM data; -- Result: 13.33
-- AVG = (10 + 0 + 30) / 3 = 13.33
```
**MIN and MAX** - Finding extremes:
```sql
SELECT MIN(price), MAX(price) FROM products;
SELECT MIN(created_at) AS first_order FROM orders;
SELECT MAX(name) FROM users; -- Alphabetically last name
```
### 5.2 Advanced Aggregates
#### Statistical Functions
```sql
SELECT
AVG(salary) AS mean,
STDDEV(salary) AS std_deviation, -- Sample standard deviation
STDDEV_POP(salary) AS pop_std_dev, -- Population standard deviation
VARIANCE(salary) AS variance, -- Sample variance
VAR_POP(salary) AS pop_variance -- Population variance
FROM employees;
```
**When to use Sample vs Population**:
- **Population** (STDDEV_POP, VAR_POP): Your data IS the entire population
- **Sample** (STDDEV, VARIANCE): Your data is a sample from a larger population
For most practical purposes, the difference is small unless your sample size is very small.
#### Covariance Functions
```sql
SELECT
COVAR_POP(y, x) AS population_covariance,
COVAR_SAMP(y, x) AS sample_covariance
FROM data_points;
```
#### Regression and Correlation
```sql
SELECT
CORR(sales, ad_spend) AS correlation, -- How related are these?
REGR_SLOPE(sales, ad_spend) AS slope, -- Linear regression slope
REGR_INTERCEPT(sales, ad_spend) AS intercept, -- Y-intercept
REGR_R2(sales, ad_spend) AS r_squared, -- How well does line fit?
REGR_AVGX(sales, ad_spend) AS avg_x, -- Average of independent var
REGR_AVGY(sales, ad_spend) AS avg_y, -- Average of dependent var
REGR_COUNT(sales, ad_spend) AS num_pairs, -- Number of non-null pairs
REGR_SXX(sales, ad_spend) AS sum_squares_x, -- Sum of squares of X
REGR_SYY(sales, ad_spend) AS sum_squares_y, -- Sum of squares of Y
REGR_SXY(sales, ad_spend) AS sum_cross_products -- Sum of cross-products
FROM marketing_data;
-- Interpretation:
-- If slope = 2.5 and intercept = 1000:
-- sales = 1000 + 2.5 * ad_spend
-- (Every $1 in ad spend = $2.50 in sales, starting from $1000 base)
```
#### Boolean Aggregates
```sql
SELECT
BOOL_AND(is_active) AS all_active, -- TRUE if all values are TRUE
BOOL_OR(is_active) AS any_active, -- TRUE if any value is TRUE
EVERY(is_verified) AS all_verified -- Alias for BOOL_AND
FROM users;
```
#### Bitwise Aggregates
```sql
SELECT
BIT_AND(permissions) AS common_permissions, -- Bitwise AND of all values
BIT_OR(permissions) AS any_permissions, -- Bitwise OR of all values
BIT_XOR(flags) AS xor_result -- Bitwise XOR of all values
FROM user_permissions
GROUP BY role;
```
#### ANY_VALUE
Returns any non-null value from the group. Useful when you know all values in the group are the same but the column is not in the GROUP BY:
```sql
SELECT
department_id,
ANY_VALUE(department_name) AS name, -- All rows have same name per dept
COUNT(*) AS employee_count
FROM employees
GROUP BY department_id;
```
#### Ordered-Set Aggregates
These require an ORDER BY specification within the function:
**MODE** - Most frequent value:
```sql
SELECT MODE() WITHIN GROUP (ORDER BY category)
FROM products;
-- Returns the most common category
```
**PERCENTILE_CONT** - Continuous percentile (interpolates):
```sql
SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY salary) AS median
FROM employees;
-- The median (50th percentile)
SELECT
PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY salary) AS q1,
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY salary) AS median,
PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY salary) AS q3
FROM employees;
-- Quartiles
```
**PERCENTILE_DISC** - Discrete percentile (actual value):
```sql
SELECT PERCENTILE_DISC(0.5) WITHIN GROUP (ORDER BY salary) AS median
FROM employees;
-- Returns an actual salary value, not an interpolation
```
#### Hypothetical-Set Aggregates
These compute what the ranking would be for a hypothetical row, if it were inserted into the group:
```sql
-- What rank would a salary of 75000 be?
SELECT
RANK(75000) WITHIN GROUP (ORDER BY salary) AS hypothetical_rank,
DENSE_RANK(75000) WITHIN GROUP (ORDER BY salary) AS hypothetical_dense_rank,
PERCENT_RANK(75000) WITHIN GROUP (ORDER BY salary) AS hypothetical_pct_rank,
CUME_DIST(75000) WITHIN GROUP (ORDER BY salary) AS hypothetical_cume_dist
FROM employees;
```
### 5.3 Collecting Values
#### ARRAY_AGG - Collect into array
```sql
SELECT
customer_id,
ARRAY_AGG(product_name ORDER BY order_date) AS products_ordered
FROM orders
JOIN products ON orders.product_id = products.id
GROUP BY customer_id;
```
#### STRING_AGG - Concatenate strings
```sql
SELECT
department,
STRING_AGG(employee_name, ', ' ORDER BY employee_name) AS team_members
FROM employees
GROUP BY department;
-- Result: "Engineering", "Alice, Bob, Charlie"
```
#### JSON Aggregates
```sql
-- Collect rows as JSON array
SELECT JSON_AGG(row_to_json(t))
FROM (SELECT id, name, email FROM users LIMIT 5) t;
-- Result: [{"id":1,"name":"Alice","email":"..."}, {...}, ...]
-- Strict variant: excludes NULL values from the array
SELECT JSON_AGG_STRICT(optional_field) FROM data;
-- Build JSON object from key-value pairs
SELECT JSONB_OBJECT_AGG(setting_name, setting_value) AS settings
FROM user_settings
WHERE user_id = 1;
-- Result: {"theme": "dark", "language": "en", "notifications": "true"}
```
**JSON aggregate variants:**
| Function | Behavior |
|----------|----------|
| `JSON_AGG(expr)` / `JSONB_AGG(expr)` | Collect values into JSON array |
| `JSON_AGG_STRICT(expr)` / `JSONB_AGG_STRICT(expr)` | Exclude NULL values |
| `JSON_OBJECT_AGG(key, value)` / `JSONB_OBJECT_AGG(key, value)` | Build JSON object |
| `JSON_OBJECT_AGG_STRICT(key, value)` | Exclude entries where value is NULL |
| `JSON_OBJECT_AGG_UNIQUE(key, value)` | Error on duplicate keys |
| `JSON_OBJECT_AGG_UNIQUE_STRICT(key, value)` | Unique keys + exclude NULL values |
### 5.4 The FILTER Clause
FILTER lets you apply conditions to specific aggregates:
```sql
SELECT
COUNT(*) AS total_orders,
COUNT(*) FILTER (WHERE status = 'completed') AS completed,
COUNT(*) FILTER (WHERE status = 'pending') AS pending,
COUNT(*) FILTER (WHERE status = 'cancelled') AS cancelled,
SUM(amount) FILTER (WHERE status = 'completed') AS completed_revenue,
AVG(amount) FILTER (WHERE status = 'completed') AS avg_completed_order
FROM orders;
```
This is much cleaner than using CASE expressions:
```sql
-- Equivalent but verbose:
SELECT
COUNT(*) AS total_orders,
COUNT(CASE WHEN status = 'completed' THEN 1 END) AS completed,
SUM(CASE WHEN status = 'completed' THEN amount END) AS completed_revenue
FROM orders;
```
### 5.5 GROUP BY Fundamentals
GROUP BY divides rows into groups, and aggregate functions operate on each group:
```sql
-- Count products per category
SELECT category, COUNT(*) AS product_count
FROM products
GROUP BY category;
-- Average salary by department and job level
SELECT department, job_level, AVG(salary) AS avg_salary
FROM employees
GROUP BY department, job_level;
```
**The Golden Rule**: Every column in SELECT must either be:
1. In the GROUP BY clause, OR
2. Inside an aggregate function
```sql
-- WRONG: name is not in GROUP BY or an aggregate
SELECT department, name, AVG(salary)
FROM employees
GROUP BY department;
-- Error: column "name" must appear in GROUP BY clause
-- RIGHT: name is aggregated
SELECT department, COUNT(name), AVG(salary)
FROM employees
GROUP BY department;
```
### 5.6 HAVING: Filtering Groups
WHERE filters rows before grouping. HAVING filters groups after aggregation:
```sql
-- Find departments with average salary over $50,000
SELECT department, AVG(salary) AS avg_salary
FROM employees
GROUP BY department
HAVING AVG(salary) > 50000;
-- Find categories with more than 10 products
SELECT category, COUNT(*) AS product_count
FROM products
GROUP BY category
HAVING COUNT(*) > 10;
```
**WHERE vs HAVING**:
```sql
-- Filter rows BEFORE grouping (use WHERE)
SELECT department, AVG(salary)
FROM employees
WHERE status = 'active' -- Only active employees
GROUP BY department;
-- Filter groups AFTER aggregating (use HAVING)
SELECT department, AVG(salary)
FROM employees
GROUP BY department
HAVING AVG(salary) > 50000; -- Only high-paying departments
-- You can use both
SELECT department, AVG(salary) AS avg_salary
FROM employees
WHERE status = 'active' -- First: only active employees
GROUP BY department
HAVING AVG(salary) > 50000; -- Then: only high-paying departments
```
### 5.7 Advanced Grouping: ROLLUP, CUBE, and GROUPING SETS
These features let you compute multiple levels of aggregation in one query.
#### ROLLUP: Hierarchical Subtotals
```sql
SELECT
COALESCE(country, 'ALL COUNTRIES') AS country,
COALESCE(city, 'ALL CITIES') AS city,
SUM(sales) AS total_sales
FROM sales_data
GROUP BY ROLLUP (country, city);
```
Result:
```
country | city | total_sales
-------------|--------------|------------
USA | New York | 50000
USA | Los Angeles | 40000
USA | ALL CITIES | 90000 <- Subtotal for USA
UK | London | 30000
UK | Manchester | 20000
UK | ALL CITIES | 50000 <- Subtotal for UK
ALL COUNTRIES| ALL CITIES | 140000 <- Grand total
```
#### CUBE: All Combinations
```sql
SELECT region, product, SUM(sales)
FROM sales_data
GROUP BY CUBE (region, product);
```
This produces:
- Sales by (region, product)
- Sales by region (all products)
- Sales by product (all regions)
- Total sales (all regions, all products)
#### GROUPING SETS: Specific Combinations
```sql
SELECT region, product, SUM(sales)
FROM sales_data
GROUP BY GROUPING SETS (
(region, product), -- Detail level
(region), -- By region
(product), -- By product
() -- Grand total
);
```
#### The GROUPING Function
How do you tell if a NULL is a real NULL or a subtotal marker? Use GROUPING():
```sql
SELECT
country,
city,
SUM(sales),
GROUPING(country) AS is_country_subtotal,
GROUPING(city) AS is_city_subtotal
FROM sales_data
GROUP BY ROLLUP (country, city);
```
GROUPING() returns 1 when the column is NULL due to subtotaling, 0 otherwise.
---
## Chapter 6: Window Functions - Analytics Without Collapsing
Window functions represent a fundamental shift in how SQL processes data, bridging the gap between row-level operations and aggregation. Before window functions were added to SQL (in SQL:2003), there was an uncomfortable dichotomy: you could either work with individual rows (getting full detail but no context) or aggregate rows (getting summaries but losing detail). Window functions eliminate this trade-off by allowing you to compute values based on groups of related rows while preserving every individual row in the output.
The name "window function" comes from the concept of a sliding window over your data. For each row being processed, the function looks at a "window" of related rows - perhaps all rows in the same department, or the three preceding rows in time order, or all rows from the beginning of the partition up to the current row. The function computes a result based on this window, and that result becomes a column value for the current row.
```mermaid
flowchart TB
subgraph Input["Input Rows"]
R1["Row 1: Sales"]
R2["Row 2: Sales"]
R3["Row 3: IT"]
R4["Row 4: IT"]
R5["Row 5: HR"]
end
subgraph Partition["PARTITION BY department"]
subgraph P1["Partition: Sales"]
S1["Row 1"]
S2["Row 2"]
end
subgraph P2["Partition: IT"]
I1["Row 3"]
I2["Row 4"]
end
subgraph P3["Partition: HR"]
H1["Row 5"]
end
end
subgraph Output["Output (All Rows Preserved)"]
O1["Row 1 + Window Result"]
O2["Row 2 + Window Result"]
O3["Row 3 + Window Result"]
O4["Row 4 + Window Result"]
O5["Row 5 + Window Result"]
end
R1 --> S1
R2 --> S2
R3 --> I1
R4 --> I2
R5 --> H1
P1 --> O1
P1 --> O2
P2 --> O3
P2 --> O4
P3 --> O5
style P1 fill:#e3f2fd,color:#1565c0
style P2 fill:#fff3e0,color:#e65100
style P3 fill:#e8f5e9,color:#2e7d32
```
### Why Window Functions Matter
Consider the common business question: "Show each employee's salary and what percentage of their department's total salary it represents." Without window functions, answering this requires either a self-join or a correlated subquery - both awkward and often slow. With window functions, the solution is elegant: compute the department total using `SUM() OVER (PARTITION BY department)` and divide each salary by that total.
Window functions are particularly powerful for analytical queries:
- **Ranking**: Number rows within categories, find top-N per group
- **Running calculations**: Cumulative sums, moving averages, running totals
- **Comparisons**: Compare each row to group average, previous row, or first row
- **Gap analysis**: Find differences between consecutive values
### The Execution Model: When Window Functions Run
Window functions execute after WHERE, GROUP BY, and HAVING, but before the final ORDER BY. This placement in SQL's logical processing order has important implications. You cannot filter on window function results in the WHERE clause (because window functions have not run yet); you must use a subquery or CTE and filter in an outer query.
Understanding this execution order also explains why window functions see the rows that remain after filtering and grouping. If your query has a WHERE clause that eliminates 80% of rows, window functions only see the surviving 20%.
### 6.1 Understanding Window Functions
#### The Core Concept
Consider this employees table:
```
id | name | department | salary
---|---------|------------|-------
1 | Alice | Sales | 50000
2 | Bob | Sales | 60000
3 | Charlie | Sales | 55000
4 | Diana | IT | 70000
5 | Eve | IT | 75000
```
With GROUP BY, we can get department totals:
```sql
SELECT department, SUM(salary) FROM employees GROUP BY department;
```
Result:
```
department | sum
-----------|-------
Sales | 165000
IT | 145000
```
But what if we want each employee's salary AND their department total? This is where window functions shine:
```sql
SELECT
name,
department,
salary,
SUM(salary) OVER (PARTITION BY department) AS dept_total
FROM employees;
```
Result:
```
name | department | salary | dept_total
--------|------------|--------|-----------
Alice | Sales | 50000 | 165000
Bob | Sales | 60000 | 165000
Charlie | Sales | 55000 | 165000
Diana | IT | 70000 | 145000
Eve | IT | 75000 | 145000
```
Every row is preserved, but we added aggregated information!
#### Window Function Syntax
```sql
function_name(arguments) OVER (
[PARTITION BY partition_expression, ...]
[ORDER BY sort_expression [ASC|DESC] [NULLS {FIRST|LAST}], ...]
[frame_clause]
)
```
- **PARTITION BY**: Divides rows into groups (like GROUP BY, but doesn't collapse)
- **ORDER BY**: Orders rows within each partition
- **Frame clause**: Specifies which rows relative to current row to consider
### 6.2 Ranking Functions
#### ROW_NUMBER: Unique Sequential Number
```sql
SELECT
name,
department,
salary,
ROW_NUMBER() OVER (ORDER BY salary DESC) AS overall_rank,
ROW_NUMBER() OVER (PARTITION BY department ORDER BY salary DESC) AS dept_rank
FROM employees;
```
Result:
```
name | department | salary | overall_rank | dept_rank
--------|------------|--------|--------------|----------
Eve | IT | 75000 | 1 | 1
Diana | IT | 70000 | 2 | 2
Bob | Sales | 60000 | 3 | 1
Charlie | Sales | 55000 | 4 | 2
Alice | Sales | 50000 | 5 | 3
```
ROW_NUMBER always gives unique values, even for ties.
#### RANK: With Gaps for Ties
```sql
-- If two people tie for 2nd place:
-- RANK: 1, 2, 2, 4 (skip 3)
SELECT name, score, RANK() OVER (ORDER BY score DESC) AS rank
FROM contestants;
```
#### DENSE_RANK: No Gaps for Ties
```sql
-- If two people tie for 2nd place:
-- DENSE_RANK: 1, 2, 2, 3 (no gap)
SELECT name, score, DENSE_RANK() OVER (ORDER BY score DESC) AS rank
FROM contestants;
```
#### NTILE: Divide Into Groups
```sql
SELECT
name,
salary,
NTILE(4) OVER (ORDER BY salary) AS quartile
FROM employees;
-- Divides employees into 4 equal groups by salary
```
#### PERCENT_RANK and CUME_DIST: Relative Position
```sql
SELECT
name,
salary,
PERCENT_RANK() OVER (ORDER BY salary) AS pct_rank,
CUME_DIST() OVER (ORDER BY salary) AS cume_dist
FROM employees;
-- PERCENT_RANK: (rank - 1) / (total rows - 1), ranges 0 to 1
-- CUME_DIST: fraction of rows <= current row, ranges 0 to 1
```
### 6.3 Value Functions
#### LAG and LEAD: Access Other Rows
**LAG**: Look at previous rows
**LEAD**: Look at following rows
```sql
SELECT
date,
sales,
LAG(sales, 1) OVER (ORDER BY date) AS prev_day,
LEAD(sales, 1) OVER (ORDER BY date) AS next_day,
sales - LAG(sales, 1) OVER (ORDER BY date) AS daily_change
FROM daily_sales;
```
Result:
```
date | sales | prev_day | next_day | daily_change
-----------|-------|----------|----------|-------------
2024-01-01 | 100 | NULL | 120 | NULL
2024-01-02 | 120 | 100 | 90 | 20
2024-01-03 | 90 | 120 | 150 | -30
2024-01-04 | 150 | 90 | NULL | 60
```
With default value for edges:
```sql
LAG(sales, 1, 0) OVER (ORDER BY date) -- Returns 0 instead of NULL for first row
```
#### FIRST_VALUE, LAST_VALUE, NTH_VALUE
```sql
SELECT
name,
department,
salary,
FIRST_VALUE(name) OVER (
PARTITION BY department ORDER BY salary DESC
) AS highest_paid_in_dept,
LAST_VALUE(name) OVER (
PARTITION BY department ORDER BY salary DESC
ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
) AS lowest_paid_in_dept
FROM employees;
```
**Important Note**: LAST_VALUE requires a frame specification! By default, the frame is "UNBOUNDED PRECEDING to CURRENT ROW", so LAST_VALUE would just return the current row. You usually want "UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING".
### 6.4 Aggregate Functions as Window Functions
Any aggregate can be used as a window function:
```sql
SELECT
date,
sales,
SUM(sales) OVER (ORDER BY date) AS running_total,
AVG(sales) OVER (ORDER BY date) AS running_avg,
COUNT(*) OVER (ORDER BY date) AS cumulative_count
FROM daily_sales;
```
### 6.5 Frame Specifications
The frame determines which rows are included in the calculation.
**Frame Types**:
- **ROWS**: Physical rows relative to current
- **RANGE**: Logical range based on ORDER BY value
- **GROUPS**: Peer groups (rows with same ORDER BY value)
**Frame Bounds**:
- `UNBOUNDED PRECEDING`: Start of partition
- `n PRECEDING`: n rows/range before current
- `CURRENT ROW`: The current row
- `n FOLLOWING`: n rows/range after current
- `UNBOUNDED FOLLOWING`: End of partition
**Common Patterns**:
```sql
-- Running total (default with ORDER BY)
SUM(sales) OVER (ORDER BY date)
-- Frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
-- Total for entire partition
SUM(sales) OVER (PARTITION BY category)
-- Frame: entire partition (no ORDER BY)
-- 7-day moving average
AVG(sales) OVER (
ORDER BY date
ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
)
-- 3-row centered moving average
AVG(sales) OVER (
ORDER BY date
ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
)
-- Total from current row to end
SUM(sales) OVER (
ORDER BY date
ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
)
```
### 6.6 Practical Window Function Patterns
#### Running Total
```sql
SELECT
date,
amount,
SUM(amount) OVER (ORDER BY date) AS running_total
FROM transactions;
```
#### Year-Over-Year Comparison
```sql
SELECT
year,
month,
revenue,
LAG(revenue, 12) OVER (ORDER BY year, month) AS prev_year_revenue,
revenue - LAG(revenue, 12) OVER (ORDER BY year, month) AS yoy_change,
ROUND(100.0 * (revenue - LAG(revenue, 12) OVER (ORDER BY year, month))
/ LAG(revenue, 12) OVER (ORDER BY year, month), 2) AS yoy_pct_change
FROM monthly_revenue;
```
#### Top N Per Group
```sql
-- Get top 3 products in each category
WITH ranked AS (
SELECT
category,
product_name,
sales,
ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) AS rank
FROM products
)
SELECT * FROM ranked WHERE rank <= 3;
```
#### Percentage of Total
```sql
SELECT
product_name,
sales,
SUM(sales) OVER () AS total_sales,
ROUND(100.0 * sales / SUM(sales) OVER (), 2) AS pct_of_total
FROM products;
```
#### Identify First/Last in Group
```sql
SELECT
*,
CASE WHEN ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date) = 1
THEN 'First Order' END AS order_type
FROM orders;
```
---
## Chapter 7: The SELECT Statement - Bringing It All Together
The SELECT statement is the most powerful and frequently used statement in SQL. It embodies SQL's declarative philosophy: you describe *what* data you want, not *how* to retrieve it. The database engine determines the optimal execution strategy.
### The Declarative Nature of SELECT
Unlike imperative programming where you write step-by-step instructions, SELECT describes a result set. Consider this query:
```sql
SELECT c.name, COUNT(o.id) AS order_count
FROM customers c
LEFT JOIN orders o ON c.id = o.customer_id
WHERE c.status = 'active'
GROUP BY c.id, c.name
HAVING COUNT(o.id) > 5
ORDER BY order_count DESC;
```
You have not told the database:
- Which table to read first
- What algorithm to use for the join
- Whether to use an index
- How to perform the grouping
The query optimizer makes all these decisions based on table statistics, available indexes, and cost estimates. Two databases given the same query may execute it in completely different ways, yet both return the same result. This abstraction is powerful: as your data grows or you add indexes, the optimizer automatically adjusts without requiring you to rewrite queries.
### Understanding What SELECT Actually Does
At its core, SELECT transforms input relations (tables) into an output relation (the result set). Every clause in a SELECT statement performs a specific transformation:
1. **FROM/JOIN**: Establishes the working set by combining tables
2. **WHERE**: Filters rows that do not meet the criteria
3. **GROUP BY**: Partitions rows into groups
4. **HAVING**: Filters groups that do not meet criteria
5. **SELECT**: Projects columns and evaluates expressions
6. **DISTINCT**: Removes duplicate rows
7. **ORDER BY**: Sorts the result
8. **LIMIT/OFFSET**: Restricts the result to a subset
This pipeline model helps explain both what is possible and what is not. For example, you cannot reference a column alias in WHERE because aliases are created in the SELECT step, which happens after WHERE in the logical pipeline.
### 7.1 Understanding SELECT's Logical Processing Order
This is crucial for understanding what works where. Although you write SELECT first, SQL processes clauses in this order:
1. **FROM** - Identify source tables
2. **WHERE** - Filter rows
3. **GROUP BY** - Group rows
4. **HAVING** - Filter groups
5. **SELECT** - Evaluate expressions
6. **DISTINCT** - Remove duplicates
7. **ORDER BY** - Sort results
8. **LIMIT/OFFSET** - Limit results
**Why This Matters**:
```sql
-- Why can't I use a column alias in WHERE?
SELECT price * quantity AS total
FROM order_items
WHERE total > 100; -- ERROR: column "total" does not exist
-- Because WHERE is processed BEFORE SELECT!
-- Solution 1: Repeat the expression
SELECT price * quantity AS total
FROM order_items
WHERE price * quantity > 100;
-- Solution 2: Use a subquery
SELECT * FROM (
SELECT price * quantity AS total FROM order_items
) t
WHERE total > 100;
-- You CAN use aliases in ORDER BY (processed after SELECT)
SELECT price * quantity AS total
FROM order_items
ORDER BY total; -- Works!
```
### 7.2 JOINs: Combining Tables
Cognica supports all standard SQL join types.
```mermaid
flowchart TB
subgraph JoinTypes["JOIN Types"]
direction LR
subgraph Inner["INNER JOIN"]
I1[("A")] --- I2(("A ∩ B")) --- I3[("B")]
end
subgraph Left["LEFT JOIN"]
L1(("A")) --- L2(("A ∩ B")) --- L3[("B")]
end
subgraph Right["RIGHT JOIN"]
R1[("A")] --- R2(("A ∩ B")) --- R3(("B"))
end
subgraph Full["FULL JOIN"]
F1(("A")) --- F2(("A ∩ B")) --- F3(("B"))
end
end
Inner ---|"Matching rows only"| Result1["Results"]
Left ---|"All left + matching right"| Result2["Results"]
Right ---|"All right + matching left"| Result3["Results"]
Full ---|"All rows from both"| Result4["Results"]
```
#### INNER JOIN
Returns only rows that match in both tables:
```sql
SELECT users.name, orders.order_id, orders.total
FROM users
INNER JOIN orders ON users.id = orders.user_id;
```
If a user has no orders, they won't appear. If an order has no matching user, it won't appear.
#### LEFT JOIN (or LEFT OUTER JOIN)
Returns all rows from the left table, with NULL for non-matching right rows:
```sql
SELECT users.name, orders.order_id
FROM users
LEFT JOIN orders ON users.id = orders.user_id;
```
All users appear, even those without orders (with NULL for order columns).
**Finding Non-Matching Rows**:
```sql
-- Users who have never ordered
SELECT users.name
FROM users
LEFT JOIN orders ON users.id = orders.user_id
WHERE orders.order_id IS NULL;
```
#### RIGHT JOIN
Opposite of LEFT JOIN - all rows from right table:
```sql
SELECT users.name, orders.order_id
FROM users
RIGHT JOIN orders ON users.id = orders.user_id;
```
RIGHT JOIN is rarely used since you can always swap the tables and use LEFT JOIN.
#### FULL OUTER JOIN
Returns all rows from both tables:
```sql
SELECT users.name, orders.order_id
FROM users
FULL JOIN orders ON users.id = orders.user_id;
```
Useful for finding orphaned records in either direction.
#### CROSS JOIN
Cartesian product - every combination:
```sql
SELECT colors.name, sizes.name
FROM colors
CROSS JOIN sizes;
-- If colors has 3 rows and sizes has 4 rows, result has 12 rows
```
#### Self Join
A table joined to itself:
```sql
-- Employees and their managers
SELECT
e.name AS employee,
m.name AS manager
FROM employees e
LEFT JOIN employees m ON e.manager_id = m.id;
```
#### LATERAL Joins
A LATERAL join allows the right-hand side of a JOIN to reference columns from the left-hand side. This is similar to a correlated subquery, but as a join it can return multiple columns and multiple rows.
**LATERAL with subqueries:**
```sql
-- For each department, find the top 3 highest-paid employees
SELECT d.name AS department, top.name, top.salary
FROM departments d
CROSS JOIN LATERAL (
SELECT e.name, e.salary
FROM employees e
WHERE e.department_id = d.id -- References 'd' from outer query
ORDER BY e.salary DESC
LIMIT 3
) AS top;
```
Without LATERAL, the subquery in FROM cannot reference `d.id` because it is not correlated. LATERAL makes this correlation possible:
```sql
-- For each customer, find their most recent 5 orders
SELECT c.name, recent.order_id, recent.amount, recent.order_date
FROM customers c
CROSS JOIN LATERAL (
SELECT o.id AS order_id, o.amount, o.order_date
FROM orders o
WHERE o.customer_id = c.id
ORDER BY o.order_date DESC
LIMIT 5
) AS recent;
```
**LATERAL with table functions:**
```sql
-- Expand array elements alongside the originating row
SELECT p.name, tag
FROM products p
CROSS JOIN LATERAL unnest(p.tags) AS tag;
-- Generate date series for each project's duration
SELECT p.name, d::date AS project_day
FROM projects p
CROSS JOIN LATERAL generate_series(p.start_date, p.end_date, '1 day') AS d;
```
**LEFT JOIN LATERAL** ensures rows from the left side are preserved even if the LATERAL produces no rows:
```sql
-- All customers with their latest order (if any)
SELECT c.name, latest.order_date, latest.amount
FROM customers c
LEFT JOIN LATERAL (
SELECT o.order_date, o.amount
FROM orders o
WHERE o.customer_id = c.id
ORDER BY o.order_date DESC
LIMIT 1
) AS latest ON true;
```
**When to use LATERAL:**
- Top-N per group queries (more efficient than window functions for small N)
- Expanding set-returning functions alongside table data
- Complex computations that reference the current row
### 7.3 Subqueries: Queries Within Queries
#### Scalar Subqueries
Return a single value:
```sql
SELECT
name,
price,
(SELECT AVG(price) FROM products) AS avg_price,
price - (SELECT AVG(price) FROM products) AS diff_from_avg
FROM products;
```
#### Table Subqueries (Derived Tables)
Return a result set used as a table:
```sql
SELECT category, avg_price
FROM (
SELECT category, AVG(price) AS avg_price
FROM products
GROUP BY category
) AS category_stats
WHERE avg_price > 50;
```
#### Correlated Subqueries
Reference the outer query:
```sql
-- Products priced above their category average
SELECT name, price, category
FROM products p
WHERE price > (
SELECT AVG(price)
FROM products
WHERE category = p.category -- References outer query
);
```
#### EXISTS and NOT EXISTS
Test whether rows exist:
```sql
-- Customers with at least one order
SELECT * FROM customers c
WHERE EXISTS (
SELECT 1 FROM orders o WHERE o.customer_id = c.id
);
-- Customers with no orders
SELECT * FROM customers c
WHERE NOT EXISTS (
SELECT 1 FROM orders o WHERE o.customer_id = c.id
);
```
EXISTS is often more efficient than IN for correlated subqueries.
### 7.4 Common Table Expressions (CTEs)
CTEs make complex queries more readable by naming subqueries. The diagram below shows how a recursive CTE processes hierarchical data:
```mermaid
flowchart TB
subgraph CTE["WITH RECURSIVE org_tree"]
BASE["Base Case
(Anchor Query)"]
REC["Recursive Case
(References org_tree)"]
UNION["UNION ALL"]
end
subgraph Execution["Execution Flow"]
I1["Iteration 1:
Top-level managers"]
I2["Iteration 2:
Direct reports"]
I3["Iteration 3:
Next level..."]
IN["Iteration N:
Until empty"]
end
subgraph Result["Final Result"]
ALL["All Rows
Combined"]
end
BASE --> I1
I1 --> REC
REC --> UNION
UNION --> I2
I2 --> REC
REC --> UNION
UNION --> I3
I3 -.-> IN
IN -.->|"No more rows"| ALL
style BASE fill:#d4edda,color:#155724
style REC fill:#fff3cd,color:#856404
style ALL fill:#e3f2fd,color:#1565c0
```
```sql
WITH active_customers AS (
SELECT id, name
FROM customers
WHERE status = 'active'
),
recent_orders AS (
SELECT customer_id, SUM(total) AS total_spent
FROM orders
WHERE order_date > CURRENT_DATE - INTERVAL '30 days'
GROUP BY customer_id
)
SELECT
ac.name,
COALESCE(ro.total_spent, 0) AS recent_spending
FROM active_customers ac
LEFT JOIN recent_orders ro ON ac.id = ro.customer_id;
```
#### Recursive CTEs
For hierarchical data:
```sql
-- Organizational hierarchy
WITH RECURSIVE org_tree AS (
-- Base case: top-level (no manager)
SELECT id, name, manager_id, 1 AS level
FROM employees
WHERE manager_id IS NULL
UNION ALL
-- Recursive case: employees with managers
SELECT e.id, e.name, e.manager_id, t.level + 1
FROM employees e
INNER JOIN org_tree t ON e.manager_id = t.id
)
SELECT
REPEAT(' ', level - 1) || name AS org_chart,
level
FROM org_tree
ORDER BY level, name;
```
### 7.5 Set Operations
Combine results from multiple queries:
#### UNION
Combines and removes duplicates:
```sql
SELECT name, email FROM customers
UNION
SELECT name, email FROM suppliers;
```
#### UNION ALL
Combines and keeps all rows (faster):
```sql
SELECT id, 'customer' AS type FROM customers
UNION ALL
SELECT id, 'supplier' AS type FROM suppliers;
```
#### INTERSECT
Returns rows in both:
```sql
SELECT product_id FROM orders_2023
INTERSECT
SELECT product_id FROM orders_2024;
-- Products ordered in both years
```
#### EXCEPT
Returns rows in first but not second:
```sql
SELECT customer_id FROM customers_2023
EXCEPT
SELECT customer_id FROM customers_2024;
-- Customers from 2023 who aren't in 2024
```
### 7.6 ORDER BY and LIMIT
```sql
-- Basic ordering
SELECT * FROM products ORDER BY price DESC;
-- Multiple columns
SELECT * FROM employees ORDER BY department, salary DESC;
-- NULLS handling
SELECT * FROM users ORDER BY last_login NULLS LAST;
-- Pagination
SELECT * FROM products ORDER BY id LIMIT 20 OFFSET 40;
-- Page 3 of 20-item pages
```
**Performance Warning for OFFSET**: Large OFFSET values are slow because the database must scan and skip all those rows. For better performance with large datasets, use keyset pagination:
```sql
-- Instead of: WHERE ... LIMIT 20 OFFSET 10000
-- Use: WHERE id > last_seen_id ORDER BY id LIMIT 20
```
---
## Chapter 8: Modifying Data - INSERT, UPDATE, DELETE
While SELECT retrieves data without changing it, the statements in this chapter actually modify your database. These Data Manipulation Language (DML) statements - INSERT, UPDATE, and DELETE - deserve careful attention because mistakes can have lasting consequences.
### The Transactional Nature of DML
Every DML statement in Cognica executes within a transaction. By default, each statement is its own transaction that commits immediately upon success. This auto-commit behavior is convenient but can be dangerous:
```sql
-- These are two separate transactions
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
-- If the system crashes here, the money is gone!
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
```
For operations that must succeed or fail together, explicitly manage your transactions:
```sql
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT; -- Both changes apply atomically
```
Chapter 13 covers transactions in depth, but the key principle applies here: think about failure scenarios whenever you modify data.
### The SET-Based Paradigm
SQL operates on sets of rows, not individual records. This is both powerful and potentially dangerous:
```sql
-- This updates ALL rows where status = 'pending'
-- Could be 1 row, could be 1 million rows
UPDATE orders SET status = 'cancelled' WHERE status = 'pending';
```
There is no "undo" button. Before executing UPDATE or DELETE statements on production data, always:
1. Run a SELECT with the same WHERE clause to see what will be affected
2. Verify the row count matches your expectation
3. Consider doing the operation in a transaction you can roll back
4. For large updates, consider processing in batches
### Constraints and Triggers
DML statements do not operate in isolation. Before, during, and after your changes, the database enforces rules:
**Constraints** ensure data validity:
- NOT NULL constraints reject NULL values
- CHECK constraints validate expressions
- UNIQUE constraints prevent duplicates
- FOREIGN KEY constraints maintain referential integrity
**Triggers** execute custom logic:
- BEFORE triggers can modify or reject data before it is written
- AFTER triggers can perform follow-up actions
- Constraint triggers enforce complex business rules
A simple INSERT statement might fail for many reasons beyond syntax errors. Understanding these mechanisms helps you write robust data modification code.
### 8.1 INSERT: Adding New Rows
#### Basic INSERT
```sql
INSERT INTO users (name, email, created_at)
VALUES ('Alice', 'alice@example.com', NOW());
```
#### Multi-Row INSERT
Much more efficient than multiple single-row INSERTs:
```sql
INSERT INTO users (name, email) VALUES
('Alice', 'alice@example.com'),
('Bob', 'bob@example.com'),
('Charlie', 'charlie@example.com');
```
#### INSERT ... SELECT
Copy data from another query:
```sql
INSERT INTO archived_orders (order_id, customer_id, total, archived_at)
SELECT id, customer_id, total, NOW()
FROM orders
WHERE created_at < '2023-01-01';
```
#### INSERT ... ON CONFLICT (UPSERT)
Insert or update if exists:
```sql
-- Update if key exists
INSERT INTO user_settings (user_id, setting_key, setting_value)
VALUES (1, 'theme', 'dark')
ON CONFLICT (user_id, setting_key)
DO UPDATE SET setting_value = EXCLUDED.setting_value;
-- Do nothing if exists
INSERT INTO users (id, name) VALUES (1, 'Alice')
ON CONFLICT (id) DO NOTHING;
```
`EXCLUDED` refers to the row that would have been inserted.
#### INSERT ... RETURNING
Get back values from the inserted row:
```sql
INSERT INTO orders (customer_id, total)
VALUES (1, 99.99)
RETURNING id, created_at;
-- Returns the auto-generated id and timestamp
-- Use with CTEs
WITH new_order AS (
INSERT INTO orders (customer_id, total) VALUES (1, 99.99)
RETURNING id
)
INSERT INTO order_items (order_id, product_id, quantity)
SELECT new_order.id, 42, 1 FROM new_order;
```
### 8.2 UPDATE: Modifying Existing Rows
#### Basic UPDATE
```sql
UPDATE users SET status = 'active' WHERE id = 1;
UPDATE products
SET
price = price * 1.1,
updated_at = NOW()
WHERE category = 'Electronics';
```
#### UPDATE ... FROM
Join with another table:
```sql
UPDATE orders o
SET status = 'shipped', shipped_at = s.ship_date
FROM shipments s
WHERE o.id = s.order_id
AND s.status = 'completed';
```
#### UPDATE ... RETURNING
```sql
UPDATE users
SET login_count = login_count + 1
WHERE id = 1
RETURNING login_count, last_login;
```
### 8.3 DELETE: Removing Rows
#### Basic DELETE
```sql
DELETE FROM sessions WHERE expires_at < NOW();
```
#### DELETE ... USING
Join with another table:
```sql
DELETE FROM order_items oi
USING orders o
WHERE oi.order_id = o.id
AND o.status = 'cancelled';
```
#### DELETE ... RETURNING
```sql
DELETE FROM expired_tokens
WHERE expires_at < NOW()
RETURNING token_id, user_id;
```
#### TRUNCATE: Fast Delete All
```sql
TRUNCATE TABLE logs; -- Much faster than DELETE FROM logs
```
TRUNCATE is faster because it doesn't scan rows. However, it can't be used with WHERE and has different transactional behavior.
### 8.4 Schema Management
Schemas are namespaces that organize database objects (tables, views, functions, etc.). The default schema is `public`.
#### CREATE SCHEMA
```sql
-- Create a schema
CREATE SCHEMA finance;
-- Create only if it doesn't exist
CREATE SCHEMA IF NOT EXISTS analytics;
-- Create with an owner
CREATE SCHEMA reporting AUTHORIZATION analyst_role;
```
**Protected schema names** that cannot be created: `pg_catalog`, `information_schema`, `pg_toast`, `pg_temp`.
By default, new schemas grant `USAGE` and `CREATE` privileges to PUBLIC.
#### DROP SCHEMA
```sql
-- Drop a schema (must be empty)
DROP SCHEMA finance;
-- Drop only if it exists
DROP SCHEMA IF EXISTS temp_schema;
-- Drop schema and all contained objects
DROP SCHEMA analytics CASCADE;
```
**Protected schema names** that cannot be dropped: `public`, `pg_catalog`, `information_schema`.
#### ALTER SCHEMA
```sql
-- Rename a schema
ALTER SCHEMA old_name RENAME TO new_name;
-- Change schema owner
ALTER SCHEMA finance OWNER TO finance_admin;
```
#### Using Schemas
```sql
-- Create a table in a specific schema
CREATE TABLE finance.transactions (
id SERIAL PRIMARY KEY,
amount NUMERIC(10,2) NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- Query with schema prefix
SELECT * FROM finance.transactions WHERE amount > 1000;
-- Set the search path to include your schema
SET search_path TO finance, public;
-- Now you can query without the schema prefix
SELECT * FROM transactions;
```
### 8.5 Sequence Management
Sequences are independent database objects that generate sequential numbers. They are automatically created for `SERIAL` and `BIGSERIAL` columns.
#### CREATE SEQUENCE
```sql
CREATE SEQUENCE order_number_seq
START WITH 1000
INCREMENT BY 1
MINVALUE 1000
MAXVALUE 9999999
NO CYCLE
CACHE 10;
-- Create only if it doesn't exist
CREATE SEQUENCE IF NOT EXISTS invoice_seq START WITH 1;
-- Owned by a column (auto-dropped when column is dropped)
CREATE SEQUENCE item_id_seq OWNED BY items.id;
```
**Options:**
| Option | Description | Default |
|--------|-------------|---------|
| `START WITH` | First value returned | 1 |
| `INCREMENT BY` | Step between values | 1 |
| `MINVALUE` / `NO MINVALUE` | Minimum value | 1 |
| `MAXVALUE` / `NO MAXVALUE` | Maximum value | 2^63 - 1 |
| `CYCLE` / `NO CYCLE` | Wrap around when limit reached | NO CYCLE |
| `CACHE` | Pre-allocate values for performance | 1 |
| `OWNED BY` | Link to a table column | NONE |
#### ALTER SEQUENCE
```sql
-- Change the increment
ALTER SEQUENCE order_number_seq INCREMENT BY 10;
-- Restart the sequence
ALTER SEQUENCE order_number_seq RESTART WITH 5000;
-- Change ownership
ALTER SEQUENCE item_id_seq OWNED BY new_table.id;
```
#### DROP SEQUENCE
```sql
DROP SEQUENCE order_number_seq;
DROP SEQUENCE IF EXISTS temp_seq CASCADE;
```
#### Using Sequences
```sql
-- Get the next value (advances the counter)
SELECT nextval('order_number_seq'); -- 1000
-- Get the current value (must call nextval first in session)
SELECT currval('order_number_seq'); -- 1000
-- Set the sequence to a specific value
SELECT setval('order_number_seq', 5000); -- Next nextval returns 5001
-- Use in INSERT
INSERT INTO orders (order_number, customer_id)
VALUES (nextval('order_number_seq'), 42);
```
---
## Chapter 9: Constraints - Enforcing Data Integrity
Constraints are rules that the database enforces to maintain data integrity. They prevent invalid data from being inserted and ensure relationships between tables remain consistent.
### The Philosophy of Database Constraints
Data types provide a coarse mechanism for controlling data: an INTEGER column rejects strings. Constraints provide fine-grained control: an INTEGER column with a CHECK constraint can require values between 1 and 100.
The fundamental principle is: **define validity as close to the data as possible**. When constraints are defined in the database:
1. **Enforcement is universal**: All applications, scripts, and ad-hoc queries respect the rules
2. **No bypass is possible**: Even raw SQL cannot violate constraints
3. **Documentation is built-in**: The schema describes what data is valid
4. **Errors occur immediately**: Invalid data is rejected at write time, not discovered later
### Database Constraints vs. Application Validation
A common question: "Should I validate in my application or in the database?"
The answer is: **both, but differently**.
**Application Validation:**
- Provides immediate user feedback with helpful messages
- Can validate complex business logic that changes frequently
- Can validate data before attempting to write
- Can be bypassed (intentionally or accidentally)
**Database Constraints:**
- Provide the last line of defense against invalid data
- Are declarative and easy to understand
- Cannot be bypassed by any client
- Should handle rules that must never be violated
Think of application validation as the helpful receptionist and database constraints as the security guard. The receptionist politely redirects you if you are in the wrong building. The security guard stops you if you try to enter a restricted area regardless of what the receptionist said.
### Performance Considerations
Constraints are not free. Every INSERT and UPDATE must verify constraint conditions:
- **NOT NULL**: Trivially fast
- **UNIQUE/PRIMARY KEY**: Requires index lookup
- **FOREIGN KEY**: Requires lookup in referenced table
- **CHECK**: Evaluates expression (speed depends on complexity)
For bulk loading operations, some databases allow temporarily disabling constraints. Cognica does not recommend this approach as it can leave your database in an invalid state. Instead, validate your data before loading.
### 9.1 Understanding Constraints
#### Why Constraints Matter
Without constraints, your database can accumulate invalid data:
```sql
-- Without constraints, these are all "valid":
INSERT INTO users (id, email) VALUES (1, NULL); -- No email?
INSERT INTO users (id, email) VALUES (1, 'test'); -- Duplicate ID!
INSERT INTO orders (customer_id) VALUES (99999); -- Non-existent customer!
INSERT INTO products (price) VALUES (-50); -- Negative price!
```
With constraints, the database rejects invalid data:
```sql
-- With proper constraints:
INSERT INTO users (id, email) VALUES (1, NULL);
-- ERROR: null value in column "email" violates not-null constraint
INSERT INTO users (id, email) VALUES (1, 'test');
-- ERROR: duplicate key value violates unique constraint "users_pkey"
INSERT INTO orders (customer_id) VALUES (99999);
-- ERROR: insert or update on table "orders" violates foreign key constraint
INSERT INTO products (price) VALUES (-50);
-- ERROR: new row for relation "products" violates check constraint "products_price_check"
```
#### Types of Constraints
| Constraint | Purpose | Example |
|-----------|---------|---------|
| `NOT NULL` | Column must have a value | Email required |
| `UNIQUE` | Values must be unique | No duplicate usernames |
| `PRIMARY KEY` | Unique identifier for rows | `id` column |
| `FOREIGN KEY` | References another table | Order belongs to customer |
| `CHECK` | Custom validation rule | Price must be positive |
### 9.2 NOT NULL Constraint
Ensures a column always has a value:
```sql
-- In CREATE TABLE
CREATE TABLE users (
id INTEGER PRIMARY KEY,
email VARCHAR(255) NOT NULL, -- Required
name VARCHAR(100) NOT NULL, -- Required
bio TEXT -- Optional (allows NULL)
);
-- Add NOT NULL to existing column
ALTER TABLE users ALTER COLUMN phone SET NOT NULL;
-- Remove NOT NULL constraint
ALTER TABLE users ALTER COLUMN phone DROP NOT NULL;
```
**Gotcha**: Before adding NOT NULL, ensure no existing rows have NULL values:
```sql
-- First, fix any NULL values
UPDATE users SET phone = 'unknown' WHERE phone IS NULL;
-- Then add the constraint
ALTER TABLE users ALTER COLUMN phone SET NOT NULL;
```
### 9.3 UNIQUE Constraint
Ensures all values in a column (or combination of columns) are unique:
```sql
-- Single column unique
CREATE TABLE users (
id SERIAL PRIMARY KEY,
email VARCHAR(255) UNIQUE, -- No duplicate emails
username VARCHAR(50) UNIQUE -- No duplicate usernames
);
-- Named constraint
CREATE TABLE users (
id SERIAL PRIMARY KEY,
email VARCHAR(255),
CONSTRAINT users_email_unique UNIQUE (email)
);
-- Multi-column unique (composite)
CREATE TABLE subscriptions (
user_id INTEGER,
plan_id INTEGER,
-- Each user can have only one subscription per plan
CONSTRAINT unique_user_plan UNIQUE (user_id, plan_id)
);
```
**NULL Behavior**: UNIQUE allows multiple NULL values (NULL is not equal to NULL):
```sql
INSERT INTO users (email) VALUES (NULL); -- OK
INSERT INTO users (email) VALUES (NULL); -- Also OK (NULLs don't conflict)
```
#### Adding UNIQUE to Existing Table
```sql
-- Add unique constraint
ALTER TABLE users ADD CONSTRAINT users_email_unique UNIQUE (email);
-- If duplicates exist, you'll get an error
-- First find and fix duplicates:
SELECT email, COUNT(*) FROM users GROUP BY email HAVING COUNT(*) > 1;
```
### 9.4 PRIMARY KEY Constraint
Combines NOT NULL and UNIQUE to create a unique identifier for each row:
```sql
-- Single column primary key
CREATE TABLE users (
id SERIAL PRIMARY KEY,
email VARCHAR(255) NOT NULL
);
-- Equivalent to:
CREATE TABLE users (
id SERIAL NOT NULL UNIQUE,
email VARCHAR(255) NOT NULL
);
-- But PRIMARY KEY also signals this is THE identifier
-- Composite primary key
CREATE TABLE order_items (
order_id INTEGER,
product_id INTEGER,
quantity INTEGER,
PRIMARY KEY (order_id, product_id) -- Combination is unique
);
-- Named primary key
CREATE TABLE products (
id INTEGER,
CONSTRAINT products_pkey PRIMARY KEY (id)
);
```
**Best Practices**:
- Every table should have a primary key
- Use simple keys (INTEGER, BIGINT) for performance
- Consider SERIAL or UUID for auto-generated keys
### 9.5 FOREIGN KEY Constraint
Ensures values reference existing rows in another table:
```sql
-- Basic foreign key
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER REFERENCES customers(id),
total DECIMAL(10,2)
);
-- Named constraint with explicit syntax
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER,
total DECIMAL(10,2),
CONSTRAINT fk_orders_customer
FOREIGN KEY (customer_id)
REFERENCES customers(id)
);
-- Composite foreign key
CREATE TABLE order_items (
id SERIAL PRIMARY KEY,
order_id INTEGER,
product_id INTEGER,
CONSTRAINT fk_orderitems_order
FOREIGN KEY (order_id) REFERENCES orders(id),
CONSTRAINT fk_orderitems_product
FOREIGN KEY (product_id) REFERENCES products(id)
);
```
#### Referential Actions
What happens when the referenced row is deleted or updated?
```sql
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER,
CONSTRAINT fk_orders_customer
FOREIGN KEY (customer_id)
REFERENCES customers(id)
ON DELETE CASCADE -- Delete order when customer is deleted
ON UPDATE CASCADE -- Update FK when customer ID changes
);
```
**Available Actions**:
| Action | On DELETE | On UPDATE |
|--------|-----------|-----------|
| `NO ACTION` | Error (default) | Error (default) |
| `RESTRICT` | Error (checked immediately) | Error (checked immediately) |
| `CASCADE` | Delete referencing rows | Update FK values |
| `SET NULL` | Set FK to NULL | Set FK to NULL |
| `SET DEFAULT` | Set FK to default value | Set FK to default value |
**Examples**:
```sql
-- CASCADE: Delete all orders when customer is deleted
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER REFERENCES customers(id) ON DELETE CASCADE
);
-- SET NULL: Keep orders but clear customer reference
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER REFERENCES customers(id) ON DELETE SET NULL
);
-- RESTRICT: Prevent deletion if orders exist
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER REFERENCES customers(id) ON DELETE RESTRICT
);
```
#### NULL Handling in Foreign Keys
```sql
-- Foreign key columns can be NULL (unless also NOT NULL)
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER REFERENCES customers(id), -- Can be NULL
guest_email VARCHAR(255) -- For guest checkouts
);
-- Insert with NULL FK is allowed
INSERT INTO orders (guest_email) VALUES ('guest@example.com');
```
### 9.6 CHECK Constraint
Validates data against a custom expression:
```sql
-- Single column check
CREATE TABLE products (
id SERIAL PRIMARY KEY,
name VARCHAR(100) NOT NULL,
price DECIMAL(10,2) CHECK (price > 0), -- Price must be positive
quantity INTEGER CHECK (quantity >= 0) -- No negative quantities
);
-- Named check constraint
CREATE TABLE products (
id SERIAL PRIMARY KEY,
price DECIMAL(10,2),
CONSTRAINT products_price_positive CHECK (price > 0)
);
-- Multi-column check
CREATE TABLE events (
id SERIAL PRIMARY KEY,
start_time TIMESTAMPTZ NOT NULL,
end_time TIMESTAMPTZ NOT NULL,
CONSTRAINT events_time_valid CHECK (end_time > start_time)
);
-- Complex check with multiple conditions
CREATE TABLE employees (
id SERIAL PRIMARY KEY,
salary DECIMAL(10,2),
department VARCHAR(50),
CONSTRAINT valid_salary CHECK (
salary >= 30000 AND salary <= 500000
),
CONSTRAINT valid_department CHECK (
department IN ('Engineering', 'Sales', 'Marketing', 'HR')
)
);
```
**CHECK Limitations**:
- Cannot reference other tables (use triggers instead)
- Cannot use subqueries
- Cannot use volatile functions like `RANDOM()` or `NOW()`
### 9.7 Deferrable Constraints
Normally, constraints are checked immediately. Deferrable constraints can be checked at the end of a transaction:
```sql
-- Create deferrable constraint
CREATE TABLE accounts (
id SERIAL PRIMARY KEY,
balance DECIMAL(10,2),
CONSTRAINT positive_balance CHECK (balance >= 0)
DEFERRABLE INITIALLY IMMEDIATE
);
-- Create initially deferred constraint
CREATE TABLE transfers (
from_account INTEGER REFERENCES accounts(id)
DEFERRABLE INITIALLY DEFERRED,
to_account INTEGER REFERENCES accounts(id)
DEFERRABLE INITIALLY DEFERRED,
amount DECIMAL(10,2)
);
```
**Usage in Transactions**:
```sql
-- Set constraints to be deferred for this transaction
BEGIN;
SET CONSTRAINTS positive_balance DEFERRED;
-- This would normally fail (balance goes negative)
UPDATE accounts SET balance = balance - 1000 WHERE id = 1;
-- But the check is deferred...
UPDATE accounts SET balance = balance + 1000 WHERE id = 2;
-- Now total balance is correct
COMMIT; -- Constraint is checked here
```
**Use Cases**:
- Circular references that must be inserted together
- Batch operations where intermediate states are invalid
- Complex multi-table updates
### 9.8 Managing Constraints
#### Adding Constraints to Existing Tables
```sql
-- Add PRIMARY KEY
ALTER TABLE users ADD CONSTRAINT users_pkey PRIMARY KEY (id);
-- Add UNIQUE
ALTER TABLE users ADD CONSTRAINT users_email_unique UNIQUE (email);
-- Add FOREIGN KEY
ALTER TABLE orders ADD CONSTRAINT fk_orders_customer
FOREIGN KEY (customer_id) REFERENCES customers(id);
-- Add CHECK
ALTER TABLE products ADD CONSTRAINT positive_price CHECK (price > 0);
-- Add NOT NULL
ALTER TABLE users ALTER COLUMN email SET NOT NULL;
```
#### NOT VALID Constraints
Add constraint without validating existing data (faster for large tables):
```sql
-- Add constraint but don't validate existing rows
ALTER TABLE large_table
ADD CONSTRAINT check_status CHECK (status IN ('active', 'inactive'))
NOT VALID;
-- Later, validate the constraint
ALTER TABLE large_table VALIDATE CONSTRAINT check_status;
```
**Use Case**: Add constraint to production table with millions of rows without locking.
#### Dropping Constraints
```sql
-- Drop by name
ALTER TABLE users DROP CONSTRAINT users_email_unique;
-- Drop if exists
ALTER TABLE users DROP CONSTRAINT IF EXISTS users_email_unique;
-- Drop primary key
ALTER TABLE users DROP CONSTRAINT users_pkey;
-- Drop NOT NULL
ALTER TABLE users ALTER COLUMN email DROP NOT NULL;
```
#### Renaming Constraints
```sql
-- Rename a constraint
ALTER TABLE users RENAME CONSTRAINT old_name TO new_name;
```
#### Viewing Constraints
```sql
-- View all constraints on a table
SELECT
conname AS constraint_name,
contype AS type,
pg_get_constraintdef(oid) AS definition
FROM pg_constraint
WHERE conrelid = 'users'::regclass;
-- Types: p = primary key, f = foreign key, u = unique, c = check
```
#### Column Modifications
Beyond constraints, ALTER TABLE supports modifying column definitions:
**Renaming Columns:**
```sql
-- Rename a column
ALTER TABLE users RENAME COLUMN name TO full_name;
-- Rename preserves all constraints, indexes, and references on the column
ALTER TABLE orders RENAME COLUMN total TO order_total;
```
**Changing Column Types:**
```sql
-- Change column type
ALTER TABLE products ALTER COLUMN price TYPE NUMERIC(10, 2);
-- Change from TEXT to INTEGER (data must be convertible)
ALTER TABLE settings ALTER COLUMN max_retries TYPE INTEGER;
-- Change with explicit conversion using USING clause
ALTER TABLE events ALTER COLUMN created_at TYPE TIMESTAMPTZ
USING created_at AT TIME ZONE 'UTC';
```
**Adding and Dropping Columns:**
```sql
-- Add a new column
ALTER TABLE users ADD COLUMN phone TEXT;
-- Add a column with a default value
ALTER TABLE orders ADD COLUMN priority INTEGER DEFAULT 0;
-- Drop a column
ALTER TABLE users DROP COLUMN phone;
-- Drop only if the column exists
ALTER TABLE users DROP COLUMN IF EXISTS legacy_field;
```
### 9.9 Constraint Best Practices
#### Name Your Constraints
```sql
-- BAD: Auto-generated names are hard to reference
CREATE TABLE orders (
customer_id INTEGER REFERENCES customers(id)
);
-- GOOD: Explicit names are clear and maintainable
CREATE TABLE orders (
customer_id INTEGER,
CONSTRAINT fk_orders_customer
FOREIGN KEY (customer_id) REFERENCES customers(id)
);
```
Naming convention: `{table}_{column(s)}_{type}`
- `users_email_unique`
- `orders_customer_id_fkey`
- `products_price_check`
- `users_pkey`
#### Design Foreign Keys Carefully
```sql
-- Consider what should happen on delete
CREATE TABLE comments (
id SERIAL PRIMARY KEY,
post_id INTEGER REFERENCES posts(id) ON DELETE CASCADE,
-- When post is deleted, delete all comments
author_id INTEGER REFERENCES users(id) ON DELETE SET NULL
-- When user is deleted, keep comments but remove author
);
```
#### Use CHECK for Business Rules
```sql
-- Enforce valid status transitions
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
status VARCHAR(20),
CONSTRAINT valid_status CHECK (
status IN ('pending', 'paid', 'shipped', 'delivered', 'cancelled')
)
);
-- Enforce date logic
CREATE TABLE projects (
id SERIAL PRIMARY KEY,
start_date DATE NOT NULL,
end_date DATE,
CONSTRAINT valid_dates CHECK (end_date IS NULL OR end_date >= start_date)
);
```
#### Combine Constraints Appropriately
```sql
-- Full example with all constraint types
CREATE TABLE products (
id SERIAL,
sku VARCHAR(50) NOT NULL,
name VARCHAR(200) NOT NULL,
description TEXT,
price DECIMAL(10,2) NOT NULL,
category_id INTEGER NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ,
-- Primary key
CONSTRAINT products_pkey PRIMARY KEY (id),
-- Unique constraints
CONSTRAINT products_sku_unique UNIQUE (sku),
-- Foreign key
CONSTRAINT products_category_fkey
FOREIGN KEY (category_id)
REFERENCES categories(id)
ON DELETE RESTRICT,
-- Check constraints
CONSTRAINT products_price_positive CHECK (price > 0),
CONSTRAINT products_updated_after_created CHECK (
updated_at IS NULL OR updated_at >= created_at
)
);
```
#### Performance Considerations
```sql
-- Foreign keys create indexes on referenced columns automatically
-- But NOT on the referencing columns - add these yourself:
CREATE TABLE orders (
id SERIAL PRIMARY KEY,
customer_id INTEGER REFERENCES customers(id)
);
-- Add index for faster FK lookups
CREATE INDEX idx_orders_customer_id ON orders(customer_id);
-- This speeds up:
-- - JOIN orders ON customer_id
-- - CASCADE DELETE performance
-- - Queries filtering by customer_id
```
---
## Chapter 10: Transactions - Ensuring Data Integrity
Transactions are fundamental to database reliability. They ensure that a series of operations either all succeed or all fail together, maintaining data consistency even when things go wrong.
```mermaid
flowchart LR
subgraph Transaction["Transaction Lifecycle"]
BEGIN["BEGIN"]
OPS["SQL Operations
(INSERT, UPDATE, DELETE)"]
DECISION{Success?}
COMMIT["COMMIT"]
ROLLBACK["ROLLBACK"]
end
subgraph Result["Outcome"]
PERSISTED["Changes
Persisted"]
DISCARDED["Changes
Discarded"]
end
BEGIN --> OPS
OPS --> DECISION
DECISION -->|Yes| COMMIT
DECISION -->|No| ROLLBACK
COMMIT --> PERSISTED
ROLLBACK --> DISCARDED
style BEGIN fill:#4a90d9,color:#fff
style COMMIT fill:#28a745,color:#fff
style ROLLBACK fill:#dc3545,color:#fff
style PERSISTED fill:#d4edda,color:#155724
style DISCARDED fill:#f8d7da,color:#721c24
```
### Why Transactions Matter
Consider what happens without transactions. You want to transfer $100 between two accounts:
1. Debit $100 from Account A
2. Credit $100 to Account B
If the system crashes between steps 1 and 2, money vanishes. If another user reads the accounts during this operation, they see inconsistent data. Transactions solve both problems by treating the two operations as a single, indivisible unit.
### The Transaction Mental Model
Think of a transaction as a protected workspace. When you `BEGIN` a transaction:
1. Your changes are isolated from other users
2. Other users' committed changes are visible to you (in most isolation levels)
3. Nothing you do is permanent until you `COMMIT`
4. If anything goes wrong, `ROLLBACK` undoes everything
This workspace metaphor helps explain why long-running transactions are problematic: they hold resources and potentially block other users while the workspace remains open.
### Common Transaction Pitfalls
**1. Forgetting You Are in a Transaction**: In some client tools, transactions start automatically. If you make changes and close the connection without committing, all work is lost.
**2. Holding Transactions Open Too Long**: While your transaction is open, locks may prevent other users from modifying the same data. Keep transactions as short as possible.
**3. Misunderstanding Isolation**: Different isolation levels provide different guarantees. What you see may not be what other users see at the same moment. The section on isolation levels explains these subtleties.
**4. Assuming Atomicity Across Statements**: Without explicit `BEGIN`, each statement is its own transaction. Two separate UPDATE statements can have a failure between them.
### 10.1 Understanding ACID Properties
Every transaction in Cognica guarantees four properties, known by the acronym ACID:
#### Atomicity: All or Nothing
A transaction is indivisible. Either all operations complete successfully, or none of them do.
```sql
-- Transfer money between accounts
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1; -- Withdraw
UPDATE accounts SET balance = balance + 100 WHERE id = 2; -- Deposit
COMMIT;
-- If anything fails (power outage, error), both operations roll back
-- You'll never have money disappear or appear from nowhere
```
**Without Atomicity**: If the system crashed between the two UPDATE statements, account 1 would be debited but account 2 wouldn't be credited. Money would vanish.
#### Consistency: Rules Are Respected
Transactions can only bring the database from one valid state to another. All constraints, triggers, and rules are enforced.
```sql
-- This transaction will fail if it violates constraints
BEGIN;
INSERT INTO orders (customer_id, total) VALUES (999, 100);
-- Fails if customer 999 doesn't exist (foreign key constraint)
COMMIT;
```
#### Isolation: Transactions Don't Interfere
Concurrent transactions don't see each other's uncommitted changes (the degree depends on isolation level).
```sql
-- Transaction A -- Transaction B
BEGIN; BEGIN;
SELECT balance FROM accounts SELECT balance FROM accounts
WHERE id = 1; -- Returns 1000 WHERE id = 1; -- Also returns 1000
UPDATE accounts SET balance = 900
WHERE id = 1;
-- B still sees 1000 (isolation!)
COMMIT;
SELECT balance FROM accounts
WHERE id = 1; -- Now sees 900
COMMIT;
```
#### Durability: Changes Persist
Once a transaction commits, the changes survive system failures.
```sql
BEGIN;
INSERT INTO audit_log (event, timestamp) VALUES ('User login', NOW());
COMMIT;
-- If COMMIT succeeds, this record is permanently stored
-- Even if power fails 1 millisecond later, the record is safe
```
### 10.2 Transaction Control Commands
#### BEGIN, COMMIT, ROLLBACK
```sql
-- Start a transaction
BEGIN;
-- or: START TRANSACTION;
-- Execute operations
INSERT INTO orders (customer_id) VALUES (1);
UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 42;
-- Commit (save all changes)
COMMIT;
-- or: END;
-- Or rollback (discard all changes)
ROLLBACK;
```
#### Auto-Commit Mode
Without explicit BEGIN, each statement runs in its own transaction:
```sql
-- Each of these is a separate transaction
INSERT INTO logs (msg) VALUES ('Event 1'); -- Auto-commits
INSERT INTO logs (msg) VALUES ('Event 2'); -- Auto-commits
INSERT INTO logs (msg) VALUES ('Event 3'); -- Auto-commits
```
### 10.3 Savepoints: Partial Rollback
Savepoints let you roll back part of a transaction without losing everything:
```sql
BEGIN;
INSERT INTO orders (customer_id) VALUES (1);
SAVEPOINT before_items;
INSERT INTO order_items (order_id, product_id) VALUES (1, 100);
INSERT INTO order_items (order_id, product_id) VALUES (1, 200);
-- Oops, product 200 is out of stock
ROLLBACK TO SAVEPOINT before_items;
-- The order_items inserts are undone, but the order insert remains
INSERT INTO order_items (order_id, product_id) VALUES (1, 100);
INSERT INTO order_items (order_id, product_id) VALUES (1, 300); -- Different product
COMMIT; -- Order with products 100 and 300 is saved
```
**Savepoint Management**:
```sql
SAVEPOINT my_savepoint; -- Create savepoint
ROLLBACK TO SAVEPOINT my_savepoint; -- Roll back to it
RELEASE SAVEPOINT my_savepoint; -- Delete it (optional)
```
### 10.4 Isolation Levels
Isolation levels control how transactions interact with each other. Higher isolation means fewer anomalies but potentially lower concurrency.
#### READ COMMITTED (Default)
Each statement sees only data committed before that statement began:
```sql
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
BEGIN;
-- Statement 1 sees committed data as of statement 1's start
SELECT * FROM accounts WHERE id = 1;
-- ... time passes, other transactions commit ...
-- Statement 2 sees committed data as of statement 2's start
-- (might be different from what statement 1 saw!)
SELECT * FROM accounts WHERE id = 1;
COMMIT;
```
**Possible Anomaly - Non-Repeatable Read**:
```sql
-- Transaction A -- Transaction B
BEGIN; BEGIN;
SELECT balance FROM accounts
WHERE id = 1; -- Returns 1000
UPDATE accounts SET balance = 500
WHERE id = 1;
COMMIT;
SELECT balance FROM accounts
WHERE id = 1; -- Returns 500!
-- Same query, different result!
COMMIT;
```
**When to Use**: Default for most applications. Good balance of isolation and performance.
#### REPEATABLE READ
The entire transaction sees a consistent snapshot from its start:
```sql
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
BEGIN;
SELECT balance FROM accounts WHERE id = 1; -- Returns 1000
-- Even if another transaction updates and commits,
-- we'll still see 1000 for the rest of our transaction
SELECT balance FROM accounts WHERE id = 1; -- Still returns 1000!
COMMIT;
```
**Possible Anomaly - Phantom Reads**:
```sql
-- Transaction A (REPEATABLE READ) -- Transaction B
BEGIN; BEGIN;
SELECT COUNT(*) FROM users
WHERE age > 30; -- Returns 100
INSERT INTO users (name, age)
VALUES ('New User', 35);
COMMIT;
SELECT COUNT(*) FROM users
WHERE age > 30; -- Might return 101!
-- Existing rows are stable, but new rows (phantoms) may appear
COMMIT;
```
**When to Use**: Long-running reports, analytics queries, or when you need consistent reads throughout a transaction.
#### SERIALIZABLE
Strictest level - transactions appear to execute one after another. Cognica implements true Serializable Snapshot Isolation (SSI) based on the research by Cahill et al. (2008) and Ports & Grittner (2012).
```sql
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN;
-- This transaction behaves as if no other transactions are running
-- If conflicts are detected, one transaction is aborted
```
**How SSI Works:**
SSI detects potential serialization anomalies by tracking read-write dependencies between concurrent transactions. When a "dangerous structure" is detected (a cycle of read-write conflicts), one of the involved transactions is aborted with SQLSTATE 40001 (`serialization_failure`).
Key properties:
- **No false negatives**: All anomalies are detected and prevented
- **Rare false positives**: Some safe transaction patterns may trigger unnecessary aborts, but this is uncommon
- **Zero overhead for non-SERIALIZABLE transactions**: SSI tracking only activates for SERIALIZABLE transactions
- **Read-only optimization**: Read-only transactions are never chosen as the abort target
- **Lock promotion**: Per-document read tracking promotes to collection-level after 256 locks to limit memory usage
**Trade-off**: May abort transactions that conflict, requiring retry logic:
```sql
-- Application code pattern for SERIALIZABLE (pseudocode)
WHILE true:
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;
try:
-- Do your work
COMMIT;
break; -- Success!
catch SerializationFailure:
ROLLBACK;
-- Retry the entire transaction
```
**DEFERRABLE transactions**: Read-only SERIALIZABLE transactions can use DEFERRABLE to wait for a safe snapshot rather than risk abort:
```sql
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE;
BEGIN;
-- May wait briefly for a safe snapshot, but is guaranteed to never abort
SELECT * FROM accounts;
COMMIT;
```
**When to Use**: Critical operations where correctness is paramount and you can handle retries. Financial transfers, inventory management, and constraint enforcement across multiple tables are classic use cases.
### 10.5 Two-Phase Commit (Distributed Transactions)
Two-phase commit (2PC) enables transaction coordination across multiple systems. A transaction is first prepared (made durable but not committed), and then committed or rolled back as a separate step.
```sql
-- Phase 1: Prepare the transaction with a global identifier
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
PREPARE TRANSACTION 'transfer_001';
-- Transaction is now persistent but not committed
-- Phase 2: Commit or rollback the prepared transaction
COMMIT PREPARED 'transfer_001';
-- or: ROLLBACK PREPARED 'transfer_001';
```
**Properties:**
- Prepared transactions survive server restarts (persisted to storage)
- The global ID must be unique across all active prepared transactions
- Prepared transactions hold their locks until committed or rolled back
- Recovery on startup automatically rediscovers prepared transactions
**When to Use**: Coordinating transactions across multiple databases or services in distributed systems.
### 10.6 Transaction Best Practices
#### Keep Transactions Short
```sql
-- BAD: Long transaction holding locks
BEGIN;
SELECT * FROM large_table; -- Reads millions of rows
-- ... application does processing for 30 seconds ...
UPDATE accounts SET balance = 100 WHERE id = 1;
COMMIT;
-- GOOD: Short, focused transaction
-- Do the read and processing outside a transaction
-- Then:
BEGIN;
UPDATE accounts SET balance = 100 WHERE id = 1;
COMMIT;
```
#### Handle Errors Properly
```sql
BEGIN;
-- Operation 1
INSERT INTO orders (customer_id) VALUES (1);
-- If this fails, the transaction is aborted
-- WRONG: Trying to continue after error
INSERT INTO order_items (order_id, product_id) VALUES (999, 1); -- Error!
-- Transaction is now aborted
INSERT INTO something_else ...; -- This will also fail!
COMMIT; -- This becomes a ROLLBACK
-- RIGHT: Use savepoints for partial recovery
BEGIN;
SAVEPOINT sp1;
INSERT INTO order_items (order_id, product_id) VALUES (999, 1); -- Error!
ROLLBACK TO sp1; -- Recover from error
-- Can continue with other operations
COMMIT;
```
#### Avoid User Interaction Inside Transactions
```sql
-- BAD: Waiting for user input inside transaction
BEGIN;
SELECT * FROM products WHERE id = 1;
-- Wait for user to confirm... (locks held!)
UPDATE products SET quantity = quantity - 1 WHERE id = 1;
COMMIT;
-- GOOD: Gather input first, then quick transaction
-- Get user confirmation first (no transaction)
BEGIN;
UPDATE products SET quantity = quantity - 1 WHERE id = 1;
COMMIT;
```
---
## Chapter 11: Views - Storing and Reusing Queries
Views are one of SQL's most elegant features. At their core, views let you save a query and give it a name, so you can use it as if it were a table. But this simple concept unlocks powerful capabilities: simplifying complex queries, providing security abstraction, and (with materialized views) dramatically improving performance for expensive calculations.
### Understanding the View Abstraction
A view creates a layer of indirection between the physical schema and the queries that access it. This abstraction is both powerful and potentially problematic, depending on how you use it.
**The Benefits of Abstraction:**
- Schema changes can be hidden behind stable view interfaces
- Complex joins become simple table references
- Security policies can be enforced consistently
- Business logic is defined once and reused
**The Costs of Abstraction:**
- Debugging becomes harder when queries traverse multiple view layers
- Performance implications are hidden from developers
- Dependencies between views can become tangled
### When NOT to Use Views
Views solve specific problems. Using them inappropriately creates new problems:
**1. Performance-Critical Queries**: Regular views re-execute their underlying query every time. If performance matters and the underlying query is complex, consider a materialized view or redesigning the query.
**2. Views on Views on Views**: Each layer of view nesting adds complexity. When debugging a slow query, tracing through 4 levels of view definitions is painful. Keep view nesting to a maximum of 2-3 levels.
**3. Views That Hide Important Complexity**: A view that joins 10 tables and aggregates millions of rows looks simple (`SELECT * FROM dashboard_metrics`), but that simplicity is deceptive. Developers may not realize the cost of what they are asking for.
**4. Frequently Modified Definitions**: If the view logic changes weekly, it belongs in application code where changes can be version-controlled, tested, and deployed systematically.
**5. Views for One-Time Queries**: Creating a view for a query you will run once is unnecessary overhead. Just run the query.
### The Materialized View Decision
The choice between regular and materialized views involves a fundamental trade-off:
| Factor | Regular View | Materialized View |
|--------|--------------|-------------------|
| Data freshness | Always current | Stale until refreshed |
| Query speed | Depends on complexity | Consistently fast |
| Storage cost | None | Proportional to result size |
| Maintenance burden | None | Refresh scheduling required |
**Choose regular views** when data must be current and the underlying query is fast enough.
**Choose materialized views** when query speed matters more than absolute freshness, or when the same expensive aggregation is queried repeatedly.
### 11.1 Understanding Views
#### What Is a View?
A view is a named query stored in the database. When you query a view, the database substitutes your view reference with the stored query and executes the combined result. Think of it as a "saved query" or a "virtual table."
```sql
-- Instead of writing this complex query every time:
SELECT
c.name AS customer_name,
COUNT(o.id) AS order_count,
SUM(o.total) AS total_spent,
MAX(o.created_at) AS last_order_date
FROM customers c
LEFT JOIN orders o ON c.id = o.customer_id
WHERE c.status = 'active'
GROUP BY c.id, c.name;
-- You can create a view and use it simply:
SELECT * FROM customer_summary WHERE total_spent > 1000;
```
#### Why Use Views?
**1. Simplification**: Complex queries with multiple joins, aggregations, and conditions can be encapsulated in a view. Users of the view don't need to understand the complexity underneath.
**2. Security**: Views can expose only specific columns or rows to users, hiding sensitive data. A view showing `employee_name` and `department` can hide `salary` and `ssn` columns.
**3. Consistency**: When business logic changes (like how "active customer" is defined), you update the view definition once rather than hunting down every query.
**4. Abstraction**: Views can hide the physical schema. If you rename a table or split it into multiple tables, you can update the view to maintain the old interface.
#### Views vs. Materialized Views
Cognica supports two types of views:
| Feature | Regular View | Materialized View |
|---------|--------------|-------------------|
| Data Storage | None (query only) | Stores actual results |
| Query Speed | Runs underlying query each time | Reads pre-computed results |
| Data Freshness | Always current | May be stale until refreshed |
| Write Overhead | None | Must refresh to update |
| Best For | Simple transformations, security | Expensive aggregations, analytics |
**Regular View** - Each time you query a view, it re-executes the underlying query. Perfect for simple joins or when you always need current data.
**Materialized View** - The query results are stored in a backing collection. Queries read from this stored data, making them extremely fast. However, the data can become stale and must be periodically refreshed.
### 11.2 Creating Regular Views
#### Basic Syntax
```sql
CREATE VIEW view_name AS
SELECT column1, column2, ...
FROM table_name
WHERE condition;
```
#### Practical Examples
**Simple View - Filtering Rows**
```sql
-- Show only active products
CREATE VIEW active_products AS
SELECT id, name, price, category
FROM products
WHERE status = 'active' AND stock_quantity > 0;
-- Now use it like a table
SELECT * FROM active_products WHERE category = 'electronics';
```
**Joining Tables**
```sql
-- Combine order information with customer details
CREATE VIEW order_details AS
SELECT
o.id AS order_id,
o.created_at AS order_date,
c.name AS customer_name,
c.email AS customer_email,
o.total AS order_total,
o.status AS order_status
FROM orders o
JOIN customers c ON o.customer_id = c.id;
-- Query the view
SELECT * FROM order_details
WHERE order_date >= '2024-01-01'
ORDER BY order_total DESC;
```
**Aggregation View**
```sql
-- Monthly sales summary
CREATE VIEW monthly_sales AS
SELECT
DATE_TRUNC('month', created_at) AS month,
COUNT(*) AS order_count,
SUM(total) AS revenue,
AVG(total) AS average_order_value
FROM orders
WHERE status = 'completed'
GROUP BY DATE_TRUNC('month', created_at);
-- Use it for reporting
SELECT * FROM monthly_sales
WHERE month >= '2024-01-01'
ORDER BY month;
```
#### Column Aliases
You can rename columns in the view definition:
```sql
-- Method 1: In the SELECT clause (preferred for clarity)
CREATE VIEW customer_contacts AS
SELECT
id AS customer_id,
name AS full_name,
email AS contact_email,
phone AS contact_phone
FROM customers;
-- Method 2: In the view column list
CREATE VIEW customer_contacts (customer_id, full_name, contact_email, contact_phone) AS
SELECT id, name, email, phone
FROM customers;
```
The second method is useful when you want different names without changing the underlying query.
### 11.3 CREATE OR REPLACE VIEW
When you need to modify a view definition, use `CREATE OR REPLACE VIEW`:
```sql
-- Original view
CREATE VIEW high_value_customers AS
SELECT id, name, email
FROM customers
WHERE total_purchases > 1000;
-- Later, add more columns without dropping the view
CREATE OR REPLACE VIEW high_value_customers AS
SELECT id, name, email, phone, created_at
FROM customers
WHERE total_purchases > 1000;
```
**Why Use OR REPLACE Instead of DROP and CREATE?**
1. **Preserves Permissions**: Any grants on the view remain intact
2. **Atomic Operation**: No window where the view doesn't exist
3. **Preserves Dependencies**: Other views that reference this view continue to work
**Limitations**: You cannot change the view type (regular to materialized or vice versa) using OR REPLACE. You must drop and recreate in that case.
### 11.4 Dropping Views
#### Basic Syntax
```sql
DROP VIEW view_name;
```
#### IF EXISTS - Avoiding Errors
```sql
-- This fails if the view doesn't exist
DROP VIEW nonexistent_view; -- Error!
-- This succeeds silently if the view doesn't exist
DROP VIEW IF EXISTS nonexistent_view; -- OK, no error
```
#### CASCADE vs RESTRICT
When other views depend on the view you're dropping:
```sql
-- Create a base view
CREATE VIEW active_customers AS
SELECT * FROM customers WHERE status = 'active';
-- Create a dependent view
CREATE VIEW active_customer_emails AS
SELECT name, email FROM active_customers;
-- RESTRICT (default): Fails if dependencies exist
DROP VIEW active_customers;
-- Error: cannot drop view active_customers because other objects depend on it
-- CASCADE: Drops the view and all dependent views
DROP VIEW active_customers CASCADE;
-- Drops both active_customers AND active_customer_emails
```
**Best Practice**: Prefer RESTRICT (the default) and explicitly handle dependencies. CASCADE can accidentally drop more than intended.
### 11.5 Materialized Views
#### When to Use Materialized Views
Materialized views excel when:
1. **The underlying query is expensive**: Complex joins, aggregations over large tables
2. **Data doesn't need to be real-time**: Analytics dashboards, reports, summaries
3. **The same query runs frequently**: Multiple users or applications hitting the same aggregation
4. **Read performance is critical**: Sub-millisecond response time requirements
#### Creating Materialized Views
```sql
CREATE MATERIALIZED VIEW view_name AS
SELECT ...
FROM ...
[WITH DATA | WITH NO DATA];
```
**WITH DATA (Default)**: Executes the query immediately and stores results
```sql
-- Creates and immediately populates with data
CREATE MATERIALIZED VIEW daily_sales_summary AS
SELECT
DATE(created_at) AS sale_date,
COUNT(*) AS transaction_count,
SUM(amount) AS total_revenue,
AVG(amount) AS average_transaction
FROM transactions
WHERE status = 'completed'
GROUP BY DATE(created_at);
```
**WITH NO DATA**: Creates the view structure but doesn't populate it
```sql
-- Creates empty view (must refresh before querying)
CREATE MATERIALIZED VIEW monthly_report AS
SELECT
DATE_TRUNC('month', created_at) AS month,
category,
SUM(revenue) AS total_revenue
FROM sales
GROUP BY DATE_TRUNC('month', created_at), category
WITH NO DATA;
-- Attempting to query before refresh fails
SELECT * FROM monthly_report; -- Error: materialized view is not populated
-- Populate it with refresh
REFRESH MATERIALIZED VIEW monthly_report;
-- Now queries work
SELECT * FROM monthly_report WHERE month = '2024-01-01';
```
**When to Use WITH NO DATA**:
- Creating views during schema setup before data exists
- When you want to control exactly when the expensive query runs
- When initial population should happen during off-peak hours
#### IF NOT EXISTS
Prevent errors when the view might already exist:
```sql
-- Safe to run multiple times
CREATE MATERIALIZED VIEW IF NOT EXISTS user_statistics AS
SELECT
user_id,
COUNT(*) AS action_count,
MAX(created_at) AS last_activity
FROM user_actions
GROUP BY user_id;
```
### 11.6 Refreshing Materialized Views
The data in a materialized view is static until you refresh it. Refreshing re-executes the underlying query and updates the stored results.
#### Standard Refresh
```sql
REFRESH MATERIALIZED VIEW view_name;
```
**What Happens During Standard Refresh**:
1. The backing collection is truncated (all existing data removed)
2. The underlying query is executed
3. Results are inserted into the backing collection
4. Queries against the view block until refresh completes
```sql
-- Refresh to get latest data
REFRESH MATERIALIZED VIEW daily_sales_summary;
-- Now queries return updated results
SELECT * FROM daily_sales_summary
WHERE sale_date = CURRENT_DATE - INTERVAL '1 day';
```
#### CONCURRENTLY - Zero-Downtime Refresh
Standard refresh blocks queries during the update. For high-availability systems, use CONCURRENTLY:
```sql
REFRESH MATERIALIZED VIEW CONCURRENTLY view_name;
```
**How CONCURRENTLY Works**:
1. Creates a shadow collection (`_mview_shadow.{name}`)
2. Populates the shadow collection with fresh data
3. Atomically swaps the shadow collection with the live collection
4. Drops the old collection
**Benefits**:
- Queries continue to read from the old data during refresh
- No blocking or downtime
- Atomic swap ensures consistency
**Trade-offs**:
- Uses approximately 2x the storage temporarily
- Slightly longer total refresh time due to the swap
```sql
-- Production-safe refresh
REFRESH MATERIALIZED VIEW CONCURRENTLY dashboard_metrics;
```
#### Incremental Refresh
For simple materialized views (those defined with basic SELECT, JOIN, and WHERE clauses), Cognica supports incremental refresh. Instead of re-executing the entire query, incremental refresh tracks changes to underlying tables via per-table changelogs and applies only the deltas.
**How it works:**
1. When a base table is modified (INSERT, UPDATE, DELETE), changes are recorded in a changelog
2. During refresh, only changelog entries since the last refresh are processed
3. New rows are inserted, updated rows are replaced, deleted rows are removed
**Benefits:**
- Dramatically faster refresh for large views where underlying data changes slowly
- Reduced I/O and CPU usage compared to full refresh
**Limitations:**
- Only works for views with simple query patterns (no complex aggregations or DISTINCT)
- Falls back to full refresh if the view definition is too complex or if the changelog is too large
#### WITH DATA vs WITH NO DATA
Control whether refresh populates the view:
```sql
-- Normal refresh (default, equivalent to WITH DATA)
REFRESH MATERIALIZED VIEW monthly_summary;
REFRESH MATERIALIZED VIEW monthly_summary WITH DATA;
-- Clear the view without repopulating
REFRESH MATERIALIZED VIEW monthly_summary WITH NO DATA;
```
`WITH NO DATA` is useful when you need to invalidate cached data without immediately recalculating it, perhaps scheduling the actual refresh for later.
### 11.7 Dropping Materialized Views
```sql
-- Basic drop
DROP MATERIALIZED VIEW view_name;
-- Safe drop (no error if doesn't exist)
DROP MATERIALIZED VIEW IF EXISTS view_name;
-- Drop with dependencies
DROP MATERIALIZED VIEW view_name CASCADE;
```
**Important**: Dropping a materialized view also drops the backing collection that stores the data. This operation cannot be undone.
### 11.8 Querying View Metadata
Cognica stores view definitions in the `_sys.views` system collection. You can query this to see all views:
```sql
-- List all views
SELECT view_name, view_type, created_at
FROM _sys.views
ORDER BY created_at DESC;
-- Find materialized views that haven't been refreshed recently
SELECT view_name, last_refresh
FROM _sys.views
WHERE view_type = 'materialized'
AND (last_refresh IS NULL
OR last_refresh < EXTRACT(EPOCH FROM NOW() - INTERVAL '1 day') * 1000);
-- View definition for a specific view
SELECT query_text
FROM _sys.views
WHERE view_name = 'daily_sales_summary';
```
### 11.9 View Dependencies
Views can reference other views, creating dependency chains:
```sql
-- Base view
CREATE VIEW active_orders AS
SELECT * FROM orders WHERE status = 'active';
-- Dependent view
CREATE VIEW active_order_summary AS
SELECT customer_id, COUNT(*) AS order_count, SUM(total) AS total_value
FROM active_orders
GROUP BY customer_id;
-- Another dependent view
CREATE VIEW high_value_active_customers AS
SELECT * FROM active_order_summary WHERE total_value > 10000;
```
Cognica tracks these dependencies and enforces referential integrity:
- You cannot drop `active_orders` while `active_order_summary` depends on it (unless using CASCADE)
- CREATE OR REPLACE validates that the new definition is compatible with dependent views
- The `dependencies` field in `_sys.views` shows what each view references
### 11.10 Best Practices
#### When to Choose Regular Views vs. Materialized Views
| Scenario | Recommendation |
|----------|----------------|
| Simple column selection/renaming | Regular view |
| Row filtering (WHERE clause) | Regular view |
| Complex joins with good indexes | Regular view |
| Heavy aggregations (SUM, COUNT, AVG) | Materialized view |
| Analytics queries over millions of rows | Materialized view |
| Data must always be current | Regular view |
| Sub-second response time required | Materialized view |
#### Materialized View Refresh Strategies
**1. Scheduled Refresh** (Most Common)
```sql
-- Refresh every hour via cron job or scheduler
REFRESH MATERIALIZED VIEW CONCURRENTLY hourly_metrics;
```
**2. Event-Driven Refresh**
After significant data changes, trigger a refresh:
```sql
-- After bulk import completes
REFRESH MATERIALIZED VIEW inventory_summary;
```
**3. Time-Based Staleness Check**
Query `last_refresh` and refresh if too old:
```sql
-- In application code:
-- if (now - last_refresh) > threshold: refresh
```
#### Performance Considerations
**Creating Indexes on Materialized Views**
Materialized views store data in backing collections. You can create indexes on them for better query performance:
```sql
-- Create the materialized view
CREATE MATERIALIZED VIEW customer_metrics AS
SELECT customer_id, SUM(amount) AS total_spent
FROM orders
GROUP BY customer_id;
-- Create an index on the backing collection for faster lookups
CREATE INDEX idx_customer_metrics_customer
ON _mview.customer_metrics (customer_id);
```
**Monitoring Refresh Performance**
Keep track of how long refreshes take. If they grow too long:
- Consider more selective queries (add WHERE clauses)
- Use CONCURRENTLY to avoid blocking
- Schedule refreshes during low-traffic periods
**Memory Considerations**
Cognica streams materialized view population in batches of 10,000 rows to avoid memory issues. Even views over very large datasets can be created safely. However, the underlying query still executes fully, so extremely complex queries may take time.
---
## Chapter 12: Indexes - The Key to Query Performance
Indexes are the single most important tool for query performance. A well-designed index can make a query run thousands of times faster. A missing index can bring your application to its knees. This chapter explains how indexes work and how to use them effectively.
```mermaid
flowchart TB
subgraph Query["Query"]
SQL["SELECT * FROM users
WHERE email = 'alice@...'"]
end
subgraph Optimizer["Query Optimizer"]
ANALYZE["Analyze
WHERE clause"]
CHECK{"Index
Available?"}
COST["Estimate Cost:
Index vs Scan"]
CHOOSE{"Index
Cheaper?"}
end
subgraph Execution["Access Method"]
IDX["Index Lookup
(O(log n))"]
SCAN["Full Table Scan
(O(n))"]
end
SQL --> ANALYZE
ANALYZE --> CHECK
CHECK -->|"Yes"| COST
CHECK -->|"No"| SCAN
COST --> CHOOSE
CHOOSE -->|"Yes"| IDX
CHOOSE -->|"No"| SCAN
style IDX fill:#d4edda,color:#155724
style SCAN fill:#fff3cd,color:#856404
```
### The Cost-Benefit Analysis of Indexes
Indexes are not free. Before creating an index, understand what you are trading:
**Benefits:**
- Dramatically faster SELECT queries that use the indexed columns
- Faster ORDER BY when sorting on indexed columns
- Faster JOINs on indexed foreign key columns
- Faster GROUP BY on indexed columns
**Costs:**
- Slower INSERT/UPDATE/DELETE operations (every data change must also update indexes)
- Additional disk space (indexes can be as large as the table data itself)
- Memory usage (indexes compete for buffer cache space)
- Maintenance overhead (indexes need occasional rebuilding)
### When NOT to Create an Index
The knee-jerk reaction to slow queries is "add an index." But indexes can hurt performance:
**1. Small Tables**: For tables with fewer than a few hundred rows, a sequential scan is often faster than an index lookup. The overhead of traversing the index structure exceeds the benefit.
**2. High-Cardinality Writes**: If you INSERT millions of rows daily and rarely query them, indexes slow down your ingestion pipeline. Consider loading data without indexes and creating them afterward.
**3. Low-Selectivity Columns**: An index on a boolean column or a status column with three possible values rarely helps. If a query matches 50% of the table, reading the index and then fetching the rows costs more than scanning the table.
**4. Unused Indexes**: Indexes created "just in case" but never used by any query waste space and slow writes. Periodically audit your indexes.
**5. Redundant Indexes**: A composite index on `(A, B)` can serve queries on `A` alone. A separate index on just `A` is redundant.
The goal is not maximum indexes but the right indexes for your workload.
### 12.1 Understanding How Indexes Work
#### The Library Analogy
Imagine a library with millions of books. Without a catalog, finding a specific book means walking through every aisle and checking every shelf. That's what a database does without an index: it scans every row in the table.
Now imagine the library has a card catalog organized alphabetically by author. To find books by "Hemingway," you flip to the H section, find "Hemingway," and get a list of shelf locations. That's exactly what a database index does: it maintains a sorted structure that points to the actual data.
#### How Cognica Indexes Work
Cognica uses LSM-tree (Log-Structured Merge-tree) storage through a customized RocksDB fork. This affects how indexes behave:
**Write Performance**: Indexes add overhead to writes because every INSERT, UPDATE, or DELETE must update both the table data and all relevant indexes. However, LSM-trees are optimized for writes, so this overhead is relatively small.
**Read Performance**: Indexes dramatically improve read performance by avoiding full table scans. The benefit is even more pronounced in LSM-trees because reading scattered data without an index requires checking multiple levels of the storage tree.
**Space Trade-off**: Indexes consume disk space. A table with many indexes uses significantly more storage than the same table without indexes.
### 12.2 Types of Indexes
#### B-tree Indexes (The Default)
B-tree indexes are the default and most versatile index type. They work well for:
- Equality comparisons (`=`)
- Range queries (`<`, `>`, `<=`, `>=`, `BETWEEN`)
- Prefix matching (`LIKE 'abc%'`)
- Sorting (`ORDER BY`)
```sql
-- Create a B-tree index (default type)
CREATE INDEX idx_users_email ON users (email);
-- These queries can use the index:
SELECT * FROM users WHERE email = 'alice@example.com';
SELECT * FROM users WHERE email > 'a' AND email < 'b';
SELECT * FROM users WHERE email LIKE 'alice%';
SELECT * FROM users ORDER BY email;
```
#### GIN Indexes (Generalized Inverted Index)
GIN indexes are designed for values that contain multiple elements, such as arrays and JSONB:
```sql
-- Index for array containment queries
CREATE INDEX idx_products_tags ON products USING GIN (tags);
-- Now these queries are fast:
SELECT * FROM products WHERE tags @> ARRAY['electronics'];
SELECT * FROM products WHERE 'sale' = ANY(tags);
-- Index for JSONB queries
CREATE INDEX idx_events_data ON events USING GIN (data);
-- Fast JSONB queries:
SELECT * FROM events WHERE data @> '{"type": "purchase"}';
SELECT * FROM events WHERE data ? 'user_id';
```
**When to Use GIN**:
- Array columns with containment queries (`@>`, `&&`)
- JSONB columns with containment queries (`@>`, `?`, `?|`, `?&`)
- Full-text search (covered in Chapter 11)
**Trade-offs**:
- GIN indexes are slower to build and update than B-tree indexes
- They're much faster for the specific operations they support
- They can be larger than B-tree indexes
#### Hash Indexes
Hash indexes are optimized for equality comparisons only:
```sql
CREATE INDEX idx_sessions_token ON sessions USING HASH (token);
-- Fast:
SELECT * FROM sessions WHERE token = 'abc123xyz';
-- CANNOT use the hash index (use B-tree instead):
SELECT * FROM sessions WHERE token > 'abc';
SELECT * FROM sessions ORDER BY token;
```
**When to Use Hash**:
- Columns used only for equality lookups
- Very long text values where B-tree overhead is high
- Generally, B-tree is more flexible and preferred
### 12.3 Composite Indexes (Multi-Column)
A composite index covers multiple columns and is essential for queries that filter on multiple columns.
```sql
CREATE INDEX idx_orders_customer_date ON orders (customer_id, order_date);
```
#### Column Order Matters Enormously
The order of columns in a composite index determines which queries can use it:
```sql
-- This index: (customer_id, order_date)
-- CAN use the full index:
SELECT * FROM orders WHERE customer_id = 1 AND order_date = '2024-12-25';
SELECT * FROM orders WHERE customer_id = 1 AND order_date > '2024-01-01';
-- CAN use the index (first column only):
SELECT * FROM orders WHERE customer_id = 1;
-- CANNOT use the index efficiently:
SELECT * FROM orders WHERE order_date = '2024-12-25';
-- (order_date is the second column, so we'd need to scan everything)
```
**The Leftmost Prefix Rule**: A composite index can be used for queries that filter on a leftmost prefix of the index columns. If your index is `(A, B, C)`:
- Queries on `A` alone: Yes
- Queries on `A, B`: Yes
- Queries on `A, B, C`: Yes
- Queries on `B` alone: No (not a leftmost prefix)
- Queries on `A, C` (skipping B): Partial (uses A only)
**Practical Guidance for Column Order**:
1. **Put equality columns first**: Columns used with `=` should come before range columns
2. **Most selective first**: Columns that filter out more rows should generally come first
3. **Consider query patterns**: Design indexes for your actual queries
```sql
-- Query: WHERE status = 'active' AND created_at > '2024-01-01'
-- Good: (status, created_at) - equality column first
-- Bad: (created_at, status) - range column first limits use of second column
-- Query: WHERE user_id = ? ORDER BY created_at DESC LIMIT 10
-- Good: (user_id, created_at DESC) - covers both filter and sort
```
### 12.4 Partial Indexes (Filtered Indexes)
A partial index only indexes rows matching a condition:
```sql
-- Index only active users (ignores the millions of inactive ones)
CREATE INDEX idx_active_users_email ON users (email)
WHERE status = 'active';
-- Index only recent orders
CREATE INDEX idx_recent_orders ON orders (customer_id, total)
WHERE order_date > '2024-01-01';
-- Index only non-null values
CREATE INDEX idx_users_phone ON users (phone)
WHERE phone IS NOT NULL;
```
**Benefits**:
- Smaller index size
- Faster index maintenance
- Better cache utilization
**When to Use**:
- When queries always include the same filter condition
- When most rows don't match the common query pattern
- To index only the "hot" subset of data
### 12.5 Expression Indexes (Computed Indexes)
Index the result of an expression:
```sql
-- Index lowercase email for case-insensitive searches
CREATE INDEX idx_users_email_lower ON users (LOWER(email));
-- Now this query uses the index:
SELECT * FROM users WHERE LOWER(email) = 'alice@example.com';
-- Index year extracted from timestamp
CREATE INDEX idx_orders_year ON orders (EXTRACT(YEAR FROM order_date));
-- Fast yearly queries:
SELECT * FROM orders WHERE EXTRACT(YEAR FROM order_date) = 2024;
-- Index JSON field
CREATE INDEX idx_events_user_id ON events ((data->>'user_id'));
-- Fast JSON field queries:
SELECT * FROM events WHERE data->>'user_id' = '123';
```
**Important**: The query must use the exact same expression as the index definition.
### 12.6 Index Design Best Practices
#### When to Create Indexes
**Do Create Indexes For**:
- Primary keys (automatic in Cognica)
- Foreign key columns (frequently joined)
- Columns used in WHERE clauses
- Columns used in ORDER BY
- Columns used in GROUP BY
**Be Cautious With**:
- Very small tables (full scan might be faster)
- Columns with very few distinct values (like boolean or status with 3 values)
- Tables with heavy write load and infrequent reads
#### Index Naming Convention
Use a consistent naming pattern:
```sql
-- Pattern: idx__
CREATE INDEX idx_users_email ON users (email);
CREATE INDEX idx_orders_customer_date ON orders (customer_id, order_date);
-- For partial indexes, include the condition hint
CREATE INDEX idx_users_email_active ON users (email) WHERE status = 'active';
```
#### Managing Indexes
```sql
-- List all indexes on a table
SELECT indexname, indexdef
FROM pg_indexes
WHERE tablename = 'users';
-- Drop an index
DROP INDEX idx_users_email;
-- Drop only if exists
DROP INDEX IF EXISTS idx_users_email;
-- Rebuild an index (for maintenance)
REINDEX INDEX idx_users_email;
```
#### Identifying Missing Indexes
If a query is slow, check if it's doing a sequential scan when it should be using an index:
```sql
-- Use EXPLAIN to see query plan
EXPLAIN SELECT * FROM users WHERE email = 'alice@example.com';
-- If you see "Seq Scan" for a large table, you might need an index
-- If you see "Index Scan using idx_users_email", the index is being used
```
---
## Chapter 13: Query Analysis and Optimization
Understanding how Cognica executes your queries is essential for writing efficient SQL. This chapter covers the tools and techniques for analyzing query performance and optimizing slow queries.
### Why Query Analysis Matters
A query that runs in 10 milliseconds on 1,000 rows might take 10 seconds on 1,000,000 rows. Understanding query execution helps you:
- Identify bottlenecks before they become problems
- Choose the right indexes for your workload
- Write queries that scale with your data
- Debug performance regressions
### Cognica's Query Execution Model
Cognica processes SQL queries through several stages:
1. **Parsing**: SQL text is parsed into an Abstract Syntax Tree (AST)
2. **Analysis**: The AST is validated against the schema
3. **Planning**: The query planner generates an execution plan
4. **Optimization**: The optimizer chooses the most efficient plan
5. **Execution**: The plan is executed against the storage engine
The query planner is LSM-tree aware. Unlike traditional B-tree databases, Cognica's planner accounts for:
- Multiple SST file levels
- Memtable presence
- Compaction state
- Bloom filter availability
### 13.1 Understanding EXPLAIN
The `EXPLAIN` command shows how Cognica will execute a query without actually running it:
```sql
EXPLAIN SELECT * FROM users WHERE email = 'alice@example.com';
```
**Example Output**:
```mermaid
flowchart BT
A["Scan(users)"] --> B["Filter(email = 'alice@example.com')"]
```
This shows a sequential scan followed by a filter. For a large table, this is inefficient. Adding an index changes the plan:
```sql
CREATE INDEX idx_users_email ON users (email);
EXPLAIN SELECT * FROM users WHERE email = 'alice@example.com';
```
**Optimized Output**:
```mermaid
flowchart BT
A["IndexScan(users, idx_users_email)"] --> B["Filter(email = 'alice@example.com')"]
```
### 13.2 EXPLAIN Output Formats
Cognica supports multiple output formats and verbosity levels for EXPLAIN:
**Text Format (Default)**:
```sql
EXPLAIN SELECT * FROM orders WHERE customer_id = 123;
```
Human-readable tree structure showing the execution plan hierarchy.
**JSON Format**:
```sql
EXPLAIN (FORMAT JSON) SELECT * FROM orders WHERE customer_id = 123;
```
Structured output for programmatic analysis:
```json
{
"type": "IndexScan",
"table": "orders",
"index": "idx_orders_customer",
"estimated_rows": 50,
"estimated_cost": 12.5,
"filter": {
"column": "customer_id",
"operator": "=",
"value": 123
}
}
```
**EXPLAIN ANALYZE - Actual Execution Statistics**:
EXPLAIN ANALYZE executes the query and reports actual runtime statistics alongside the plan:
```sql
EXPLAIN ANALYZE SELECT * FROM orders WHERE customer_id = 123;
```
This shows:
- **Actual row counts** vs. estimated row counts
- **Actual execution time** per operator
- **Total execution time** for the query
Use EXPLAIN ANALYZE when estimated row counts are inaccurate, or when you need to measure real performance. Be cautious with DML statements — EXPLAIN ANALYZE will actually execute INSERT, UPDATE, and DELETE. Wrap them in a transaction and roll back:
```sql
BEGIN;
EXPLAIN ANALYZE DELETE FROM logs WHERE created_at < '2024-01-01';
ROLLBACK; -- Undo the actual deletion
```
**EXPLAIN VERBOSE - Detailed Output**:
```sql
EXPLAIN VERBOSE SELECT name, email FROM users WHERE active = true;
```
VERBOSE adds additional detail to the plan output:
- Column lists at each plan node
- Schema-qualified table names
- Output column mappings
- Internal operator parameters
**Combining Options**:
```sql
-- Analyze with JSON format for programmatic use
EXPLAIN (ANALYZE, FORMAT JSON) SELECT * FROM products WHERE price > 100;
-- Verbose analysis
EXPLAIN (ANALYZE, VERBOSE) SELECT u.name, COUNT(o.id)
FROM users u JOIN orders o ON u.id = o.user_id
GROUP BY u.name;
```
| Option | Effect | Executes Query? |
|--------|--------|-----------------|
| `EXPLAIN` | Shows estimated plan | No |
| `EXPLAIN ANALYZE` | Shows actual execution stats | Yes |
| `EXPLAIN VERBOSE` | Shows detailed plan info | No |
| `EXPLAIN (FORMAT JSON)` | JSON structured output | No |
| `EXPLAIN (ANALYZE, VERBOSE)` | Full detail with execution | Yes |
### 13.3 Reading Query Plans
#### Common Plan Operators
| Operator | Description | Performance Implication |
|----------|-------------|------------------------|
| `Scan` | Sequential table scan | Slow for large tables |
| `IndexScan` | Index-based lookup | Fast for selective queries |
| `Filter` | Row filtering | Applied after scan |
| `Sort` | Row ordering | Memory-intensive |
| `Limit` | Result truncation | Can short-circuit execution |
| `Join` | Table combination | Performance varies by type |
| `Aggregate` | Grouping/aggregation | May require sorting |
#### Plan Hierarchy
Plans are trees where child operators feed into parent operators:
```mermaid
flowchart BT
A["Scan(orders)"] --> B["Filter(status = 'active')"]
B --> C["Sort(created_at DESC)"]
C --> D["Limit(10)"]
```
Read bottom-up: scan orders, filter by status, sort by date, take top 10.
#### Estimated Costs
JSON format includes cost estimates:
- **estimated_rows**: Expected result count
- **estimated_cost**: Relative execution cost (higher = slower)
These are estimates based on table statistics. Actual performance may vary.
### 13.4 Index Selection
The query planner automatically selects indexes based on:
**1. Selectivity**: How many rows the index will filter out
```sql
-- High selectivity (good for index): email is unique
WHERE email = 'alice@example.com'
-- Low selectivity (may skip index): status has few distinct values
WHERE status = 'active'
```
**2. Index Coverage**: Whether the index contains all needed columns
```sql
-- Covered by index on (customer_id, order_date)
SELECT customer_id, order_date FROM orders WHERE customer_id = 123;
-- Requires table lookup even with index
SELECT * FROM orders WHERE customer_id = 123;
```
**3. Sort Optimization**: Index order matching ORDER BY
```sql
-- Index on (customer_id, order_date) can satisfy both filter and sort
SELECT * FROM orders
WHERE customer_id = 123
ORDER BY order_date DESC;
```
#### When Indexes Are Not Used
Indexes may be skipped when:
- Table is small (full scan is faster)
- Query returns most rows (index overhead not worth it)
- Expression prevents index use
- Index statistics are stale
**Common Index-Preventing Patterns**:
```sql
-- Function on indexed column (prevents index use)
WHERE LOWER(email) = 'alice@example.com'
-- Solution: Create expression index
CREATE INDEX idx_users_email_lower ON users (LOWER(email));
-- OR conditions on different columns
WHERE email = 'x' OR name = 'y'
-- Solution: Consider separate queries with UNION
-- LIKE with leading wildcard
WHERE name LIKE '%smith'
-- Solution: Full-text search or trigram index
```
#### Index Intersection
When a query has multiple AND conditions on different indexed columns, Cognica can use **index intersection** to combine results from multiple indexes without requiring a composite index:
```sql
-- Given separate indexes on status and region:
CREATE INDEX idx_orders_status ON orders (status);
CREATE INDEX idx_orders_region ON orders (region);
-- This query can intersect both indexes
SELECT * FROM orders
WHERE status = 'active' AND region = 'US';
-- Scans both indexes independently, then intersects the matching row sets
```
Index intersection is automatically considered by the query optimizer when:
- Multiple single-column indexes cover the WHERE clause predicates
- No suitable composite index exists
- The intersection of index scans is cheaper than a full table scan
This optimization reduces the need to create composite indexes for every possible query pattern, keeping index maintenance overhead lower.
### 13.5 Common Performance Issues
#### Issue 1: Missing Index
**Symptom**: Slow queries with `Scan` instead of `IndexScan`
**Diagnosis**:
```sql
EXPLAIN SELECT * FROM orders WHERE customer_id = 123;
-- Shows: Scan(orders) + Filter
```
**Solution**:
```sql
CREATE INDEX idx_orders_customer ON orders (customer_id);
```
#### Issue 2: Inefficient Joins
**Symptom**: Slow joins, especially with large tables
**Diagnosis**: Check join order and method in EXPLAIN output
**Solutions**:
- Ensure join columns are indexed
- Filter before joining when possible
- Consider denormalization for frequently-joined data
```sql
-- Inefficient: Join then filter
SELECT o.*, c.name
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.status = 'pending';
-- Better: Filter orders first (if status is selective)
SELECT o.*, c.name
FROM (SELECT * FROM orders WHERE status = 'pending') o
JOIN customers c ON o.customer_id = c.id;
```
#### Issue 3: Sorting Large Result Sets
**Symptom**: Slow queries with `Sort` operator
**Solutions**:
- Add index matching ORDER BY
- Limit results before sorting
```sql
-- Slow: Sort all orders, then take 10
SELECT * FROM orders ORDER BY created_at DESC LIMIT 10;
-- If no index on created_at, create one:
CREATE INDEX idx_orders_created ON orders (created_at DESC);
```
#### Issue 4: N+1 Query Pattern
**Symptom**: Application makes many sequential queries
**Solution**: Use JOINs or batch queries
```sql
-- N+1 pattern (bad): One query per order
SELECT * FROM orders WHERE customer_id = 1;
SELECT * FROM orders WHERE customer_id = 2;
-- ... repeated for each customer
-- Batch query (good)
SELECT * FROM orders WHERE customer_id IN (1, 2, 3, ...);
```
### 13.6 Optimization Best Practices
#### 1. Index Strategically
```sql
-- Index columns used in:
-- - WHERE clauses (equality before range)
-- - JOIN conditions
-- - ORDER BY clauses
-- Good composite index for this query pattern:
SELECT * FROM orders
WHERE customer_id = 123
AND status = 'active'
ORDER BY created_at DESC;
CREATE INDEX idx_orders_composite
ON orders (customer_id, status, created_at DESC);
```
#### 2. Limit Early
```sql
-- Apply LIMIT as early as possible
SELECT * FROM large_table
WHERE condition = true
ORDER BY score DESC
LIMIT 10; -- Planner can use Top-N sort
```
#### 3. Avoid SELECT *
```sql
-- Slower: Fetches all columns
SELECT * FROM users WHERE id = 1;
-- Faster: Fetches only needed columns
SELECT name, email FROM users WHERE id = 1;
```
#### 4. Use Appropriate Data Types
```sql
-- Slower: String comparison
WHERE order_id = '12345'
-- Faster: Integer comparison
WHERE order_id = 12345
```
#### 5. Partition Large Tables
For very large tables, consider partitioning by date or other natural boundaries:
```sql
-- Queries on recent data only scan recent partitions
SELECT * FROM events
WHERE event_date >= '2025-01-01'
AND event_type = 'purchase';
```
#### 6. Monitor and Iterate
Query performance changes as data grows. Regularly:
- Review slow query logs
- Re-analyze query plans after data changes
- Update statistics after bulk operations
### 13.7 LSM-Tree Specific Considerations
Cognica uses an LSM-tree storage engine based on a customized RocksDB fork, which has different performance characteristics than traditional B-tree databases:
**Write Performance**: LSM-trees excel at writes. Sequential writes are buffered in memory and flushed to disk in batches.
**Read Performance**: Point lookups may check multiple levels (memtable + SST files). Bloom filters help skip irrelevant files.
**Compaction Impact**: Background compaction can temporarily affect query latency. Schedule bulk operations during low-traffic periods.
**Index Considerations**:
- LSM indexes are append-only, making inserts very fast
- Range queries may span multiple SST files
- Bloom filters work best for equality lookups
---
## Chapter 14: Full-Text Search - Finding Words in Documents
Full-text search (FTS) lets you search for words and phrases within text content. Unlike LIKE or regular expressions, FTS understands language: it handles word boundaries, stemming (finding "running" when you search "run"), and relevance ranking.
### When Full-Text Search Is the Right Tool
FTS excels at finding documents that contain specific words or phrases, especially when you need:
- Relevance ranking (most relevant results first)
- Linguistic processing (stemming, stop words)
- Fast searches over large text collections
However, FTS is not always the answer. Consider these alternatives:
| Use Case | Best Tool |
|----------|-----------|
| Exact string match | `= 'value'` with B-tree index |
| Prefix matching | `LIKE 'prefix%'` with B-tree index |
| Finding words in text | Full-Text Search |
| Fuzzy matching (typos) | pg_trgm extension with GiST/GIN index |
| Regular expressions | `~` operator (slow, use sparingly) |
### Limitations to Understand
FTS is powerful but not magic. Know its boundaries:
**1. Language Dependency**: FTS configurations are language-specific. A document written in French will not be properly stemmed using the English configuration. If your content spans multiple languages, you must handle this explicitly.
**2. No Substring Matching**: FTS finds words, not arbitrary substrings. Searching for "lap" will not find "laptop" because "lap" and "laptop" are different words. For substring matching, consider the pg_trgm extension.
**3. Precision vs. Recall Trade-offs**: Stemming increases recall (finding more results) but may decrease precision (finding irrelevant results). Searching for "run" will match "running", "runs", and "ran" - which is usually helpful but occasionally returns false positives.
**4. Index Maintenance**: FTS indexes (GIN) are larger and slower to update than B-tree indexes. For write-heavy workloads with infrequent searches, evaluate whether the index overhead is justified.
**5. No Semantic Understanding**: FTS matches words, not concepts. Searching for "automobile" will not find documents that only use "car" unless you explicitly handle synonyms.
With these considerations in mind, let us explore how FTS works in practice.
### 14.1 Understanding Full-Text Search
#### Why Not Just Use LIKE?
Consider searching for products containing "laptop":
```sql
-- Using LIKE (problematic)
SELECT * FROM products WHERE description LIKE '%laptop%';
```
Problems with LIKE:
1. **No index support**: Must scan every row and every character
2. **No word boundaries**: Matches "laptops", but also "mylaptop" inside another word
3. **Case sensitive**: Won't find "Laptop" unless you use ILIKE
4. **No relevance**: All matches are equal - no ranking
Full-text search solves all of these:
```sql
-- Using full-text search (fast and smart)
SELECT * FROM products
WHERE to_tsvector('english', description) @@ to_tsquery('english', 'laptop');
```
This:
1. Uses inverted indexes for fast lookup
2. Respects word boundaries
3. Handles case and stemming automatically
4. Can rank results by relevance
### 14.2 Text Search Data Types
#### tsvector: The Document Representation
A `tsvector` is a sorted list of normalized words (lexemes) with position information:
```sql
SELECT to_tsvector('english', 'The quick brown foxes jumped over lazy dogs');
-- Result: 'brown':3 'dog':9 'fox':4 'jump':5 'lazi':8 'quick':2
-- Notice:
-- 'The' is removed (stop word)
-- 'foxes' becomes 'fox' (stemming)
-- 'jumped' becomes 'jump' (stemming)
-- 'lazy' becomes 'lazi' (stemming)
-- Numbers indicate word positions
```
#### tsquery: The Search Query
A `tsquery` is a search pattern:
```sql
SELECT to_tsquery('english', 'quick & brown'); -- Both words
SELECT to_tsquery('english', 'quick | brown'); -- Either word
SELECT to_tsquery('english', '!quick'); -- NOT quick
SELECT to_tsquery('english', 'quick <-> brown'); -- Adjacent (phrase)
SELECT to_tsquery('english', 'quick <2> brown'); -- Within 2 words
```
### 14.3 The @@ Match Operator
The `@@` operator tests if a tsvector matches a tsquery:
```sql
-- Basic match
SELECT to_tsvector('english', 'The quick brown fox')
@@ to_tsquery('english', 'quick & fox');
-- Result: true
-- Using with table
SELECT title, description
FROM products
WHERE to_tsvector('english', description) @@ to_tsquery('english', 'laptop & fast');
```
#### Implicit Type Coercion
Cognica supports implicit type coercion for the `@@` operator, allowing you to use raw text columns directly without explicit `to_tsvector()` or `to_tsquery()` calls:
```sql
-- Implicit conversion: raw text column @@ search term
-- Cognica automatically converts the column to tsvector and the search term to tsquery
SELECT title, content
FROM articles
WHERE content @@ 'database';
-- Equivalent to the explicit form:
SELECT title, content
FROM articles
WHERE to_tsvector('english', content) @@ to_tsquery('english', 'database');
```
**Field-Specific Queries**:
You can also search specific fields using the `field:term` syntax:
```sql
-- Search for 'database' specifically in the 'body' field
SELECT title, body
FROM articles
WHERE title @@ 'body:database';
-- Phrase search in a specific field
SELECT title, content
FROM articles
WHERE title @@ 'content:"machine learning"';
```
**How Implicit Coercion Works**:
| Left Operand | Right Operand | Behavior |
|--------------|---------------|----------|
| Raw text | Plain term | Auto-tokenizes both using default analyzer |
| Raw text | tsquery format (`'word' & 'word2'`) | Auto-tokenizes left, parses right as tsquery |
| tsvector format | tsquery format | Direct matching (no conversion) |
This feature enables simple, PostgreSQL-like FTS queries without requiring an FTS index, using in-memory matching as a fallback. For production workloads with large datasets, create an FTS index for optimal performance.
### 14.4 Creating FTS Indexes
For fast full-text search, create a GIN index on the tsvector:
```sql
-- Option 1: Index on expression
CREATE INDEX idx_products_fts ON products
USING GIN (to_tsvector('english', description));
-- Option 2: Add a tsvector column and index it
ALTER TABLE products ADD COLUMN search_vector tsvector;
UPDATE products SET search_vector =
to_tsvector('english', COALESCE(name, '') || ' ' || COALESCE(description, ''));
CREATE INDEX idx_products_search ON products USING GIN (search_vector);
-- Keep the column updated with a trigger (if using Option 2)
```
**Which Option to Choose**:
- **Expression index**: Simpler, no schema change, but recalculates tsvector each search
- **Stored column**: Faster queries, but requires maintaining the column
### 14.5 Search Query Syntax
#### Simple Searches
```sql
-- Single word
SELECT * FROM products
WHERE to_tsvector('english', description) @@ to_tsquery('english', 'laptop');
-- Simpler syntax for user input
SELECT * FROM products
WHERE to_tsvector('english', description) @@ plainto_tsquery('english', 'fast laptop');
-- plainto_tsquery converts "fast laptop" to "fast & laptop" automatically
-- Phrase search
SELECT * FROM products
WHERE to_tsvector('english', description) @@ phraseto_tsquery('english', 'solid state drive');
-- Searches for the exact phrase
-- Web-search syntax (supports quotes, +, -, OR)
SELECT * FROM products
WHERE to_tsvector('english', description)
@@ websearch_to_tsquery('english', '"solid state" laptop -refurbished');
-- Converts: "solid state" => phrase, laptop => AND, -refurbished => NOT
```
The four `tsquery` constructors serve different purposes:
| Function | Input | Converts To | Best For |
|----------|-------|-------------|----------|
| `to_tsquery` | `'fast & laptop'` | AND/OR/NOT/phrase operators | Programmatic query building |
| `plainto_tsquery` | `'fast laptop'` | All words ANDed | Simple keyword search |
| `phraseto_tsquery` | `'solid state drive'` | Adjacent word matching | Exact phrase search |
| `websearch_to_tsquery` | `'"SSD" laptop -old'` | Quotes, +, -, OR | User-facing search boxes |
#### Complex Queries
```sql
-- AND, OR, NOT combinations
SELECT * FROM products
WHERE to_tsvector('english', description)
@@ to_tsquery('english', '(laptop | notebook) & !refurbished & (fast | quick)');
-- Weighted search across multiple fields
SELECT * FROM products
WHERE setweight(to_tsvector('english', name), 'A') ||
setweight(to_tsvector('english', description), 'B')
@@ to_tsquery('english', 'laptop');
-- Matches in 'name' are weighted higher than matches in 'description'
```
### 14.6 Ranking Search Results
FTS can rank results by relevance using the BM25 algorithm:
```sql
SELECT
title,
description,
ts_rank(to_tsvector('english', description), query) AS rank
FROM products, to_tsquery('english', 'laptop & lightweight') AS query
WHERE to_tsvector('english', description) @@ query
ORDER BY rank DESC
LIMIT 20;
```
**Understanding BM25**:
- BM25 (Best Matching 25) is a probabilistic ranking algorithm
- It considers term frequency (how often the word appears in the document)
- It considers inverse document frequency (rare words are weighted higher)
- It normalizes for document length (longer documents don't get unfair advantage)
```sql
-- More control over ranking
SELECT
title,
ts_rank_cd( -- _cd variant considers proximity of terms
to_tsvector('english', description),
to_tsquery('english', 'laptop & fast'),
32 -- Normalization flags
) AS rank
FROM products
WHERE to_tsvector('english', description) @@ to_tsquery('english', 'laptop & fast')
ORDER BY rank DESC;
```
### 14.7 Highlighting Search Results
Show users where their search terms matched:
```sql
SELECT
title,
ts_headline('english', description,
to_tsquery('english', 'laptop'),
'StartSel=, StopSel=, MaxWords=35, MinWords=15, MaxFragments=3'
) AS highlighted_excerpt
FROM products
WHERE to_tsvector('english', description) @@ to_tsquery('english', 'laptop');
-- Result might show:
-- "The new laptop features a stunning display and..."
```
### 14.8 Full-Text Search Best Practices
**Choosing a Language Configuration**:
```sql
-- Available configurations
SELECT cfgname FROM pg_ts_config;
-- english, simple, spanish, german, etc.
-- 'simple' doesn't stem - use for exact matches
-- Language-specific configs handle stemming and stop words
```
**Handling Multiple Languages**:
```sql
-- Store the language with the content
ALTER TABLE articles ADD COLUMN lang TEXT DEFAULT 'english';
-- Search with appropriate configuration
SELECT * FROM articles
WHERE to_tsvector(lang::regconfig, content) @@ to_tsquery(lang::regconfig, 'search term');
```
**Combining FTS with Other Filters**:
```sql
-- FTS + additional filters
SELECT * FROM products
WHERE category = 'Electronics'
AND price < 1000
AND to_tsvector('english', description) @@ to_tsquery('english', 'laptop')
ORDER BY ts_rank(to_tsvector('english', description),
to_tsquery('english', 'laptop')) DESC;
```
### 14.9 Vector Search Integration
Cognica integrates vector search directly into the Full-Text Search engine. For comprehensive coverage of vector search capabilities including HNSW indexes, hybrid search, and RAG patterns, see [Chapter 12: Vector Search](#chapter-12-vector-search---semantic-similarity-at-scale).
**Quick Example**:
```sql
-- Hybrid search: keyword + semantic similarity
SELECT id, title, _meta.score
FROM articles
WHERE _all @@ 'content:machine learning AND embedding:[[0.1, 0.2, ...]]'
ORDER BY _meta.score DESC
LIMIT 10;
```
---
## Chapter 15: Vector Search - Semantic Similarity at Scale
Vector search enables semantic similarity queries using dense vector embeddings. Unlike keyword search that matches exact terms, vector search finds conceptually similar content by comparing mathematical representations of meaning. This is the foundation for modern AI applications including Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems.
### Why Vector Search Matters
Traditional keyword search fails when:
- Users search "automobile" but documents contain "car"
- Users search "how to fix a slow computer" but documents describe "performance optimization"
- Users want conceptually similar items, not exact matches
Vector search solves these problems by converting text to high-dimensional vectors (embeddings) where semantically similar content clusters together. A search for "automobile" will find documents about "car", "vehicle", and "transportation" because their vector representations are mathematically close.
### Cognica's Vector Search Architecture
Cognica's approach to vector search differs fundamentally from standalone vector databases. Rather than treating vectors as a separate system, Cognica integrates vector search into its unified query engine alongside full-text search, SQL filtering, and transactions.
This integration provides three unique capabilities:
**1. Unified Boolean Operations**: Vector queries participate in the same boolean framework as text queries. Combine text and vector conditions with AND, OR, and NOT:
```sql
-- Documents must match "machine learning" AND be similar to query vector
SELECT id, title, _meta.score
FROM articles
WHERE _all @@ 'content:machine learning AND embedding:[[0.1, 0.2, 0.3, ...]]'
ORDER BY _meta.score DESC
LIMIT 10;
```
**2. Native Vector Exclusion (NOT)**: Most vector databases cannot efficiently exclude similar items because approximate nearest neighbor algorithms optimize for finding matches, not exclusions. Cognica implements set operations at the index level:
```sql
-- Find AI articles but exclude science fiction topics
SELECT id, title
FROM articles
WHERE _all @@ 'topic_embedding:[[0.1, ...]] AND NOT topic_embedding:[[0.8, ...]]'
ORDER BY _meta.score DESC;
```
**3. Two-Dimensional Vector Storage**: Long documents require chunking, with each chunk having its own embedding. Instead of storing chunks as separate rows (requiring joins) or separate collections (fragmenting data), Cognica stores multiple vectors per document:
```sql
-- Search all chunks, return parent documents
SELECT doc_id, title, _meta.score
FROM legal_documents
WHERE _all @@ 'chunk_embeddings:[[0.1, 0.2, ...]]'
ORDER BY _meta.score DESC;
```
### 15.1 Understanding Vector Embeddings
Vector embeddings are numerical representations of content in high-dimensional space. An embedding model (like OpenAI's ada-002 or open-source alternatives like Sentence-BERT) converts text into a fixed-size array of floating-point numbers.
**How Embeddings Work**:
| Text | Embedding (simplified 4D example) |
|------|-----------------------------------|
| "king" | [0.8, 0.2, 0.9, 0.1] |
| "queen" | [0.75, 0.25, 0.85, 0.15] |
| "apple" | [0.1, 0.9, 0.2, 0.3] |
Similar concepts have similar vectors. The distance between "king" and "queen" is small; the distance to "apple" is large.
**Common Embedding Dimensions**:
| Model | Dimensions | Notes |
|-------|------------|-------|
| all-MiniLM-L6-v2 | 384 | Fast, lightweight |
| BERT-base | 768 | Good balance |
| OpenAI text-embedding-3-small | 1536 | High quality |
| OpenAI text-embedding-3-large | 3072 | Highest quality |
### 15.2 Vector Query Syntax
Cognica uses tensor syntax within the `@@` operator for vector searches:
```sql
-- Basic vector similarity search
SELECT * FROM products
WHERE embedding @@ '[[0.1, 0.2, 0.3, 0.4, 0.5]]'
ORDER BY _meta.score DESC
LIMIT 20;
-- Field-specific vector search
SELECT * FROM articles
WHERE _all @@ 'content_embedding:[[0.1, 0.2, ...]]'
ORDER BY _meta.score DESC;
```
**The `_meta.score` Column**:
Every vector search returns a relevance score in the virtual `_meta.score` column. Higher scores indicate greater similarity. Always ORDER BY this column for meaningful results.
**Boolean Operations with Vectors**:
```sql
-- AND: Must be similar to vector AND match text
SELECT * FROM products
WHERE _all @@ 'description:laptop AND embedding:[[0.1, ...]]'
ORDER BY _meta.score DESC;
-- AND NOT: Similar to one vector but not another
SELECT * FROM articles
WHERE _all @@ 'embedding:[[0.1, ...]] AND NOT embedding:[[0.9, ...]]'
ORDER BY _meta.score DESC;
-- Unary operators (+ for must match, - for must not match)
SELECT * FROM products
WHERE _all @@ '+embedding:[[0.1, ...]] -embedding:[[0.9, ...]]'
ORDER BY _meta.score DESC;
```
### 15.3 pgvector-Compatible Distance Operators and Functions
Cognica provides full compatibility with PostgreSQL's pgvector extension, enabling you to use familiar distance operators and functions. This allows seamless migration from pgvector-based applications and leverages existing pgvector knowledge.
#### Distance Operators
Cognica supports three pgvector-style distance operators that can be used in `SELECT`, `WHERE`, and `ORDER BY` clauses:
| Operator | Distance Type | Description | Range |
|----------|--------------|-------------|-------|
| `<->` | L2 (Euclidean) | Square root of sum of squared differences | [0, +inf) |
| `<=>` | Cosine | 1 - cosine similarity | [0, 2] |
| `<#>` | Negative Inner Product | Negated dot product (for ORDER BY ASC) | (-inf, +inf) |
**Basic Usage**:
```sql
-- L2 (Euclidean) distance
SELECT id, embedding <-> '[1,2,3]' AS distance
FROM items
ORDER BY embedding <-> '[1,2,3]'
LIMIT 10;
-- Cosine distance
SELECT id, embedding <=> '[1,2,3]' AS distance
FROM items
ORDER BY embedding <=> '[1,2,3]'
LIMIT 10;
-- Negative inner product (for maximum inner product search)
SELECT id, embedding <#> '[1,2,3]' AS neg_inner_product
FROM items
ORDER BY embedding <#> '[1,2,3]'
LIMIT 10;
```
**Using in WHERE Clauses**:
```sql
-- Find items within a distance threshold
SELECT id, name, embedding <-> '[0.1, 0.2, 0.3]' AS distance
FROM products
WHERE embedding <-> '[0.1, 0.2, 0.3]' < 1.5
ORDER BY distance;
-- Combine with other conditions
SELECT id, name
FROM products
WHERE category = 'electronics'
AND embedding <=> '[0.5, 0.3, 0.2]' < 0.3
ORDER BY embedding <=> '[0.5, 0.3, 0.2]';
```
#### Vector Distance Functions
For more explicit operations, Cognica provides distance functions that mirror pgvector:
| Function | Description | Equivalent Operator |
|----------|-------------|---------------------|
| `l2_distance(vec1, vec2)` | Euclidean distance | `<->` |
| `cosine_distance(vec1, vec2)` | Cosine distance | `<=>` |
| `inner_product(vec1, vec2)` | Dot product | (positive of `<#>`) |
**Function Usage**:
```sql
-- Using distance functions explicitly
SELECT
id,
l2_distance(embedding, '[1,2,3]') AS l2_dist,
cosine_distance(embedding, '[1,2,3]') AS cos_dist,
inner_product(embedding, '[1,2,3]') AS dot_product
FROM items;
-- In WHERE clause
SELECT * FROM products
WHERE l2_distance(embedding, '[0.1, 0.2, 0.3]') < 2.0;
-- Ordering by function result
SELECT id, name
FROM products
ORDER BY cosine_distance(embedding, '[0.5, 0.5, 0.5]')
LIMIT 10;
```
#### Vector Utility Functions
Additional functions for working with vectors:
| Function | Description | Example |
|----------|-------------|---------|
| `vector_dims(vec)` | Returns the number of dimensions | `vector_dims('[1,2,3]')` returns `3` |
| `vector_norm(vec)` | Returns the L2 norm (magnitude) | `vector_norm('[3,4]')` returns `5.0` |
**Utility Function Examples**:
```sql
-- Check vector dimensions
SELECT id, vector_dims(embedding) AS dims
FROM items
WHERE vector_dims(embedding) = 384;
-- Filter by vector magnitude
SELECT id, vector_norm(embedding) AS magnitude
FROM items
WHERE vector_norm(embedding) > 0.9; -- Near-unit vectors
-- Verify vectors are normalized
SELECT id,
vector_norm(embedding) AS norm,
CASE WHEN ABS(vector_norm(embedding) - 1.0) < 0.001
THEN 'normalized'
ELSE 'not normalized'
END AS status
FROM items;
```
#### Vector Format
Vectors can be specified in pgvector-compatible string format:
```sql
-- pgvector format: '[val1, val2, val3, ...]'
SELECT '[1.0, 2.0, 3.0]'::vector <-> '[4.0, 5.0, 6.0]'::vector;
-- Also works with array format
SELECT ARRAY[1.0, 2.0, 3.0] <-> ARRAY[4.0, 5.0, 6.0];
```
#### Choosing Between Operators and Functions
| Use Case | Recommended | Reason |
|----------|-------------|--------|
| ORDER BY nearest neighbor | Operators (`<->`, `<=>`) | Cleaner syntax, index-friendly |
| Compute distance for display | Functions | Self-documenting code |
| Complex expressions | Functions | Better readability |
| Migration from pgvector | Operators | Direct compatibility |
**Performance Note**: Both operators and functions use the same underlying implementation. Choose based on readability and coding style preferences.
### 15.4 HNSW Index Architecture
Cognica uses the Hierarchical Navigable Small World (HNSW) algorithm for approximate nearest neighbor search. HNSW builds a multi-layer graph where each layer contains progressively fewer nodes, enabling efficient logarithmic-time search.
#### Index Types
| Index Type | Backend | Best For |
|------------|---------|----------|
| `hnsw` | Customized FAISS | General use, maximum performance |
| `ivf_hnsw` | Customized FAISS | Very large datasets (10M+ vectors) |
| `hnsw_usearch` | USearch | Memory-constrained environments |
**When to Use Each Type**:
- **hnsw**: Default choice. Uses Cognica's customized FAISS fork, which includes optimizations for concurrent access, incremental updates, and integration with the transaction system. Provides excellent query performance with reasonable memory usage.
- **ivf_hnsw**: For datasets with millions of vectors. Uses the customized FAISS backend with an Inverted File (IVF) layer that partitions vectors into clusters, reducing search space.
- **hnsw_usearch**: When memory is constrained. USearch uses less memory than FAISS with slightly different performance characteristics.
#### Distance Metrics
| Metric | Mathematical Operation | Use When |
|--------|----------------------|----------|
| `inner_product` | Maximize dot product | Vectors are normalized (cosine similarity) |
| `l2` | Minimize Euclidean distance | General purpose, unnormalized vectors |
| `l1` | Minimize Manhattan distance | Sparse or high-dimensional data |
**Cosine Similarity via Inner Product**:
Most embedding models produce normalized vectors (unit length). For normalized vectors, cosine similarity equals inner product, making `inner_product` the correct choice. If your vectors are not normalized, normalize them before storage or use `l2`.
#### Key Parameters
| Parameter | Default | Description | Impact |
|-----------|---------|-------------|--------|
| `dims` | Required | Vector dimension | Must match embedding model output |
| `M` | 16 | Max connections per node | Higher = better recall, more memory |
| `ef_construction` | 200 | Build-time search depth | Higher = better index, slower build |
| `ef_search` | 50 | Query-time search depth | Higher = better accuracy, slower query |
**Tuning Guidelines**:
- **Recall vs Speed**: Increase `ef_search` for better recall at cost of latency
- **Index Quality**: Higher `ef_construction` builds better indexes but takes longer
- **Memory Usage**: `M` directly affects memory; 16 is a good default, increase to 32 or 64 for higher recall requirements
- **Dimension Matching**: `dims` must exactly match your embedding model; mismatches cause errors or silent failures
### 15.5 Two-Dimensional Vector Storage
Traditional vector databases force a choice for long documents: either store the entire document as one embedding (losing detail) or store each chunk as a separate row (complicating queries with joins).
Cognica solves this with 2D vector arrays. A single document can contain multiple chunk embeddings stored together:
```sql
-- Document structure with 2D embeddings
-- Each document has: id, title, content, chunk_embeddings (2D array)
-- Query searches all chunks, returns parent documents
SELECT doc_id, title, _meta.score
FROM research_papers
WHERE _all @@ 'chunk_embeddings:[[0.1, 0.2, ...]]'
ORDER BY _meta.score DESC
LIMIT 10;
```
**Benefits of 2D Storage**:
| Approach | Rows per Document | Query Complexity | Cognica Advantage |
|----------|-------------------|------------------|-------------------|
| Single embedding | 1 | Simple | Loses chunk-level detail |
| Chunk per row | N (one per chunk) | Requires JOIN | Complex, duplicates metadata |
| 2D array (Cognica) | 1 | Simple | Best of both worlds |
**Practical Example: Legal Document Search**:
```sql
-- Search 10,000 legal contracts, each with 50+ page chunks
-- Query finds the most relevant contract, not individual chunks
SELECT
case_id,
case_name,
court,
_meta.score AS relevance
FROM legal_cases
WHERE jurisdiction = 'Federal'
AND _all @@ 'chunk_embeddings:[[0.15, 0.28, 0.33, ...]]'
ORDER BY _meta.score DESC
LIMIT 20;
```
### 15.6 Hybrid Search: Combining Text and Vectors
Pure vector search has limitations. A search for "Python programming" might return articles about snakes if the semantic similarity is high enough. Hybrid search combines keyword matching with semantic similarity for more precise results.
**Pattern 1: Keyword Filter + Vector Ranking**
```sql
-- Must contain "Python", ranked by semantic similarity to query
SELECT id, title, _meta.score
FROM articles
WHERE _all @@ 'content:Python AND content_embedding:[[0.1, 0.2, ...]]'
ORDER BY _meta.score DESC
LIMIT 10;
```
**Pattern 2: SQL Filter + Vector Search**
```sql
-- Combine SQL WHERE with vector similarity
SELECT id, name, price, _meta.score
FROM products
WHERE category = 'Electronics'
AND price BETWEEN 500 AND 1500
AND in_stock = true
AND _all @@ 'description_embedding:[[0.12, 0.34, ...]]'
ORDER BY _meta.score DESC
LIMIT 20;
```
**Pattern 3: Multi-Vector Queries**
```sql
-- Must be similar to positive_query, must NOT be similar to negative_query
SELECT id, title, _meta.score
FROM articles
WHERE _all @@ 'embedding:[[positive_query...]] AND NOT embedding:[[negative_query...]]'
ORDER BY _meta.score DESC;
```
### 15.7 RAG Application Patterns
Retrieval-Augmented Generation (RAG) combines vector search with Large Language Models (LLMs). Cognica's SQL interface makes building RAG pipelines straightforward.
#### Basic RAG Retrieval
```sql
-- Step 1: Retrieve relevant context for the LLM
SELECT content, _meta.score AS relevance
FROM knowledge_base
WHERE _all @@ 'content_embedding:[[user_query_embedding...]]'
ORDER BY _meta.score DESC
LIMIT 5;
-- Step 2: Pass retrieved content + user query to LLM (application layer)
```
#### RAG with Metadata Filtering
```sql
-- Retrieve only recent, authoritative sources
SELECT
content,
source,
published_date,
_meta.score AS relevance
FROM knowledge_base
WHERE published_date > '2024-01-01'
AND source_type IN ('official_docs', 'peer_reviewed')
AND _all @@ 'content_embedding:[[query_embedding...]]'
ORDER BY _meta.score DESC
LIMIT 5;
```
#### RAG with Negative Examples
When users report bad answers, store the problematic context embeddings and exclude them:
```sql
-- Avoid retrieving content similar to known-bad examples
SELECT content, _meta.score
FROM knowledge_base
WHERE _all @@ 'content_embedding:[[query...]] AND NOT content_embedding:[[bad_example...]]'
ORDER BY _meta.score DESC
LIMIT 5;
```
#### Multi-Hop RAG
For complex questions requiring multiple retrieval steps:
```sql
-- First hop: Find relevant documents
WITH first_hop AS (
SELECT doc_id, content, _meta.score AS score1
FROM documents
WHERE _all @@ 'embedding:[[initial_query...]]'
ORDER BY _meta.score DESC
LIMIT 10
)
-- Second hop: Find related documents using first hop results
SELECT d.doc_id, d.content, d._meta.score AS score2, f.score1
FROM documents d
JOIN first_hop f ON true
WHERE d._all @@ 'embedding:[[refined_query_from_first_hop...]]'
ORDER BY d._meta.score DESC
LIMIT 5;
```
### 15.8 Performance Tuning
#### Index Parameter Selection
| Dataset Size | Recommended M | Recommended ef_construction | Recommended ef_search |
|--------------|---------------|----------------------------|----------------------|
| < 100K vectors | 16 | 100 | 50 |
| 100K - 1M vectors | 24 | 200 | 100 |
| 1M - 10M vectors | 32 | 300 | 150 |
| > 10M vectors | 48+ (use ivf_hnsw) | 400 | 200 |
#### Query Optimization
**1. Filter Before Vector Search**:
SQL filters applied before vector search reduce the candidate set:
```sql
-- Good: SQL filter first, then vector search on smaller set
SELECT * FROM products
WHERE category = 'Electronics' AND price < 1000
AND _all @@ 'embedding:[[...]]'
ORDER BY _meta.score DESC;
```
**2. Limit Early**:
Always use LIMIT. Vector search without limits scans the entire index:
```sql
-- Always include LIMIT
SELECT * FROM articles
WHERE _all @@ 'embedding:[[...]]'
ORDER BY _meta.score DESC
LIMIT 20; -- Essential for performance
```
**3. Choose Appropriate ef_search**:
Higher `ef_search` improves recall but increases latency:
| Recall Target | ef_search Setting |
|--------------|-------------------|
| 90% | 50 (default) |
| 95% | 100 |
| 99% | 200+ |
### 15.9 Best Practices
**1. Match Embedding Dimensions**:
The index `dims` must exactly match your embedding model output. Mismatches cause errors or corrupted results.
```
Model: all-MiniLM-L6-v2 (384 dims) -> Index dims: 384
Model: OpenAI ada-002 (1536 dims) -> Index dims: 1536
```
**2. Normalize Vectors for Cosine Similarity**:
When using `inner_product` metric, ensure vectors are unit normalized. Most embedding models output normalized vectors, but verify with your specific model.
**3. Use 2D Vectors for Chunked Documents**:
Store chunk embeddings together in a 2D array rather than creating separate rows. This eliminates joins and keeps documents atomic.
**4. Leverage NOT for Semantic Filtering**:
Use vector exclusion to remove unwanted semantic clusters. This is particularly powerful for avoiding known-bad results in RAG applications.
**5. Combine SQL and Vector Filters**:
Apply SQL filters for structured constraints (date ranges, categories, permissions) and vector search for semantic relevance:
```sql
SELECT * FROM products
WHERE price < 1000
AND category = 'Electronics'
AND in_stock = true
AND _all @@ 'embedding:[[...]]'
ORDER BY _meta.score DESC
LIMIT 20;
```
**6. Monitor Recall vs Latency**:
Use application-level metrics to track search quality. If users report missing relevant results, increase `ef_search`. If latency is too high, consider `ivf_hnsw` for large datasets or reduce `ef_search`.
---
## Chapter 16: Graph Database - Modeling Connected Data
Graph databases excel at modeling and querying highly connected data such as social networks, knowledge graphs, fraud detection systems, and recommendation engines. Cognica provides native graph capabilities through SQL table functions, offering Apache AGE-compatible semantics while maintaining full SQL integration.
### Why Graph Databases Matter
Relational databases struggle with queries that traverse relationships:
```sql
-- Finding friends-of-friends in SQL requires complex JOINs
SELECT DISTINCT f2.user_id
FROM friendships f1
JOIN friendships f2 ON f1.friend_id = f2.user_id
WHERE f1.user_id = 'alice'
AND f2.user_id != 'alice';
-- Finding 6 degrees of separation? Nearly impossible with JOINs.
```
Graph databases treat relationships as first-class citizens:
```sql
-- Same query with graph functions - simple and scalable
SELECT * FROM graph_neighbors('social', 'alice', 'FRIEND', 'both', 2);
```
### Cognica's Graph Model
Cognica uses the **Property Graph Model**:
- **Nodes**: Entities with labels and properties (JSON)
- **Edges**: Directed relationships with types and properties (JSON)
- **Graphs**: Named containers holding nodes and edges
Internally, each graph is stored as two document collections:
- `{graph_name}_nodes`: All nodes with their labels and properties
- `{graph_name}_edges`: All edges with source, target, type, and properties
### 16.1 Graph Management
#### Creating a Graph
```sql
-- Create a new graph namespace
SELECT * FROM graph_create('social_network');
-- Result: name = 'social_network'
-- This creates two collections:
-- social_network_nodes
-- social_network_edges
```
#### Listing and Checking Graphs
```sql
-- List all graphs
SELECT * FROM graph_list();
-- | name |
-- |-----------------|
-- | social_network |
-- | knowledge_base |
-- Check if a graph exists
SELECT * FROM graph_exists('social_network');
-- Result: exists = true
```
#### Dropping a Graph
```sql
-- Drop a graph and all its data
SELECT * FROM graph_drop('social_network');
-- Result: dropped = true
```
### 16.2 Working with Nodes
#### Creating Nodes
```sql
-- Create a person node
SELECT * FROM graph_create_node(
'social', -- graph name
'Person', -- label
'{"name": "Alice", "age": 30, "city": "Seoul"}' -- properties (JSONB)
);
-- Result: node_id = 'social:Person:1'
-- Create more nodes
SELECT * FROM graph_create_node('social', 'Person',
'{"name": "Bob", "age": 28, "city": "Busan"}');
SELECT * FROM graph_create_node('social', 'Company',
'{"name": "TechCorp", "industry": "Software"}');
```
**Node ID Format**: `{graph}:{label}:{sequence}`
#### Querying Nodes
```sql
-- Get a specific node by ID
SELECT * FROM graph_get_node('social', 'social:Person:1');
-- Returns the full node document as JSONB
-- Query all Person nodes
SELECT * FROM graph_nodes('social', 'Person');
-- | node_id | label | properties |
-- |-------------------|--------|------------------------------------------|
-- | social:Person:1 | Person | {"name": "Alice", "age": 30, ...} |
-- | social:Person:2 | Person | {"name": "Bob", "age": 28, ...} |
-- Query with property filter
SELECT * FROM graph_nodes('social', 'Person', '{"city": "Seoul"}');
-- Returns only nodes where city = "Seoul"
-- Query all nodes (no label filter)
SELECT * FROM graph_nodes('social', NULL);
```
#### Updating Nodes
```sql
-- Update a node's properties in-place
SELECT * FROM graph_update_node(
'social', -- graph name
'social:Person:1', -- node ID
'{"age": 31, "city": "Incheon"}' -- new properties (merged)
);
-- Result: success = true
```
#### Deleting Nodes
```sql
-- Delete a node (also removes connected edges)
SELECT * FROM graph_delete_node('social', 'social:Person:2');
-- Result: deleted = true
```
### 16.3 Working with Edges
#### Creating Edges
```sql
-- Create a KNOWS relationship
SELECT * FROM graph_create_edge(
'social', -- graph name
'KNOWS', -- edge type
'social:Person:1', -- source node ID
'social:Person:2', -- target node ID
'{"since": "2020-01-15", "strength": 0.8}' -- properties (JSONB)
);
-- Result: edge_id = 'social:KNOWS:1'
-- Create a WORKS_AT relationship
SELECT * FROM graph_create_edge('social', 'WORKS_AT',
'social:Person:1', 'social:Company:1',
'{"role": "Engineer", "since": "2022-03"}');
```
**Edge ID Format**: `{graph}:{type}:{sequence}`
#### Getting a Single Edge
```sql
-- Get a specific edge by ID
SELECT * FROM graph_get_edge('social', 'social:KNOWS:1');
-- Returns the full edge document as JSONB, or NULL if not found
```
#### Querying Edges
```sql
-- Get edges connected to a node (outgoing by default)
SELECT * FROM graph_edges('social', 'social:Person:1');
-- | edge_id | type | source | target | properties |
-- |---------------|----------|-------------------|-------------------|-------------|
-- | social:KNOWS:1| KNOWS | social:Person:1 | social:Person:2 | {...} |
-- Get incoming edges
SELECT * FROM graph_edges('social', 'social:Person:1', NULL, 'incoming');
-- Get edges of specific type
SELECT * FROM graph_edges('social', 'social:Person:1', 'KNOWS', 'outgoing');
-- Get all edges (both directions)
SELECT * FROM graph_edges('social', 'social:Person:1', NULL, 'both');
```
#### Updating Edges
```sql
-- Update an edge's properties in-place
SELECT * FROM graph_update_edge(
'social', -- graph name
'social:KNOWS:1', -- edge ID
'{"since": "2025-01-01", "strength": 0.95}' -- new properties (merged)
);
-- Result: success = true
```
#### Deleting Edges
```sql
-- Delete an edge
SELECT * FROM graph_delete_edge('social', 'social:KNOWS:1');
-- Result: deleted = true
```
### 16.4 Graph Traversal
#### Finding Neighbors
```sql
-- Find direct neighbors (depth 1)
SELECT * FROM graph_neighbors('social', 'social:Person:1');
-- | node_id | label | properties | depth | path |
-- |-------------------|--------|------------|-------|-------------------------|
-- | social:Person:2 | Person | {...} | 1 | ["social:Person:1", ...] |
-- Find neighbors within 2 hops
SELECT * FROM graph_neighbors('social', 'social:Person:1', NULL, 'both', 2);
-- Find neighbors via specific edge type
SELECT * FROM graph_neighbors('social', 'social:Person:1', 'KNOWS', 'outgoing', 3);
```
#### BFS/DFS Traversal
```sql
-- BFS traversal (default)
SELECT * FROM graph_traverse(
'social', -- graph
'social:Person:1', -- start node
NULL, -- edge types (NULL = all)
'outgoing', -- direction
5, -- max depth
'bfs' -- strategy: 'bfs' or 'dfs'
);
-- DFS traversal with specific edge types
SELECT * FROM graph_traverse(
'social', 'social:Person:1',
ARRAY['KNOWS', 'WORKS_WITH'], -- only follow these edge types
'both', 10, 'dfs'
);
```
**Return Columns**: `node_id`, `label`, `properties`, `depth`, `path`
### 16.5 Path Finding
#### Shortest Path
```sql
-- Find shortest path between two nodes
SELECT * FROM graph_shortest_path(
'social',
'social:Person:1', -- start
'social:Person:100', -- end
NULL, -- edge types (NULL = all)
'both', -- direction
10 -- max depth
);
-- | path | edges | length | total_weight |
-- |-------------------------------|-----------------|--------|--------------|
-- | ["Person:1","Person:5",...] | ["KNOWS:1",...] | 3 | 3.0 |
```
#### All Paths
```sql
-- Find all paths between two nodes
SELECT * FROM graph_all_paths(
'social',
'social:Person:1',
'social:Person:10',
NULL, -- edge types
2, -- min depth
5, -- max depth
10 -- limit (max number of paths)
);
```
#### Reachability Check
```sql
-- Check if two nodes are connected (optimized for early termination)
SELECT * FROM graph_reachable(
'social',
'social:Person:1',
'social:Person:1000',
NULL, -- edge types
6 -- max depth (6 degrees of separation)
);
-- Result: reachable = true/false
```
### 16.6 Graph Analytics
#### Node Degree
```sql
-- Count all edges connected to a node
SELECT * FROM graph_degree('social', 'social:Person:1');
-- Result: degree = 15
-- Count only outgoing KNOWS edges
SELECT * FROM graph_degree('social', 'social:Person:1', 'KNOWS', 'outgoing');
-- Result: degree = 8
```
#### Common Neighbors
```sql
-- Find mutual friends
SELECT * FROM graph_common_neighbors(
'social',
'social:Person:1',
'social:Person:2',
'KNOWS'
);
-- | node_id | label | properties |
-- |-------------------|--------|---------------------------|
-- | social:Person:5 | Person | {"name": "Charlie", ...} |
-- | social:Person:8 | Person | {"name": "Diana", ...} |
```
### 16.7 Performance Optimization
#### Adjacency Cache
Cognica maintains an in-memory adjacency cache for fast multi-hop traversals:
```sql
-- Pre-warm the cache for a graph
SELECT * FROM graph_warm_cache('social', 100000);
-- | cached | time_ms |
-- |---------|---------|
-- | 85432 | 2341 |
-- View cache statistics
SELECT * FROM graph_cache_stats();
-- | entries | hits | misses | hit_rate |
-- |---------|---------|--------|----------|
-- | 85432 | 1234567 | 12345 | 0.99 |
-- Clear cache (if needed)
SELECT * FROM graph_clear_cache('social');
```
#### Best Practices
**1. Pre-warm Cache for Read-Heavy Workloads**:
```sql
-- Warm cache during off-peak hours
SELECT * FROM graph_warm_cache('social', 1000000);
```
**2. Limit Traversal Depth**:
```sql
-- Avoid unbounded traversals
SELECT * FROM graph_traverse('social', 'node:1', NULL, 'both', 5); -- Good
SELECT * FROM graph_traverse('social', 'node:1', NULL, 'both', 100); -- Risky
```
**3. Use Edge Type Filters**:
```sql
-- Filter edge types to reduce search space
SELECT * FROM graph_neighbors('social', 'node:1', 'KNOWS', 'outgoing', 3);
```
**4. Create Indexes on Properties**:
```sql
-- Index frequently queried node properties
CREATE INDEX ON social_nodes(properties->>'name');
CREATE INDEX ON social_edges(type);
```
### 16.8 Use Cases
#### Social Network
```sql
-- Create social graph
SELECT * FROM graph_create('social');
-- Add users
SELECT * FROM graph_create_node('social', 'User', '{"name": "Alice"}');
SELECT * FROM graph_create_node('social', 'User', '{"name": "Bob"}');
-- Add friendship
SELECT * FROM graph_create_edge('social', 'FRIEND',
'social:User:1', 'social:User:2', '{}');
-- Find friends-of-friends
SELECT * FROM graph_neighbors('social', 'social:User:1', 'FRIEND', 'both', 2);
```
#### Knowledge Graph
```sql
-- Create knowledge graph
SELECT * FROM graph_create('knowledge');
-- Add entities
SELECT * FROM graph_create_node('knowledge', 'Concept', '{"name": "Machine Learning"}');
SELECT * FROM graph_create_node('knowledge', 'Concept', '{"name": "Neural Networks"}');
SELECT * FROM graph_create_node('knowledge', 'Concept', '{"name": "Deep Learning"}');
-- Add relationships
SELECT * FROM graph_create_edge('knowledge', 'IS_A',
'knowledge:Concept:2', 'knowledge:Concept:1', '{}'); -- NN is a ML
SELECT * FROM graph_create_edge('knowledge', 'USES',
'knowledge:Concept:3', 'knowledge:Concept:2', '{}'); -- DL uses NN
-- Query related concepts
SELECT * FROM graph_neighbors('knowledge', 'knowledge:Concept:1', NULL, 'incoming', 3);
```
#### Fraud Detection
```sql
-- Find suspicious transaction patterns
SELECT * FROM graph_all_paths(
'transactions',
'account:suspicious_1',
'account:suspicious_2',
ARRAY['TRANSFER'],
1, -- min depth
5, -- max depth
100 -- limit
);
```
### 16.9 Cypher Query Language
In addition to SQL table functions, Cognica supports the **Cypher** query language for graph operations. Cypher is the industry-standard declarative language for property graphs, providing expressive pattern matching that is often more natural than equivalent SQL for graph queries.
Cypher queries are embedded in SQL using the `cypher()` table function in the FROM clause. Cognica's Cypher implementation parses Cypher syntax and rewrites it into equivalent SQL subqueries that use the graph table functions internally.
#### Basic Syntax
```sql
-- Cypher queries are embedded in SQL via the cypher() function
-- The $$ ... $$ delimiter encloses the Cypher query text
SELECT * FROM cypher('graph_name', $$
MATCH (n:Label)
RETURN n.property AS result
$$) AS (result TEXT);
```
The first argument is the graph name, and the Cypher query is enclosed in dollar-quoted strings (`$$ ... $$`). The AS clause defines the output column names and types.
#### MATCH: Pattern Matching
MATCH finds patterns in the graph using an intuitive visual syntax:
```sql
-- Find all Person nodes
SELECT * FROM cypher('social', $$
MATCH (p:Person)
RETURN p.name AS name, p.age AS age
$$) AS (name TEXT, age INTEGER);
-- Find relationships between nodes
SELECT * FROM cypher('social', $$
MATCH (a:Person)-[r:KNOWS]->(b:Person)
RETURN a.name AS from_name, b.name AS to_name, r.since AS since
$$) AS (from_name TEXT, to_name TEXT, since TEXT);
-- Undirected relationship matching
SELECT * FROM cypher('social', $$
MATCH (a:Person)-[r:KNOWS]-(b:Person)
WHERE a.name = 'Alice'
RETURN b.name AS friend
$$) AS (friend TEXT);
-- Multi-hop pattern matching
SELECT * FROM cypher('social', $$
MATCH (a:Person)-[:KNOWS]->(b:Person)-[:WORKS_AT]->(c:Company)
WHERE a.name = 'Alice'
RETURN b.name AS colleague, c.name AS company
$$) AS (colleague TEXT, company TEXT);
```
**Pattern Syntax:**
| Pattern | Meaning |
|---------|---------|
| `(n)` | Any node bound to variable `n` |
| `(n:Label)` | Node with label `Label` |
| `(n:Label {key: value})` | Node with label and property filter |
| `-[r]->` | Outgoing relationship |
| `<-[r]-` | Incoming relationship |
| `-[r]-` | Relationship in either direction |
| `-[r:TYPE]->` | Relationship with specific type |
| `-[r:TYPE*1..3]->` | Variable-length path (1 to 3 hops) |
#### WHERE: Filtering
```sql
-- Filter with WHERE clause
SELECT * FROM cypher('social', $$
MATCH (p:Person)
WHERE p.age > 25 AND p.city = 'Seoul'
RETURN p.name AS name
$$) AS (name TEXT);
-- String operators
SELECT * FROM cypher('social', $$
MATCH (p:Person)
WHERE p.name STARTS WITH 'A'
RETURN p.name AS name
$$) AS (name TEXT);
```
Supported comparison operators: `=`, `<>`, `<`, `>`, `<=`, `>=`, `IN`, `STARTS WITH`, `ENDS WITH`, `CONTAINS`, `=~` (regex match).
#### CREATE: Creating Nodes and Edges
```sql
-- Create a node
SELECT * FROM cypher('social', $$
CREATE (p:Person {name: 'Charlie', age: 35})
RETURN p.name AS name
$$) AS (name TEXT);
-- Create a relationship
SELECT * FROM cypher('social', $$
MATCH (a:Person {name: 'Alice'}), (b:Person {name: 'Charlie'})
CREATE (a)-[r:KNOWS {since: 2025}]->(b)
RETURN type(r) AS rel_type
$$) AS (rel_type TEXT);
```
#### SET: Updating Properties
```sql
-- Update node properties
SELECT * FROM cypher('social', $$
MATCH (p:Person {name: 'Alice'})
SET p.age = 31, p.city = 'Incheon'
RETURN p.name AS name, p.age AS age
$$) AS (name TEXT, age INTEGER);
-- Merge properties from a map
SELECT * FROM cypher('social', $$
MATCH (p:Person {name: 'Bob'})
SET p += {hobby: 'cycling', active: true}
RETURN p.name AS name
$$) AS (name TEXT);
```
#### DELETE: Removing Nodes and Edges
```sql
-- Delete a relationship
SELECT * FROM cypher('social', $$
MATCH (a:Person {name: 'Alice'})-[r:KNOWS]->(b:Person {name: 'Charlie'})
DELETE r
RETURN a.name AS from_name
$$) AS (from_name TEXT);
-- DETACH DELETE removes a node and all connected edges
SELECT * FROM cypher('social', $$
MATCH (p:Person {name: 'Charlie'})
DETACH DELETE p
$$) AS (dummy TEXT);
```
#### REMOVE: Removing Properties and Labels
```sql
-- Remove a property from a node
SELECT * FROM cypher('social', $$
MATCH (p:Person {name: 'Alice'})
REMOVE p.hobby
RETURN p.name AS name
$$) AS (name TEXT);
-- Remove a label from a node
SELECT * FROM cypher('social', $$
MATCH (p:Person:Employee {name: 'Alice'})
REMOVE p:Employee
RETURN p.name AS name
$$) AS (name TEXT);
```
#### MERGE: Upsert Operations
MERGE finds an existing pattern or creates it if it does not exist:
```sql
-- Create node if not exists
SELECT * FROM cypher('social', $$
MERGE (p:Person {name: 'Diana'})
ON CREATE SET p.created_at = '2025-03-26'
ON MATCH SET p.last_seen = '2025-03-26'
RETURN p.name AS name
$$) AS (name TEXT);
```
#### WITH: Chaining Query Parts
```sql
-- Use WITH to chain query stages
SELECT * FROM cypher('social', $$
MATCH (p:Person)
WITH p ORDER BY p.age DESC LIMIT 5
RETURN p.name AS name, p.age AS age
$$) AS (name TEXT, age INTEGER);
```
#### UNWIND: Expanding Lists
```sql
-- Expand a list into rows
SELECT * FROM cypher('social', $$
UNWIND ['Alice', 'Bob', 'Charlie'] AS name
MATCH (p:Person {name: name})
RETURN p.name AS name, p.age AS age
$$) AS (name TEXT, age INTEGER);
```
#### Aggregation Functions
Cypher supports standard aggregation within RETURN and WITH clauses:
```sql
-- Count and aggregate
SELECT * FROM cypher('social', $$
MATCH (p:Person)-[:KNOWS]->(friend:Person)
RETURN p.name AS person, count(friend) AS friend_count
ORDER BY friend_count DESC
$$) AS (person TEXT, friend_count BIGINT);
-- Collect into lists
SELECT * FROM cypher('social', $$
MATCH (p:Person)-[:KNOWS]->(friend:Person)
RETURN p.name AS person, collect(friend.name) AS friends
$$) AS (person TEXT, friends TEXT);
```
#### Built-in Cypher Functions
| Function | Description | Example |
|----------|-------------|---------|
| `type(r)` | Relationship type | `type(r)` returns `'KNOWS'` |
| `id(n)` | Node or edge ID | `id(n)` returns `'social:Person:1'` |
| `labels(n)` | Node labels | `labels(n)` returns `['Person']` |
| `properties(n)` | All properties as map | `properties(n)` returns `{name: 'Alice', ...}` |
| `startNode(r)` | Source node of relationship | `startNode(r)` |
| `endNode(r)` | Target node of relationship | `endNode(r)` |
| `count(x)` | Count values | `count(n)` |
| `collect(x)` | Collect into list | `collect(n.name)` |
| `exists(pattern)` | Check pattern existence | `exists((n)-[:KNOWS]->())` |
| `all(x IN list WHERE pred)` | All elements satisfy predicate | `all(x IN [1,2,3] WHERE x > 0)` |
| `any(x IN list WHERE pred)` | Any element satisfies predicate | `any(x IN nodes(p) WHERE x.age > 30)` |
| `none(x IN list WHERE pred)` | No element satisfies predicate | `none(x IN list WHERE x IS NULL)` |
| `single(x IN list WHERE pred)` | Exactly one element satisfies | `single(x IN list WHERE x = 1)` |
| `reduce(acc = init, x IN list \| expr)` | Reduce a list | `reduce(s = 0, x IN [1,2,3] \| s + x)` |
| `size(list)` | List length | `size(collect(n))` |
#### OPTIONAL MATCH
OPTIONAL MATCH works like a left outer join, returning NULL for missing matches:
```sql
SELECT * FROM cypher('social', $$
MATCH (p:Person)
OPTIONAL MATCH (p)-[:WORKS_AT]->(c:Company)
RETURN p.name AS person, c.name AS company
$$) AS (person TEXT, company TEXT);
```
#### CALL Subqueries
CALL allows running a subquery for each incoming row:
```sql
SELECT * FROM cypher('social', $$
MATCH (p:Person)
CALL {
WITH p
MATCH (p)-[:KNOWS]->(friend:Person)
RETURN count(friend) AS cnt
}
RETURN p.name AS name, cnt AS friend_count
$$) AS (name TEXT, friend_count BIGINT);
```
#### UNION
Combine results from multiple Cypher queries:
```sql
SELECT * FROM cypher('social', $$
MATCH (p:Person)
RETURN p.name AS name
UNION ALL
MATCH (c:Company)
RETURN c.name AS name
$$) AS (name TEXT);
```
#### FOREACH
Apply updates to each element of a list:
```sql
SELECT * FROM cypher('social', $$
MATCH (p:Person {name: 'Alice'})
FOREACH (tag IN ['active', 'premium'] |
SET p.status = tag
)
RETURN p.name AS name
$$) AS (name TEXT);
```
#### Combining Cypher with SQL
Cypher queries are embedded in standard SQL, so they compose naturally with SQL features:
```sql
-- Join Cypher results with a regular SQL table
SELECT c.person, t.department
FROM cypher('social', $$
MATCH (p:Person)-[:KNOWS]->(f:Person)
WHERE p.name = 'Alice'
RETURN f.name AS person
$$) AS c(person TEXT)
JOIN employees t ON c.person = t.name;
-- Use Cypher in a CTE
WITH friends AS (
SELECT * FROM cypher('social', $$
MATCH (p:Person {name: 'Alice'})-[:KNOWS]->(f:Person)
RETURN f.name AS name, f.age AS age
$$) AS (name TEXT, age INTEGER)
)
SELECT name, age FROM friends WHERE age > 25;
```
### 16.10 Graph Function Reference
#### Table Functions (Graph Management)
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_create(name)` | graph name | name TEXT | Create a new graph namespace |
| `graph_drop(name)` | graph name | success BOOLEAN | Drop a graph and all its data |
| `graph_list()` | none | name TEXT | List all graphs |
| `graph_exists(name)` | graph name | exists BOOLEAN | Check if a graph exists |
#### Table Functions (Node Operations)
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_create_node(graph, label, properties)` | graph, label, JSONB | id TEXT | Create a new node |
| `graph_get_node(graph, node_id)` | graph, node ID | id, label, properties | Get a node by ID |
| `graph_update_node(graph, node_id, properties)` | graph, node ID, JSONB | success BOOLEAN | Update node properties |
| `graph_nodes(graph, label?, properties?)` | graph, optional label, optional JSONB filter | id, label, properties | Query nodes |
| `graph_delete_node(graph, node_id)` | graph, node ID | success BOOLEAN | Delete a node and its edges |
#### Table Functions (Edge Operations)
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_create_edge(graph, type, source, target, properties)` | graph, type, source ID, target ID, JSONB | id TEXT | Create a new edge |
| `graph_get_edge(graph, edge_id)` | graph, edge ID | id, type, source, target, properties | Get an edge by ID |
| `graph_update_edge(graph, edge_id, properties)` | graph, edge ID, JSONB | success BOOLEAN | Update edge properties |
| `graph_edges(graph, node_id, type?, direction?)` | graph, node ID, optional type, optional direction | id, type, source, target, properties | Query edges for a node |
| `graph_delete_edge(graph, edge_id)` | graph, edge ID | success BOOLEAN | Delete an edge |
#### Table Functions (Traversal and Paths)
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_neighbors(graph, start, type?, direction?, max_depth?)` | graph, start node, optional type, direction, depth | node_id, label, properties, depth, path | Find neighbors within depth |
| `graph_traverse(graph, start, types[]?, direction?, max_depth?, strategy?)` | graph, start, edge types, direction, depth, bfs/dfs | node_id, label, properties, depth, path | BFS/DFS traversal |
| `graph_shortest_path(graph, start, end, types[]?, direction?, max_depth?)` | graph, start, end, edge types, direction, depth | path, edges, length, total_weight | Shortest path via bidirectional BFS |
| `graph_all_paths(graph, start, end, types[]?, min_depth?, max_depth?, limit?)` | graph, start, end, edge types, min/max depth, limit | path, edges, length | All paths between two nodes |
| `graph_reachable(graph, start, end, types[]?, max_depth?)` | graph, start, end, edge types, depth | reachable BOOLEAN | Check connectivity |
#### Table Functions (Analytics)
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_degree(graph, node_id, type?, direction?)` | graph, node ID, optional type, direction | degree INTEGER | Count edges for a node |
| `graph_common_neighbors(graph, node_a, node_b, type?)` | graph, node A, node B, optional type | node_id, label, properties | Find mutual neighbors |
#### Table Functions (Cache Management)
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_warm_cache(graph, max_nodes?)` | graph, optional max nodes | cached INTEGER, time_ms INTEGER | Pre-populate adjacency cache |
| `graph_cache_stats()` | none | entries, hits, misses, hit_rate | Cache statistics |
| `graph_clear_cache(graph?)` | optional graph | success BOOLEAN | Clear adjacency cache |
#### Scalar Functions (Internal)
These scalar functions are used internally by the Cypher query rewriter for path materialization. They are not intended for direct use but are available for advanced queries:
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_get_node_json(graph, node_id)` | graph, node ID | JSONB | Get a single node as JSON |
| `graph_get_nodes_json(graph, node_ids)` | graph, node ID array | JSONB | Get multiple nodes as JSON array |
| `graph_get_edge_json(graph, edge_id)` | graph, edge ID | JSONB | Get a single edge as JSON |
| `graph_get_edges_json(graph, edge_ids)` | graph, edge ID array | JSONB | Get multiple edges as JSON array |
---
## Chapter 17: Triggers - Automating Database Actions
Triggers are database objects that automatically execute in response to specific events on a table. They allow you to implement complex business rules, maintain audit trails, and enforce data integrity at the database level.
```mermaid
flowchart TB
subgraph DML["DML Statement"]
INSERT["INSERT"]
UPDATE["UPDATE"]
DELETE["DELETE"]
end
subgraph Before["BEFORE Trigger"]
BEFORE_TRG["Validate/Modify
NEW values"]
end
subgraph Operation["Table Operation"]
EXECUTE["Execute
DML"]
end
subgraph After["AFTER Trigger"]
AFTER_TRG["Audit Log/
Notifications"]
end
INSERT --> BEFORE_TRG
UPDATE --> BEFORE_TRG
DELETE --> BEFORE_TRG
BEFORE_TRG -->|"Pass"| EXECUTE
BEFORE_TRG -.->|"Reject"| CANCEL["Operation
Cancelled"]
EXECUTE --> AFTER_TRG
AFTER_TRG --> DONE["Complete"]
style BEFORE_TRG fill:#fff3cd,color:#856404
style EXECUTE fill:#4a90d9,color:#fff
style AFTER_TRG fill:#d4edda,color:#155724
style CANCEL fill:#f8d7da,color:#721c24
```
### The Power and Danger of Triggers
Triggers are among the most powerful features in a database, but with that power comes significant responsibility. A well-designed trigger can save countless lines of application code and guarantee business rules are enforced regardless of how data is modified. A poorly designed trigger can create debugging nightmares, performance problems, and data inconsistencies.
**Why Triggers Can Be Problematic:**
1. **Hidden Logic**: Code in triggers executes invisibly. A simple UPDATE statement may cascade through multiple triggers, causing unexpected side effects. Developers unfamiliar with the trigger logic will be surprised.
2. **Cascading Triggers**: When a trigger modifies data, it may fire additional triggers. These chains can be difficult to trace and may cause infinite loops if not carefully designed.
3. **Performance Impact**: Row-level triggers execute once per affected row. An UPDATE affecting 1 million rows means 1 million trigger executions. Statement-level triggers help but are not always applicable.
4. **Testing Difficulty**: Triggers are hard to unit test in isolation. They depend on database state and fire automatically, making controlled testing challenging.
**When Triggers Are the Right Choice:**
- Audit logging that must capture all changes regardless of source
- Automatic timestamp maintenance (created_at, updated_at)
- Denormalized summary tables that must stay synchronized
- Enforcing rules that cannot be expressed as CHECK constraints
- Cross-table validation that foreign keys cannot handle
**When to Consider Alternatives:**
- Complex business logic that changes frequently (application code is easier to test and deploy)
- Operations that may need to be bypassed in certain circumstances
- Logic that requires external system calls or long-running operations
### Trigger Execution Order
When multiple triggers exist on the same table for the same event and timing, they execute in alphabetical order by name. This is predictable but can lead to subtle bugs if triggers depend on each other's effects. Name triggers thoughtfully if order matters.
### 17.1 Understanding Triggers
#### What Are Triggers?
A trigger is a stored procedure that automatically runs when certain events occur on a table:
- **INSERT**: A new row is added
- **UPDATE**: An existing row is modified
- **DELETE**: A row is removed
- **TRUNCATE**: All rows are removed at once
Triggers can run **before** or **after** the event, and can operate on **each row** individually or **once per statement**.
#### Why Use Triggers?
```sql
-- Without triggers: Must remember to update timestamp everywhere
UPDATE users SET name = 'Alice', updated_at = NOW() WHERE id = 1;
UPDATE users SET email = 'a@b.com', updated_at = NOW() WHERE id = 1;
-- Easy to forget updated_at!
-- With triggers: Automatic timestamp updates
CREATE TRIGGER users_update_timestamp
BEFORE UPDATE ON users
FOR EACH ROW
EXECUTE FUNCTION set_timestamp('updated_at');
-- Now you can just:
UPDATE users SET name = 'Alice' WHERE id = 1;
-- updated_at is automatically set!
```
**Common Use Cases**:
- Automatically set `created_at` and `updated_at` timestamps
- Maintain audit logs of all changes
- Validate complex business rules
- Synchronize denormalized data
- Enforce referential integrity beyond foreign keys
- Send notifications on data changes
### 17.2 Trigger Timing and Events
#### Trigger Timing
| Timing | When It Runs | Use Case |
|--------|-------------|----------|
| `BEFORE` | Before the operation | Validate/modify incoming data |
| `AFTER` | After the operation | Audit logging, notifications |
| `INSTEAD OF` | Replaces the operation | Updatable views |
#### Trigger Events
| Event | Fires When | Available Data |
|-------|-----------|----------------|
| `INSERT` | New row is added | `NEW` row |
| `UPDATE` | Row is modified | `OLD` and `NEW` rows |
| `DELETE` | Row is removed | `OLD` row |
| `TRUNCATE` | Table is truncated | No row data (statement-level only) |
You can combine multiple events in a single trigger:
```sql
-- Single trigger for multiple events
-- (See Section 20.5 for the audit_changes() function implementation)
CREATE TRIGGER audit_changes
AFTER INSERT OR UPDATE OR DELETE ON orders
FOR EACH ROW EXECUTE FUNCTION audit_changes();
```
### 17.3 Row-Level vs Statement-Level Triggers
#### Row-Level Triggers (FOR EACH ROW)
Run once for each row affected by the operation:
```sql
-- First, create the trigger function in PL/Python
CREATE FUNCTION validate_stock_level()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
new_row = TD['new']
if new_row['stock_quantity'] < 0:
plpy.error("Stock quantity cannot be negative")
return None
$$;
-- Then create the trigger (fires once per row)
CREATE TRIGGER check_inventory
BEFORE UPDATE ON products
FOR EACH ROW
EXECUTE FUNCTION validate_stock_level();
-- If UPDATE affects 100 rows, trigger runs 100 times
UPDATE products SET price = price * 1.1 WHERE category = 'electronics';
```
Row-level triggers have access to:
- `NEW`: The new row values (INSERT, UPDATE)
- `OLD`: The old row values (UPDATE, DELETE)
#### Statement-Level Triggers (FOR EACH STATEMENT)
Run once per SQL statement, regardless of how many rows are affected:
```sql
-- Statement-level trigger function
CREATE FUNCTION send_inventory_notification()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
# This runs once per statement, not per row
# Good for batch notifications
plpy.execute("""
INSERT INTO notifications (type, message, created_at)
VALUES ('inventory_update', 'Inventory was updated', NOW())
""")
return None
$$;
-- Fires once per statement
CREATE TRIGGER notify_bulk_update
AFTER UPDATE ON inventory
FOR EACH STATEMENT
EXECUTE FUNCTION send_inventory_notification();
-- Fires only once, even if 1000 rows updated
UPDATE inventory SET quantity = quantity - 1 WHERE warehouse_id = 5;
```
Statement-level triggers are useful for:
- Notifications that shouldn't spam (one per batch)
- Summary operations
- Cleanup after bulk operations
### 17.4 Creating Triggers
#### Basic Syntax
```sql
CREATE TRIGGER trigger_name
{ BEFORE | AFTER | INSTEAD OF }
{ INSERT | UPDATE | DELETE | TRUNCATE }
[ OR { INSERT | UPDATE | DELETE | TRUNCATE } ... ]
ON table_name
[ FOR EACH { ROW | STATEMENT } ]
[ WHEN ( condition ) ]
EXECUTE FUNCTION function_name(arguments);
```
#### CREATE OR REPLACE
```sql
-- Create or update an existing trigger
-- (See Section 20.9 Example 1 for auto_timestamp() implementation)
CREATE OR REPLACE TRIGGER update_timestamp
BEFORE UPDATE ON users
FOR EACH ROW
EXECUTE FUNCTION auto_timestamp('updated_at');
```
#### UPDATE OF Specific Columns
Trigger only on changes to specific columns:
```sql
-- Create a function to log price changes
CREATE FUNCTION log_price_change()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
plpy.execute("""
INSERT INTO price_history (product_id, old_price, new_price, changed_at)
VALUES ($1, $2, $3, NOW())
""", [TD['new']['id'], TD['old']['price'], TD['new']['price']])
return None
$$;
-- Only fires when price or quantity changes
CREATE TRIGGER audit_price_changes
AFTER UPDATE OF price, quantity ON products
FOR EACH ROW
EXECUTE FUNCTION log_price_change();
-- This will NOT fire the trigger:
UPDATE products SET description = 'New desc' WHERE id = 1;
-- This WILL fire the trigger:
UPDATE products SET price = 29.99 WHERE id = 1;
```
### 17.5 Trigger Functions
Triggers execute trigger functions. Cognica provides built-in functions for common operations.
#### Built-in Trigger Functions
**set_timestamp(column_name)**
Automatically sets a column to the current timestamp:
```sql
-- Set updated_at on every update
CREATE TRIGGER update_timestamp
BEFORE UPDATE ON orders
FOR EACH ROW
EXECUTE FUNCTION set_timestamp('updated_at');
-- Set created_at on insert
CREATE TRIGGER set_created
BEFORE INSERT ON orders
FOR EACH ROW
EXECUTE FUNCTION set_timestamp('created_at');
```
**validate_not_null(column_name)**
Ensures a column is not null, raising an error if it is:
```sql
-- Enforce NOT NULL with custom error message
CREATE TRIGGER validate_email
BEFORE INSERT OR UPDATE ON users
FOR EACH ROW
EXECUTE FUNCTION validate_not_null('email');
```
**suppress_operation()**
Prevents the operation from executing (for BEFORE triggers):
```sql
-- Prevent deletion of admin users
CREATE TRIGGER protect_admins
BEFORE DELETE ON users
FOR EACH ROW
WHEN (OLD.role = 'admin')
EXECUTE FUNCTION suppress_operation();
```
#### Custom Trigger Functions (PL/Python)
For complex trigger logic, create a PL/Python function that accesses trigger context through the `TD` dictionary:
```sql
-- Step 1: Create a trigger function in PL/Python
CREATE FUNCTION audit_changes()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import json
from datetime import datetime
# TD dictionary provides trigger context
event = TD['event'] # 'INSERT', 'UPDATE', or 'DELETE'
table = TD['table_name']
old_row = TD['old'] # None for INSERT
new_row = TD['new'] # None for DELETE
# Build audit record
audit_data = {
'table': table,
'action': event,
'timestamp': datetime.now().isoformat()
}
if event == 'INSERT':
audit_data['new_data'] = new_row
elif event == 'DELETE':
audit_data['old_data'] = old_row
elif event == 'UPDATE':
audit_data['old_data'] = old_row
audit_data['new_data'] = new_row
# Track which fields changed
audit_data['changed_fields'] = [
k for k in new_row.keys()
if old_row.get(k) != new_row.get(k)
]
# Log to audit table
plpy.execute(
"INSERT INTO audit_log (event_data) VALUES ($1)",
[json.dumps(audit_data)]
)
return None # Proceed with operation
$$;
-- Step 2: Attach trigger to table
CREATE TRIGGER audit_user_changes
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW
EXECUTE FUNCTION audit_changes();
```
**Important**: Trigger functions in PL/Python:
- Access context via `TD` dictionary (not function parameters)
- Return `None` or `"OK"` to proceed normally
- Return `"SKIP"` to prevent the operation (BEFORE ROW only)
- Return `"MODIFY"` after changing `TD['new']` to modify incoming data
> **See Also**: For complete PL/Python documentation including all `TD` dictionary keys, allowed Python modules, and security considerations, see [Chapter 22: PL/Python](#chapter-22-plpython---user-defined-functions-and-stored-procedures).
### 17.6 The WHEN Clause
The WHEN clause adds a condition that must be true for the trigger to fire:
```sql
-- Only log significant price changes (> 10%)
CREATE TRIGGER log_significant_changes
AFTER UPDATE ON products
FOR EACH ROW
WHEN (OLD.price * 1.1 < NEW.price OR OLD.price * 0.9 > NEW.price)
EXECUTE FUNCTION log_price_change();
-- Only update timestamp if data actually changed
CREATE TRIGGER smart_timestamp
BEFORE UPDATE ON documents
FOR EACH ROW
WHEN (OLD.* IS DISTINCT FROM NEW.*)
EXECUTE FUNCTION set_timestamp('updated_at');
-- Only fire for active records
CREATE TRIGGER process_active_only
AFTER INSERT ON tasks
FOR EACH ROW
WHEN (NEW.status = 'active')
EXECUTE FUNCTION schedule_task();
```
**Important**: In WHEN conditions:
- `OLD` is available for UPDATE and DELETE
- `NEW` is available for INSERT and UPDATE
- You cannot use subqueries in WHEN clauses
### 17.7 Managing Triggers
#### Viewing Triggers
```sql
-- List all triggers on a table
SELECT * FROM pg_trigger WHERE tgrelid = 'orders'::regclass;
-- Using information_schema
SELECT trigger_name, event_manipulation, action_timing, action_orientation
FROM information_schema.triggers
WHERE event_object_table = 'orders';
```
#### Enabling and Disabling Triggers
```sql
-- Disable a specific trigger
ALTER TABLE orders DISABLE TRIGGER update_timestamp;
-- Enable a specific trigger
ALTER TABLE orders ENABLE TRIGGER update_timestamp;
-- Disable ALL triggers on a table
ALTER TABLE orders DISABLE TRIGGER ALL;
-- Enable ALL triggers on a table
ALTER TABLE orders ENABLE TRIGGER ALL;
-- Disable only user triggers (not system triggers)
ALTER TABLE orders DISABLE TRIGGER USER;
```
**Use Case**: Bulk data loading
```sql
-- Disable triggers for fast bulk insert
ALTER TABLE large_table DISABLE TRIGGER ALL;
COPY large_table FROM '/data/millions_of_rows.csv' WITH (FORMAT csv);
-- Re-enable triggers
ALTER TABLE large_table ENABLE TRIGGER ALL;
-- Manually run any necessary post-load triggers
UPDATE large_table SET updated_at = NOW() WHERE updated_at IS NULL;
```
#### Dropping Triggers
```sql
-- Drop a trigger
DROP TRIGGER update_timestamp ON orders;
-- Drop only if exists (no error if not found)
DROP TRIGGER IF EXISTS update_timestamp ON orders;
```
#### Renaming Triggers
```sql
-- Rename a trigger
ALTER TRIGGER old_name ON orders RENAME TO new_name;
```
### 17.8 Constraint Triggers
Constraint triggers can be deferred until the end of a transaction:
```sql
-- Create a deferrable constraint trigger
CREATE CONSTRAINT TRIGGER check_balance
AFTER UPDATE ON accounts
DEFERRABLE INITIALLY DEFERRED
FOR EACH ROW
EXECUTE FUNCTION verify_account_balance();
-- In a transaction:
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1; -- Might go negative
UPDATE accounts SET balance = balance + 100 WHERE id = 2; -- Fixes the balance
COMMIT; -- Constraint trigger runs here, after both updates
```
**When to Use**:
- Complex integrity checks that span multiple rows
- Checks that need to see the final state of multiple operations
- Cross-table validation that may be temporarily violated during a transaction
### 17.9 Practical Trigger Examples
#### Example 1: Automatic Timestamps with PL/Python
```sql
-- Create a flexible timestamp trigger function
CREATE FUNCTION auto_timestamp()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
from datetime import datetime
# Get column name from trigger arguments (TD['args'])
column = TD['args'][0] if TD['args'] else 'updated_at'
current_time = datetime.now().isoformat()
# Modify the NEW row
if TD['new'] is not None:
TD['new'][column] = current_time
return 'MODIFY'
$$;
-- Apply to products table
CREATE TRIGGER products_created
BEFORE INSERT ON products
FOR EACH ROW
EXECUTE FUNCTION auto_timestamp('created_at');
CREATE TRIGGER products_updated
BEFORE UPDATE ON products
FOR EACH ROW
EXECUTE FUNCTION auto_timestamp('updated_at');
-- Usage: timestamps are set automatically
INSERT INTO products (name, price) VALUES ('Widget', 29.99);
-- created_at is automatically set!
UPDATE products SET price = 34.99 WHERE name = 'Widget';
-- updated_at is automatically updated!
```
#### Example 2: Comprehensive Audit Trail
```sql
-- Create audit table
CREATE TABLE audit_log (
id SERIAL PRIMARY KEY,
table_name TEXT NOT NULL,
record_id TEXT,
action TEXT NOT NULL,
old_data JSONB,
new_data JSONB,
changed_fields TEXT[],
changed_at TIMESTAMPTZ DEFAULT NOW(),
session_user TEXT
);
-- Create comprehensive audit trigger function
CREATE FUNCTION audit_table_changes()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import json
event = TD['event']
table = TD['table_name']
old_row = TD['old']
new_row = TD['new']
# Determine the record ID (try common ID field names)
record_id = None
for id_field in ['id', 'ID', '_id', 'uuid']:
if new_row and id_field in new_row:
record_id = str(new_row[id_field])
break
elif old_row and id_field in old_row:
record_id = str(old_row[id_field])
break
# Calculate changed fields for UPDATE
changed_fields = None
if event == 'UPDATE' and old_row and new_row:
changed_fields = [
k for k in new_row.keys()
if k in old_row and old_row[k] != new_row[k]
]
# Insert audit record
plan = plpy.prepare("""
INSERT INTO audit_log
(table_name, record_id, action, old_data, new_data, changed_fields)
VALUES ($1, $2, $3, $4, $5, $6)
""", ["text", "text", "text", "jsonb", "jsonb", "text[]"])
plpy.execute(plan, [
table,
record_id,
event,
json.dumps(old_row) if old_row else None,
json.dumps(new_row) if new_row else None,
changed_fields
])
return None # Proceed with the operation
$$;
-- Apply to users table
CREATE TRIGGER audit_users
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW
EXECUTE FUNCTION audit_table_changes();
-- Apply to orders table (reuse same function!)
CREATE TRIGGER audit_orders
AFTER INSERT OR UPDATE OR DELETE ON orders
FOR EACH ROW
EXECUTE FUNCTION audit_table_changes();
```
#### Example 3: Prevent Destructive Operations
```sql
-- Create a protection trigger function
CREATE FUNCTION protect_record()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
# Get the condition field and value from trigger arguments
# Usage: protect_record('status', 'finalized')
if len(TD['args']) >= 2:
field = TD['args'][0]
protected_value = TD['args'][1]
old_row = TD['old']
if old_row and old_row.get(field) == protected_value:
plpy.error(
f"Cannot {TD['event'].lower()} record: "
f"{field} is '{protected_value}'"
)
return None
$$;
-- Prevent deletion of finalized orders
CREATE TRIGGER protect_finalized_orders
BEFORE DELETE ON orders
FOR EACH ROW
EXECUTE FUNCTION protect_record('status', 'finalized');
-- Prevent updates to archived records
CREATE TRIGGER protect_archived_users
BEFORE UPDATE OR DELETE ON users
FOR EACH ROW
EXECUTE FUNCTION protect_record('status', 'archived');
-- Create a statement-level protection function
CREATE FUNCTION prevent_truncate()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
plpy.error(f"TRUNCATE is not allowed on table '{TD['table_name']}'")
$$;
-- Prevent truncation of critical tables
CREATE TRIGGER prevent_customers_truncate
BEFORE TRUNCATE ON customers
FOR EACH STATEMENT
EXECUTE FUNCTION prevent_truncate();
```
#### Example 4: Cascade Updates (Denormalization)
```sql
-- Update denormalized order total when items change
CREATE FUNCTION recalculate_order_total()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
# Determine which order_id to update
if TD['event'] == 'DELETE':
order_id = TD['old']['order_id']
elif TD['event'] in ('INSERT', 'UPDATE'):
order_id = TD['new']['order_id']
# For UPDATE, also update old order if order_id changed
if TD['event'] == 'UPDATE' and TD['old']['order_id'] != order_id:
old_order_id = TD['old']['order_id']
plpy.execute("""
UPDATE orders
SET total = COALESCE((
SELECT SUM(quantity * unit_price)
FROM order_items
WHERE order_id = $1
), 0)
WHERE id = $1
""", [old_order_id])
# Update the order total
plpy.execute("""
UPDATE orders
SET total = COALESCE((
SELECT SUM(quantity * unit_price)
FROM order_items
WHERE order_id = $1
), 0)
WHERE id = $1
""", [order_id])
return None
$$;
CREATE TRIGGER update_order_total
AFTER INSERT OR UPDATE OR DELETE ON order_items
FOR EACH ROW
EXECUTE FUNCTION recalculate_order_total();
```
#### Example 5: Data Validation with Business Rules
```sql
-- Complex validation that cannot be expressed as CHECK constraint
CREATE FUNCTION validate_order()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
new_row = TD['new']
# Rule 1: Order total must not exceed customer credit limit
result = plpy.execute("""
SELECT credit_limit FROM customers WHERE id = $1
""", [new_row['customer_id']])
if result:
credit_limit = result[0]['credit_limit']
if new_row['total'] > credit_limit:
plpy.error(
f"Order total ({new_row['total']}) exceeds "
f"customer credit limit ({credit_limit})"
)
# Rule 2: Express shipping requires valid phone number
if new_row.get('shipping_method') == 'express':
if not new_row.get('phone'):
plpy.error("Express shipping requires a phone number")
# Rule 3: Cannot order discontinued products
result = plpy.execute("""
SELECT p.name FROM order_items oi
JOIN products p ON p.id = oi.product_id
WHERE oi.order_id = $1 AND p.discontinued = true
""", [new_row['id']])
if result:
discontinued = [r['name'] for r in result]
plpy.error(f"Cannot order discontinued products: {', '.join(discontinued)}")
return None # Validation passed
$$;
CREATE TRIGGER validate_order_rules
BEFORE INSERT OR UPDATE ON orders
FOR EACH ROW
EXECUTE FUNCTION validate_order();
```
#### Example 6: Soft Delete Implementation
```sql
-- Convert DELETE to UPDATE (soft delete)
CREATE FUNCTION soft_delete()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
from datetime import datetime
# Instead of deleting, mark as deleted
record_id = TD['old']['id']
plpy.execute("""
UPDATE users
SET deleted_at = $1, deleted = true
WHERE id = $2
""", [datetime.now().isoformat(), record_id])
# Return SKIP to prevent the actual DELETE
return 'SKIP'
$$;
CREATE TRIGGER users_soft_delete
BEFORE DELETE ON users
FOR EACH ROW
EXECUTE FUNCTION soft_delete();
-- Now DELETE becomes a soft delete
DELETE FROM users WHERE id = 123;
-- Row is not deleted, just marked with deleted_at timestamp
```
### 17.10 Trigger Best Practices
#### Keep Triggers Simple and Focused
Each trigger function should do one thing well:
```sql
-- GOOD: Simple, focused trigger function
CREATE FUNCTION set_updated_timestamp()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
from datetime import datetime
TD['new']['updated_at'] = datetime.now().isoformat()
return 'MODIFY'
$$;
CREATE TRIGGER users_set_updated_at
BEFORE UPDATE ON users
FOR EACH ROW
EXECUTE FUNCTION set_updated_timestamp();
-- BAD: Function tries to do too many things
-- Split into multiple focused triggers instead:
-- - One trigger for timestamps
-- - One trigger for validation
-- - One trigger for audit logging
```
#### Be Careful with Cascading Triggers
Triggers can fire other triggers. Avoid infinite loops by tracking depth:
```sql
-- Create a sync function with recursion protection
CREATE FUNCTION sync_inventory()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
# Check trigger depth to prevent infinite recursion
result = plpy.execute("SELECT pg_trigger_depth() as depth")
if result[0]['depth'] > 1:
plpy.notice("Skipping nested trigger execution")
return None
# Safe to proceed with sync
product_id = TD['new']['product_id']
new_quantity = TD['new']['quantity']
plpy.execute("""
UPDATE inventory_summary
SET total_quantity = (
SELECT SUM(quantity) FROM warehouse_stock
WHERE product_id = $1
)
WHERE product_id = $1
""", [product_id])
return None
$$;
CREATE TRIGGER sync_inventory_on_change
AFTER INSERT OR UPDATE OR DELETE ON warehouse_stock
FOR EACH ROW
EXECUTE FUNCTION sync_inventory();
```
#### Document Your Triggers
```sql
-- Comment on trigger purpose
COMMENT ON TRIGGER users_set_updated_at ON users IS
'Automatically sets updated_at timestamp on any row update';
-- Good trigger names are self-documenting:
-- Format: {table}_{action}_{purpose}
-- Examples:
-- orders_before_insert_validate
-- users_after_update_audit
-- products_before_delete_protect
```
#### Optimize Performance
Avoid expensive operations in row-level triggers:
```sql
-- BAD: Expensive query on every insert
CREATE FUNCTION bad_credit_check()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
# This runs for EVERY row - very slow for bulk inserts!
result = plpy.execute("""
SELECT COUNT(*) as cnt FROM orders
WHERE customer_id = $1 AND status = 'unpaid'
""", [TD['new']['customer_id']])
# ...validation logic
return None
$$;
-- GOOD: Queue expensive checks for async processing
CREATE FUNCTION queue_credit_check()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import json
# Just queue the check - don't block the insert
plpy.execute("""
INSERT INTO pending_validations (type, payload, created_at)
VALUES ('credit_check', $1, NOW())
""", [json.dumps({'order_id': TD['new']['id']})])
return None # Let the insert proceed immediately
$$;
CREATE TRIGGER orders_queue_validation
AFTER INSERT ON orders
FOR EACH ROW
EXECUTE FUNCTION queue_credit_check();
```
#### Use WHEN Clauses to Filter
Filter at the trigger level, not inside the function:
```sql
-- GOOD: Only fire when status actually changes
CREATE FUNCTION log_order_status()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import json
plpy.execute("""
INSERT INTO status_history (order_id, old_status, new_status)
VALUES ($1, $2, $3)
""", [TD['new']['id'], TD['old']['status'], TD['new']['status']])
return None
$$;
CREATE TRIGGER orders_log_status_change
AFTER UPDATE ON orders
FOR EACH ROW
WHEN (OLD.status IS DISTINCT FROM NEW.status) -- Filter here!
EXECUTE FUNCTION log_order_status();
-- BAD: Fire on every update, check inside function
-- This is less efficient because the function is called unnecessarily
```
#### Handle Errors Gracefully
Use try/except for robust error handling:
```sql
CREATE FUNCTION resilient_audit()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import json
try:
plpy.execute("""
INSERT INTO audit_log (table_name, action, data)
VALUES ($1, $2, $3)
""", [TD['table_name'], TD['event'], json.dumps(TD['new'])])
except Exception as e:
# Log the error but don't fail the main operation
plpy.warning(f"Audit logging failed: {e}")
return None # Always allow the main operation to proceed
$$;
```
#### Trigger Execution Order
When multiple triggers exist on the same table and event, they fire in alphabetical order by trigger name:
```sql
-- These fire in order: a_first, b_second, c_third
-- Use naming conventions to control execution order
CREATE TRIGGER a_validate_data BEFORE INSERT ON t
EXECUTE FUNCTION validate_data();
CREATE TRIGGER b_set_defaults BEFORE INSERT ON t
EXECUTE FUNCTION set_defaults();
CREATE TRIGGER c_log_creation BEFORE INSERT ON t
EXECUTE FUNCTION log_creation();
```
---
## Chapter 18: PL/pgSQL - PostgreSQL Procedural Language
PL/pgSQL is PostgreSQL's native procedural language extension. Cognica provides full support for PL/pgSQL, allowing you to write functions and stored procedures using familiar PostgreSQL syntax. This chapter covers all aspects of PL/pgSQL programming in Cognica.
### 18.1 Understanding PL/pgSQL
#### What is PL/pgSQL?
PL/pgSQL (Procedural Language/PostgreSQL) is a loadable procedural language that extends SQL with control structures, variables, and exception handling. Unlike Python-based UDFs, PL/pgSQL runs directly within the database engine, offering tight integration with SQL and efficient execution.
```sql
-- A simple PL/pgSQL function
CREATE FUNCTION greet(name TEXT)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
BEGIN
RETURN 'Hello, ' || name || '!';
END;
$$;
SELECT greet('World');
-- Result: Hello, World!
```
#### Why Choose PL/pgSQL?
**SQL Integration**: PL/pgSQL is designed to work seamlessly with SQL:
```sql
CREATE FUNCTION get_customer_orders(customer_id INTEGER)
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
order_count INTEGER;
BEGIN
SELECT COUNT(*) INTO order_count
FROM orders
WHERE orders.customer_id = get_customer_orders.customer_id;
RETURN order_count;
END;
$$;
```
**PostgreSQL Compatibility**: Functions written in PL/pgSQL for PostgreSQL work in Cognica with minimal or no changes, enabling easy migration of existing code.
**Trigger Support**: PL/pgSQL is the standard language for writing trigger functions:
```sql
CREATE FUNCTION update_timestamp()
RETURNS TRIGGER
LANGUAGE plpgsql
AS $$
BEGIN
NEW.updated_at := NOW();
RETURN NEW;
END;
$$;
```
#### PL/pgSQL vs PL/Python: When to Use Each
| Aspect | PL/pgSQL | PL/Python |
|--------|----------|-----------|
| SQL integration | Native, seamless | Via plpy module |
| Control structures | SQL-like syntax | Python syntax |
| Complex algorithms | Possible but verbose | More natural |
| String manipulation | Basic functions | Rich Python libraries |
| Data science | Not suited | NumPy, statistics available |
| PostgreSQL migration | Direct compatibility | Requires rewrite |
| Trigger functions | Standard choice | Supported |
**Use PL/pgSQL when:**
- Migrating from PostgreSQL
- Writing trigger functions
- Logic is primarily SQL-based
- Maximum database integration is needed
**Use PL/Python when:**
- Complex string processing is required
- Mathematical/statistical computations are needed
- Python's rich standard library would help
- Starting fresh with no PostgreSQL legacy
### 18.2 Block Structure and Declarations
#### Basic Block Structure
Every PL/pgSQL function body consists of one or more blocks:
```sql
CREATE FUNCTION example()
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
-- Variable declarations (optional)
counter INTEGER := 0;
BEGIN
-- Executable statements
counter := counter + 1;
RETURN counter;
END;
$$;
```
The structure is:
1. **DECLARE** (optional): Variable declarations
2. **BEGIN**: Start of executable section
3. **END**: End of block
#### Nested Blocks
Blocks can be nested for scope control:
```sql
CREATE FUNCTION nested_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
outer_var TEXT := 'outer';
BEGIN
DECLARE
inner_var TEXT := 'inner';
BEGIN
RETURN outer_var || ' and ' || inner_var;
END;
-- inner_var is not accessible here
END;
$$;
```
#### Labels for Blocks
Labels allow referencing outer blocks and controlling loop flow:
```sql
CREATE FUNCTION labeled_blocks()
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
<>
DECLARE
x INTEGER := 10;
BEGIN
<>
DECLARE
x INTEGER := 20;
BEGIN
-- Reference outer block's variable
RETURN outer_block.x + inner_block.x; -- Returns 30
END;
END;
$$;
```
### 18.3 Variables and Data Types
#### Variable Declaration
Variables are declared in the DECLARE section:
```sql
CREATE FUNCTION variable_examples()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
-- Basic types
count INTEGER;
name TEXT;
price NUMERIC(10, 2);
is_active BOOLEAN;
-- With initialization
status TEXT := 'pending';
total INTEGER DEFAULT 0;
-- NOT NULL constraint
required_field TEXT NOT NULL := 'must have value';
-- CONSTANT (cannot be changed)
tax_rate CONSTANT NUMERIC := 0.08;
-- %TYPE: Copy type from column
user_email users.email%TYPE;
-- %ROWTYPE: Copy entire row structure
user_record users%ROWTYPE;
BEGIN
-- Variable usage
count := 42;
name := 'Alice';
is_active := TRUE;
RETURN name || ' has ' || count || ' items';
END;
$$;
```
#### Type Reference: %TYPE and %ROWTYPE
These special syntax elements ensure type consistency with table columns:
```sql
CREATE FUNCTION type_reference_example(user_id INTEGER)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
-- Variable has same type as users.email column
v_email users.email%TYPE;
-- Variable can hold an entire users row
v_user users%ROWTYPE;
BEGIN
SELECT email INTO v_email FROM users WHERE id = user_id;
SELECT * INTO v_user FROM users WHERE id = user_id;
RETURN 'Email: ' || v_email || ', Name: ' || v_user.name;
END;
$$;
```
#### Record Type
For dynamic row structures:
```sql
CREATE FUNCTION record_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
rec RECORD;
BEGIN
-- Record adapts to any query result
SELECT id, name, email INTO rec FROM users LIMIT 1;
RETURN rec.name || ' <' || rec.email || '>';
END;
$$;
```
#### Array Variables
```sql
CREATE FUNCTION array_example()
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
numbers INTEGER[] := ARRAY[1, 2, 3, 4, 5];
names TEXT[] := '{Alice, Bob, Charlie}';
total INTEGER := 0;
num INTEGER;
BEGIN
FOREACH num IN ARRAY numbers LOOP
total := total + num;
END LOOP;
RETURN total; -- Returns 15
END;
$$;
```
### 18.4 Control Flow Statements
#### IF Statement
```sql
CREATE FUNCTION if_example(score INTEGER)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
BEGIN
IF score >= 90 THEN
RETURN 'A';
ELSIF score >= 80 THEN
RETURN 'B';
ELSIF score >= 70 THEN
RETURN 'C';
ELSIF score >= 60 THEN
RETURN 'D';
ELSE
RETURN 'F';
END IF;
END;
$$;
SELECT if_example(85); -- Returns 'B'
```
#### Simple CASE Statement
```sql
CREATE FUNCTION case_example(status TEXT)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
BEGIN
CASE status
WHEN 'pending' THEN
RETURN 'Order is being processed';
WHEN 'shipped' THEN
RETURN 'Order is on its way';
WHEN 'delivered' THEN
RETURN 'Order has arrived';
ELSE
RETURN 'Unknown status';
END CASE;
END;
$$;
```
#### Searched CASE Statement
```sql
CREATE FUNCTION searched_case_example(amount NUMERIC)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
BEGIN
CASE
WHEN amount < 0 THEN
RETURN 'Negative balance';
WHEN amount = 0 THEN
RETURN 'Zero balance';
WHEN amount < 100 THEN
RETURN 'Low balance';
WHEN amount < 1000 THEN
RETURN 'Normal balance';
ELSE
RETURN 'High balance';
END CASE;
END;
$$;
```
### 18.5 Loops
#### Basic LOOP
```sql
CREATE FUNCTION basic_loop_example()
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
counter INTEGER := 0;
BEGIN
LOOP
counter := counter + 1;
EXIT WHEN counter >= 10;
END LOOP;
RETURN counter; -- Returns 10
END;
$$;
```
#### WHILE Loop
```sql
CREATE FUNCTION while_loop_example(n INTEGER)
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
result INTEGER := 1;
i INTEGER := 1;
BEGIN
WHILE i <= n LOOP
result := result * i;
i := i + 1;
END LOOP;
RETURN result; -- Returns n! (factorial)
END;
$$;
SELECT while_loop_example(5); -- Returns 120
```
#### FOR Loop (Integer Range)
```sql
CREATE FUNCTION for_loop_example(n INTEGER)
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
total INTEGER := 0;
BEGIN
-- Count from 1 to n
FOR i IN 1..n LOOP
total := total + i;
END LOOP;
RETURN total;
END;
$$;
-- Reverse loop
CREATE FUNCTION reverse_for_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT := '';
BEGIN
FOR i IN REVERSE 5..1 LOOP
result := result || i::TEXT;
END LOOP;
RETURN result; -- Returns '54321'
END;
$$;
-- Loop with step
CREATE FUNCTION step_for_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT := '';
BEGIN
FOR i IN 1..10 BY 2 LOOP
result := result || i::TEXT || ' ';
END LOOP;
RETURN result; -- Returns '1 3 5 7 9 '
END;
$$;
```
#### FOR Loop (Query Results)
```sql
CREATE FUNCTION query_loop_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
rec RECORD;
result TEXT := '';
BEGIN
FOR rec IN SELECT id, name FROM users ORDER BY id LIMIT 5 LOOP
result := result || rec.id || ': ' || rec.name || E'\n';
END LOOP;
RETURN result;
END;
$$;
```
#### FOREACH Loop (Array Iteration)
```sql
CREATE FUNCTION foreach_example(items TEXT[])
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
item TEXT;
result TEXT := '';
BEGIN
FOREACH item IN ARRAY items LOOP
result := result || item || ', ';
END LOOP;
RETURN RTRIM(result, ', ');
END;
$$;
SELECT foreach_example(ARRAY['apple', 'banana', 'cherry']);
-- Returns 'apple, banana, cherry'
```
#### Loop Control: EXIT and CONTINUE
```sql
CREATE FUNCTION loop_control_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT := '';
BEGIN
<>
FOR i IN 1..5 LOOP
FOR j IN 1..5 LOOP
-- Skip when j = 3
CONTINUE WHEN j = 3;
-- Exit outer loop when i * j > 12
EXIT outer_loop WHEN i * j > 12;
result := result || '(' || i || ',' || j || ') ';
END LOOP;
END LOOP;
RETURN result;
END;
$$;
```
### 18.6 Exception Handling
#### RAISE Statement
RAISE generates messages or exceptions:
```sql
CREATE FUNCTION raise_examples(level TEXT)
RETURNS VOID
LANGUAGE plpgsql
AS $$
BEGIN
-- Different message levels
RAISE DEBUG 'Debug message: %', level;
RAISE LOG 'Log message: %', level;
RAISE INFO 'Info message: %', level;
RAISE NOTICE 'Notice message: %', level;
RAISE WARNING 'Warning message: %', level;
-- EXCEPTION level raises an error and aborts
IF level = 'error' THEN
RAISE EXCEPTION 'This is an error: %', level;
END IF;
END;
$$;
```
#### RAISE with USING Clause
```sql
CREATE FUNCTION raise_with_options(value INTEGER)
RETURNS VOID
LANGUAGE plpgsql
AS $$
BEGIN
IF value < 0 THEN
RAISE EXCEPTION 'Value must be positive'
USING ERRCODE = 'P0001',
HINT = 'Provide a value greater than or equal to zero',
DETAIL = 'Received value: ' || value;
END IF;
END;
$$;
```
#### RAISE with Condition Name
```sql
CREATE FUNCTION raise_condition_example(divisor INTEGER)
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
BEGIN
IF divisor = 0 THEN
RAISE division_by_zero;
END IF;
RETURN 100 / divisor;
END;
$$;
```
#### EXCEPTION Blocks
```sql
CREATE FUNCTION exception_example(a INTEGER, b INTEGER)
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
result INTEGER;
BEGIN
result := a / b;
RETURN result;
EXCEPTION
WHEN division_by_zero THEN
RAISE NOTICE 'Division by zero, returning NULL';
RETURN NULL;
WHEN OTHERS THEN
RAISE NOTICE 'Unknown error: %', SQLERRM;
RETURN -1;
END;
$$;
SELECT exception_example(10, 0); -- Returns NULL with notice
SELECT exception_example(10, 2); -- Returns 5
```
#### Multiple Exception Handlers
```sql
CREATE FUNCTION multi_exception_example(input TEXT)
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
result INTEGER;
BEGIN
result := input::INTEGER;
RETURN result;
EXCEPTION
WHEN invalid_text_representation THEN
RAISE NOTICE 'Cannot convert "%" to integer', input;
RETURN 0;
WHEN numeric_value_out_of_range THEN
RAISE NOTICE 'Value "%" is out of range', input;
RETURN -1;
WHEN OTHERS THEN
RAISE NOTICE 'Error: % (SQLSTATE: %)', SQLERRM, SQLSTATE;
RETURN -999;
END;
$$;
```
#### Class-Based Exception Handling
Exception conditions can be caught by class (first two characters of SQLSTATE):
```sql
CREATE FUNCTION class_exception_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
BEGIN
-- This will raise a data exception (class 22)
PERFORM 1/0;
RETURN 'OK';
EXCEPTION
-- data_exception catches all 22xxx errors including division_by_zero
WHEN data_exception THEN
RETURN 'Caught data exception: ' || SQLERRM;
-- integrity_constraint_violation catches all 23xxx errors
WHEN integrity_constraint_violation THEN
RETURN 'Caught constraint violation: ' || SQLERRM;
END;
$$;
```
#### Re-raising Exceptions
```sql
CREATE FUNCTION reraise_example()
RETURNS TEXT
LANGUAGE plpgsql
AS $$
BEGIN
BEGIN
RAISE EXCEPTION 'Original error';
EXCEPTION
WHEN OTHERS THEN
RAISE NOTICE 'Logging error before re-raise';
RAISE; -- Re-raise the current exception
END;
RETURN 'Never reached';
END;
$$;
```
#### Common SQLSTATE Conditions
| Condition Name | SQLSTATE | Description |
|----------------|----------|-------------|
| `division_by_zero` | 22012 | Division by zero |
| `null_value_not_allowed` | 22004 | NULL value not allowed |
| `numeric_value_out_of_range` | 22003 | Numeric value out of range |
| `invalid_text_representation` | 22P02 | Invalid text representation |
| `unique_violation` | 23505 | Unique constraint violation |
| `not_null_violation` | 23502 | NOT NULL constraint violation |
| `foreign_key_violation` | 23503 | Foreign key violation |
| `check_violation` | 23514 | CHECK constraint violation |
| `no_data_found` | P0002 | No data found |
| `too_many_rows` | P0003 | Too many rows |
| `raise_exception` | P0001 | User-raised exception |
| `assert_failure` | P0004 | ASSERT failure |
### 18.7 Returning Data
#### Simple RETURN
```sql
CREATE FUNCTION simple_return(a INTEGER, b INTEGER)
RETURNS INTEGER
LANGUAGE plpgsql
AS $$
BEGIN
RETURN a + b;
END;
$$;
```
#### RETURN QUERY
Return results of a query:
```sql
CREATE FUNCTION get_active_users()
RETURNS SETOF users
LANGUAGE plpgsql
AS $$
BEGIN
RETURN QUERY SELECT * FROM users WHERE active = TRUE;
END;
$$;
-- With TABLE return type
CREATE FUNCTION get_user_stats()
RETURNS TABLE(user_count BIGINT, active_count BIGINT, avg_age NUMERIC)
LANGUAGE plpgsql
AS $$
BEGIN
RETURN QUERY
SELECT
COUNT(*),
COUNT(*) FILTER (WHERE active),
AVG(age)::NUMERIC
FROM users;
END;
$$;
```
#### RETURN NEXT
Return rows one at a time:
```sql
CREATE FUNCTION fibonacci(n INTEGER)
RETURNS SETOF INTEGER
LANGUAGE plpgsql
AS $$
DECLARE
a INTEGER := 0;
b INTEGER := 1;
temp INTEGER;
BEGIN
FOR i IN 1..n LOOP
RETURN NEXT a;
temp := a;
a := b;
b := temp + b;
END LOOP;
END;
$$;
SELECT * FROM fibonacci(10);
-- Returns: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34
```
#### OUT Parameters
```sql
CREATE FUNCTION get_stats(
IN table_name TEXT,
OUT row_count BIGINT,
OUT min_id BIGINT,
OUT max_id BIGINT
)
LANGUAGE plpgsql
AS $$
BEGIN
EXECUTE 'SELECT COUNT(*), MIN(id), MAX(id) FROM ' || quote_ident(table_name)
INTO row_count, min_id, max_id;
END;
$$;
SELECT * FROM get_stats('users');
-- Returns: row_count, min_id, max_id as columns
```
#### INOUT Parameters
```sql
CREATE FUNCTION double_value(INOUT val INTEGER)
LANGUAGE plpgsql
AS $$
BEGIN
val := val * 2;
END;
$$;
SELECT double_value(5); -- Returns 10
```
### 18.8 Dynamic SQL with EXECUTE
#### Basic EXECUTE
```sql
CREATE FUNCTION dynamic_query(table_name TEXT, id_value INTEGER)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT;
BEGIN
EXECUTE 'SELECT name FROM ' || quote_ident(table_name) ||
' WHERE id = ' || quote_literal(id_value)
INTO result;
RETURN result;
END;
$$;
```
#### EXECUTE with USING Clause
Safer and more efficient way to pass parameters:
```sql
CREATE FUNCTION safe_dynamic_query(table_name TEXT, search_term TEXT)
RETURNS SETOF RECORD
LANGUAGE plpgsql
AS $$
BEGIN
RETURN QUERY EXECUTE
'SELECT * FROM ' || quote_ident(table_name) ||
' WHERE name ILIKE $1'
USING '%' || search_term || '%';
END;
$$;
```
#### EXECUTE INTO
```sql
CREATE FUNCTION get_column_value(
table_name TEXT,
column_name TEXT,
id_value INTEGER
)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT;
BEGIN
EXECUTE format('SELECT %I FROM %I WHERE id = $1', column_name, table_name)
INTO result
USING id_value;
RETURN result;
END;
$$;
```
#### PERFORM for Side Effects
Use PERFORM when you do not need the query result:
```sql
CREATE FUNCTION log_action(action TEXT)
RETURNS VOID
LANGUAGE plpgsql
AS $$
BEGIN
-- PERFORM discards the result
PERFORM pg_notify('actions', action);
-- This would cause an error without INTO or PERFORM:
-- SELECT pg_notify('actions', action);
END;
$$;
```
### 18.9 Trigger Functions in PL/pgSQL
#### Creating a Trigger Function
Trigger functions return TRIGGER type and access special variables:
```sql
CREATE FUNCTION audit_trigger()
RETURNS TRIGGER
LANGUAGE plpgsql
AS $$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO audit_log (table_name, operation, new_data, changed_at)
VALUES (TG_TABLE_NAME, 'INSERT', row_to_json(NEW), NOW());
RETURN NEW;
ELSIF TG_OP = 'UPDATE' THEN
INSERT INTO audit_log (table_name, operation, old_data, new_data, changed_at)
VALUES (TG_TABLE_NAME, 'UPDATE', row_to_json(OLD), row_to_json(NEW), NOW());
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
INSERT INTO audit_log (table_name, operation, old_data, changed_at)
VALUES (TG_TABLE_NAME, 'DELETE', row_to_json(OLD), NOW());
RETURN OLD;
END IF;
RETURN NULL;
END;
$$;
-- Attach to table
CREATE TRIGGER users_audit_trigger
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW EXECUTE FUNCTION audit_trigger();
```
#### Trigger Special Variables
| Variable | Description |
|----------|-------------|
| `NEW` | New row for INSERT/UPDATE (NULL for DELETE) |
| `OLD` | Old row for UPDATE/DELETE (NULL for INSERT) |
| `TG_NAME` | Trigger name |
| `TG_WHEN` | BEFORE, AFTER, or INSTEAD OF |
| `TG_LEVEL` | ROW or STATEMENT |
| `TG_OP` | INSERT, UPDATE, DELETE, or TRUNCATE |
| `TG_TABLE_NAME` | Table name |
| `TG_TABLE_SCHEMA` | Schema name |
| `TG_NARGS` | Number of trigger arguments |
| `TG_ARGV` | Array of trigger arguments |
#### BEFORE Trigger (Modifying Data)
```sql
CREATE FUNCTION normalize_email()
RETURNS TRIGGER
LANGUAGE plpgsql
AS $$
BEGIN
-- Normalize email to lowercase
NEW.email := LOWER(TRIM(NEW.email));
-- Set timestamps
IF TG_OP = 'INSERT' THEN
NEW.created_at := NOW();
END IF;
NEW.updated_at := NOW();
RETURN NEW;
END;
$$;
CREATE TRIGGER users_normalize
BEFORE INSERT OR UPDATE ON users
FOR EACH ROW EXECUTE FUNCTION normalize_email();
```
#### Preventing Operations
```sql
CREATE FUNCTION prevent_delete()
RETURNS TRIGGER
LANGUAGE plpgsql
AS $$
BEGIN
IF OLD.protected = TRUE THEN
RAISE EXCEPTION 'Cannot delete protected records';
END IF;
RETURN OLD;
END;
$$;
```
### 18.10 VARIADIC Parameters
VARIADIC allows functions to accept a variable number of arguments:
```sql
CREATE FUNCTION concat_all(VARIADIC items TEXT[])
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT := '';
item TEXT;
BEGIN
FOREACH item IN ARRAY items LOOP
result := result || item;
END LOOP;
RETURN result;
END;
$$;
SELECT concat_all('Hello', ' ', 'World', '!');
-- Returns: 'Hello World!'
```
#### VARIADIC with Regular Parameters
```sql
CREATE FUNCTION format_message(prefix TEXT, VARIADIC parts TEXT[])
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT;
BEGIN
result := prefix || ': ' || array_to_string(parts, ', ');
RETURN result;
END;
$$;
SELECT format_message('Items', 'apple', 'banana', 'cherry');
-- Returns: 'Items: apple, banana, cherry'
```
#### Calling with VARIADIC Keyword
When passing an existing array:
```sql
SELECT concat_all(VARIADIC ARRAY['a', 'b', 'c']);
-- Returns: 'abc'
```
### 18.11 Polymorphic Types
Polymorphic types allow functions to work with multiple data types while maintaining type consistency.
#### anyelement
```sql
CREATE FUNCTION identity(val anyelement)
RETURNS anyelement
LANGUAGE plpgsql
AS $$
BEGIN
RETURN val;
END;
$$;
SELECT identity(42); -- Returns 42 (integer)
SELECT identity('hello'); -- Returns 'hello' (text)
SELECT identity(3.14); -- Returns 3.14 (numeric)
```
#### anyarray
```sql
CREATE FUNCTION array_first(arr anyarray)
RETURNS anyelement
LANGUAGE plpgsql
AS $$
BEGIN
RETURN arr[1];
END;
$$;
SELECT array_first(ARRAY[10, 20, 30]); -- Returns 10
SELECT array_first(ARRAY['a', 'b', 'c']); -- Returns 'a'
```
#### Type Consistency
When multiple parameters use polymorphic types, they must resolve to the same actual type:
```sql
CREATE FUNCTION safe_add(a anyelement, b anyelement)
RETURNS anyelement
LANGUAGE plpgsql
AS $$
BEGIN
RETURN a + b;
END;
$$;
SELECT safe_add(10, 20); -- Returns 30 (both integers)
SELECT safe_add(1.5, 2.5); -- Returns 4.0 (both numeric)
-- SELECT safe_add(10, 'x'); -- Error: types must match
```
#### Supported Polymorphic Types
| Type | Description |
|------|-------------|
| `anyelement` | Matches any single data type |
| `anyarray` | Matches any array type |
| `anynonarray` | Matches any non-array type |
| `anyenum` | Matches any enum type |
| `anyrange` | Matches any range type |
| `anycompatible` | Matches types with implicit conversions |
| `anycompatiblearray` | Array version of anycompatible |
### 18.12 Best Practices
#### 1. Use Explicit Types
```sql
-- Good: Explicit types
DECLARE
user_id INTEGER;
user_name TEXT;
BEGIN
...
END;
-- Better: Reference table types for consistency
DECLARE
user_id users.id%TYPE;
user_name users.name%TYPE;
BEGIN
...
END;
```
#### 2. Handle NULL Values
```sql
CREATE FUNCTION safe_divide(a NUMERIC, b NUMERIC)
RETURNS NUMERIC
LANGUAGE plpgsql
AS $$
BEGIN
IF b IS NULL OR b = 0 THEN
RETURN NULL;
END IF;
RETURN a / b;
END;
$$;
```
#### 3. Use Exception Handling for Robustness
```sql
CREATE FUNCTION robust_lookup(search_id INTEGER)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
result TEXT;
BEGIN
SELECT name INTO STRICT result FROM users WHERE id = search_id;
RETURN result;
EXCEPTION
WHEN no_data_found THEN
RETURN NULL;
WHEN too_many_rows THEN
RAISE EXCEPTION 'Multiple users found for id %', search_id;
END;
$$;
```
#### 4. Use PERFORM for Queries Without Results
```sql
-- Wrong: Causes "query has no destination" error
CREATE FUNCTION bad_example()
RETURNS VOID
LANGUAGE plpgsql
AS $$
BEGIN
SELECT 1 + 1; -- Error!
END;
$$;
-- Correct: Use PERFORM
CREATE FUNCTION good_example()
RETURNS VOID
LANGUAGE plpgsql
AS $$
BEGIN
PERFORM pg_sleep(1); -- Discard result
END;
$$;
```
#### 5. Use quote_ident and quote_literal for Dynamic SQL
```sql
CREATE FUNCTION safe_dynamic_insert(
table_name TEXT,
column_name TEXT,
value TEXT
)
RETURNS VOID
LANGUAGE plpgsql
AS $$
BEGIN
EXECUTE format(
'INSERT INTO %I (%I) VALUES (%L)',
table_name,
column_name,
value
);
END;
$$;
```
#### 6. Keep Functions Focused
```sql
-- Good: Single responsibility
CREATE FUNCTION validate_email(email TEXT)
RETURNS BOOLEAN
LANGUAGE plpgsql
AS $$
BEGIN
RETURN email ~ '^[^@]+@[^@]+\.[^@]+$';
END;
$$;
CREATE FUNCTION normalize_email(email TEXT)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
BEGIN
RETURN LOWER(TRIM(email));
END;
$$;
-- Use them together
CREATE FUNCTION process_email(email TEXT)
RETURNS TEXT
LANGUAGE plpgsql
AS $$
DECLARE
normalized TEXT;
BEGIN
normalized := normalize_email(email);
IF NOT validate_email(normalized) THEN
RAISE EXCEPTION 'Invalid email: %', email;
END IF;
RETURN normalized;
END;
$$;
```
#### 7. Document Complex Functions
```sql
-- Function calculates the compound annual growth rate (CAGR)
-- Parameters:
-- start_value: Initial investment value
-- end_value: Final investment value
-- years: Number of years
-- Returns: CAGR as a decimal (e.g., 0.15 for 15%)
CREATE FUNCTION calculate_cagr(
start_value NUMERIC,
end_value NUMERIC,
years INTEGER
)
RETURNS NUMERIC
LANGUAGE plpgsql
AS $$
BEGIN
IF start_value <= 0 OR end_value <= 0 OR years <= 0 THEN
RETURN NULL;
END IF;
RETURN POWER(end_value / start_value, 1.0 / years) - 1;
END;
$$;
```
---
## Chapter 19: PL/Python - User-Defined Functions and Stored Procedures
PL/Python is Cognica's procedural language extension that allows you to write user-defined functions (UDFs) and stored procedures in Python. This chapter provides a comprehensive guide to creating, managing, and using Python functions within your SQL queries.
### 19.1 Understanding PL/Python
#### What is PL/Python?
PL/Python brings the power of Python programming directly into your database. Instead of being limited to SQL's declarative syntax, you can write procedural code that performs complex calculations, string manipulations, data transformations, and more.
```sql
-- A simple PL/Python function
CREATE FUNCTION greet(name TEXT)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
return f"Hello, {name}!"
$$;
SELECT greet('World');
-- Result: Hello, World!
```
#### Why Use PL/Python?
**Complex Business Logic**: When SQL becomes unwieldy for complex calculations:
```sql
-- Calculate compound interest with Python precision
CREATE FUNCTION compound_interest(
principal NUMERIC,
rate NUMERIC,
years INTEGER,
compounds_per_year INTEGER
)
RETURNS NUMERIC
LANGUAGE plpython3u
AS $$
from decimal import Decimal, ROUND_HALF_UP
p = Decimal(str(principal))
r = Decimal(str(rate))
n = Decimal(str(compounds_per_year))
t = Decimal(str(years))
# A = P(1 + r/n)^(nt)
amount = p * (1 + r/n) ** (n * t)
return amount.quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)
$$;
SELECT compound_interest(10000, 0.05, 10, 12);
-- Result: 16470.09
```
**String Processing**: Python excels at text manipulation:
```sql
-- Extract email domain with validation
CREATE FUNCTION extract_email_domain(email TEXT)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
import re
if email is None:
return None
# Validate email format
pattern = r'^[a-zA-Z0-9._%+-]+@([a-zA-Z0-9.-]+\.[a-zA-Z]{2,})$'
match = re.match(pattern, email)
if match:
return match.group(1).lower()
return None
$$;
SELECT email, extract_email_domain(email) AS domain
FROM users;
```
**Data Transformation**: Transform complex data structures:
```sql
-- Flatten nested JSON structure
CREATE FUNCTION flatten_address(address_json JSON)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
if address_json is None:
return None
parts = []
if 'street' in address_json:
parts.append(address_json['street'])
if 'city' in address_json:
parts.append(address_json['city'])
if 'state' in address_json:
parts.append(address_json['state'])
if 'zip' in address_json:
parts.append(str(address_json['zip']))
return ', '.join(parts)
$$;
SELECT flatten_address('{"street": "123 Main St", "city": "Boston", "state": "MA", "zip": "02101"}');
-- Result: 123 Main St, Boston, MA, 02101
```
#### The Language Name: plpython3u
The `plpython3u` language name has specific meaning:
- **plpython**: Procedural Language Python
- **3**: Python 3 (required, Python 2 is not supported)
- **u**: "Untrusted" - has access to database operations
Cognica also accepts `plpythonu` and `python` as aliases for compatibility.
### 19.2 Creating Functions
#### Basic Syntax
```sql
CREATE [OR REPLACE] FUNCTION function_name(
[parameter_name parameter_type [, ...]]
)
RETURNS return_type
LANGUAGE plpython3u
[IMMUTABLE | STABLE | VOLATILE]
[STRICT]
AS $$
# Python code here
return result
$$;
```
#### Components Explained
**Function Name and Parameters**:
```sql
-- Function with multiple parameters
CREATE FUNCTION calculate_discount(
price NUMERIC,
discount_percent NUMERIC,
apply_tax BOOLEAN
)
RETURNS NUMERIC
LANGUAGE plpython3u
AS $$
discounted = float(price) * (1 - float(discount_percent) / 100)
if apply_tax:
discounted *= 1.08 # 8% tax
return round(discounted, 2)
$$;
```
**Return Types**:
```sql
-- Return a single value
CREATE FUNCTION get_pi()
RETURNS DOUBLE PRECISION
LANGUAGE plpython3u
AS $$ return 3.14159265358979 $$;
-- Return NULL explicitly
CREATE FUNCTION maybe_null(val INTEGER)
RETURNS INTEGER
LANGUAGE plpython3u
AS $$
if val < 0:
return None # Python None becomes SQL NULL
return val * 2
$$;
```
**Volatility Classification**:
The volatility setting tells the query optimizer how to treat function calls:
```sql
-- IMMUTABLE: Always returns the same result for the same inputs
-- Safe for indexing, can be cached aggressively
CREATE FUNCTION square(x INTEGER)
RETURNS INTEGER
LANGUAGE plpython3u
IMMUTABLE
AS $$ return x * x $$;
-- STABLE: Returns the same result within a single query
-- Can use current settings, user info, etc.
CREATE FUNCTION current_user_greeting()
RETURNS TEXT
LANGUAGE plpython3u
STABLE
AS $$
import plpy
result = plpy.execute("SELECT current_user AS username")
return f"Welcome, {result[0]['username']}!"
$$;
-- VOLATILE (default): May return different results each time
-- Required for functions that modify data or call random()
CREATE FUNCTION generate_code()
RETURNS TEXT
LANGUAGE plpython3u
VOLATILE
AS $$
import random
import string
return ''.join(random.choices(string.ascii_uppercase + string.digits, k=8))
$$;
```
**STRICT Functions**:
STRICT functions automatically return NULL if any argument is NULL:
```sql
-- Without STRICT: Must handle NULL manually
CREATE FUNCTION safe_divide(a NUMERIC, b NUMERIC)
RETURNS NUMERIC
LANGUAGE plpython3u
AS $$
if a is None or b is None:
return None
if b == 0:
return None
return float(a) / float(b)
$$;
-- With STRICT: NULL handling is automatic
CREATE FUNCTION safe_divide_strict(a NUMERIC, b NUMERIC)
RETURNS NUMERIC
LANGUAGE plpython3u
STRICT
AS $$
if b == 0:
return None
return float(a) / float(b)
$$;
```
#### CREATE OR REPLACE
Use `CREATE OR REPLACE` to update an existing function without dropping it first:
```sql
-- Initial version
CREATE FUNCTION format_phone(phone TEXT)
RETURNS TEXT
LANGUAGE plpython3u
AS $$ return phone $$;
-- Updated version with formatting
CREATE OR REPLACE FUNCTION format_phone(phone TEXT)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
import re
digits = re.sub(r'\D', '', phone or '')
if len(digits) == 10:
return f"({digits[:3]}) {digits[3:6]}-{digits[6:]}"
return phone
$$;
SELECT format_phone('5551234567');
-- Result: (555) 123-4567
```
#### Dollar Quoting
PL/Python functions use dollar quoting (`$$`) to avoid escaping issues:
```sql
-- Simple dollar quoting
CREATE FUNCTION simple()
RETURNS TEXT
LANGUAGE plpython3u
AS $$
return "Hello, World!"
$$;
-- Tagged dollar quoting for nested quotes
CREATE FUNCTION with_sql()
RETURNS TEXT
LANGUAGE plpython3u
AS $BODY$
import plpy
result = plpy.execute("SELECT 'Hello' AS msg")
return result[0]['msg']
$BODY$;
```
### 19.3 Creating Stored Procedures
Stored procedures are called with `CALL` and can perform operations without returning a value:
```sql
-- Create a procedure
CREATE PROCEDURE log_event(
event_type TEXT,
event_data JSON
)
LANGUAGE plpython3u
AS $$
import plpy
from datetime import datetime
plpy.execute(f"""
INSERT INTO event_log (event_type, event_data, created_at)
VALUES (
{plpy.quote_literal(event_type)},
{plpy.quote_literal(str(event_data))},
NOW()
)
""")
plpy.notice(f"Logged event: {event_type}")
$$;
-- Call the procedure
CALL log_event('user_login', '{"user_id": 123, "ip": "192.168.1.1"}');
```
### 19.4 Type Mapping Between SQL and Python
Understanding type conversion is essential for writing correct PL/Python functions.
#### SQL to Python Conversion
| SQL Type | Python Type | Notes |
|----------|-------------|-------|
| INTEGER, SMALLINT | int | 32-bit signed integer |
| BIGINT | int | 64-bit signed integer |
| REAL | float | 32-bit floating point |
| DOUBLE PRECISION | float | 64-bit floating point |
| NUMERIC, DECIMAL | Decimal (when using decimal module) or float | Use decimal module for precision |
| BOOLEAN | bool | True/False |
| TEXT, VARCHAR, CHAR | str | Unicode string |
| JSON, JSONB | dict or list | Automatically parsed |
| NULL | None | Python None |
| ARRAY | list | Converted recursively |
```sql
-- Demonstration of type conversion
CREATE FUNCTION show_types(
int_val INTEGER,
float_val DOUBLE PRECISION,
text_val TEXT,
bool_val BOOLEAN,
json_val JSON
)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
types = [
f"int_val: {type(int_val).__name__} = {int_val}",
f"float_val: {type(float_val).__name__} = {float_val}",
f"text_val: {type(text_val).__name__} = {text_val}",
f"bool_val: {type(bool_val).__name__} = {bool_val}",
f"json_val: {type(json_val).__name__} = {json_val}",
]
return '\n'.join(types)
$$;
SELECT show_types(42, 3.14, 'hello', true, '{"key": "value"}');
-- Result:
-- int_val: int = 42
-- float_val: float = 3.14
-- text_val: str = hello
-- bool_val: bool = True
-- json_val: dict = {'key': 'value'}
```
#### Python to SQL Conversion
| Python Type | SQL Type | Notes |
|-------------|----------|-------|
| int | INTEGER or BIGINT | Based on value range |
| float | DOUBLE PRECISION | |
| Decimal | NUMERIC | Preserves precision |
| bool | BOOLEAN | |
| str | TEXT | |
| None | NULL | |
| dict | JSON/JSONB | Serialized to JSON |
| list | ARRAY or JSON | Depends on return type |
| tuple | ARRAY | Converted to array |
```sql
-- Returning different Python types
CREATE FUNCTION return_dict()
RETURNS JSON
LANGUAGE plpython3u
AS $$
return {
'name': 'Alice',
'age': 30,
'active': True,
'scores': [95, 87, 92]
}
$$;
SELECT return_dict();
-- Result: {"name": "Alice", "age": 30, "active": true, "scores": [95, 87, 92]}
```
#### Working with JSON
JSON types are automatically parsed to Python dict/list:
```sql
-- JSON input is automatically a Python dict
CREATE FUNCTION process_user(user_data JSON)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
# user_data is already a dict, no need to parse
name = user_data.get('name', 'Unknown')
email = user_data.get('email', 'N/A')
return f"{name} <{email}>"
$$;
SELECT process_user('{"name": "Bob", "email": "bob@example.com"}');
-- Result: Bob
-- Modifying and returning JSON
CREATE FUNCTION enrich_user(user_data JSON)
RETURNS JSON
LANGUAGE plpython3u
AS $$
from datetime import datetime
# Add processing timestamp
user_data['processed_at'] = datetime.now().isoformat()
user_data['source'] = 'api'
# Return modified dict (automatically converted to JSON)
return user_data
$$;
```
### 19.5 The plpy Module - Database Access
The `plpy` module provides PostgreSQL-compatible database access from within PL/Python functions.
#### Executing Queries with plpy.execute()
```sql
-- Basic query execution
CREATE FUNCTION count_users()
RETURNS INTEGER
LANGUAGE plpython3u
AS $$
import plpy
result = plpy.execute("SELECT COUNT(*) AS cnt FROM users")
return result[0]['cnt']
$$;
-- Query with dynamic values (use quote functions!)
CREATE FUNCTION find_user_by_email(email TEXT)
RETURNS JSON
LANGUAGE plpython3u
AS $$
import plpy
# Always use quote_literal for user input!
query = f"SELECT * FROM users WHERE email = {plpy.quote_literal(email)}"
result = plpy.execute(query)
if len(result) > 0:
return dict(result[0])
return None
$$;
```
#### Working with PLPyResult
The result from `plpy.execute()` is a PLPyResult object:
```sql
CREATE FUNCTION analyze_orders()
RETURNS TEXT
LANGUAGE plpython3u
AS $$
import plpy
result = plpy.execute("SELECT * FROM orders ORDER BY created_at DESC")
# Number of rows
row_count = result.nrows() # or len(result)
# Iterate over rows
total = 0
for row in result:
total += row['amount']
# Access by index (supports negative indexing)
first_order = result[0]
last_order = result[-1]
# Get column names
columns = result.colnames()
return f"Found {row_count} orders, total: ${total}"
$$;
```
#### Prepared Statements with plpy.prepare()
For repeated queries, use prepared statements for better performance:
```sql
CREATE FUNCTION get_orders_by_status(status TEXT)
RETURNS INTEGER
LANGUAGE plpython3u
AS $$
import plpy
# Prepare once, execute many times
plan = plpy.prepare(
"SELECT COUNT(*) AS cnt FROM orders WHERE status = $1",
["TEXT"]
)
result = plan.execute([status])
return result[0]['cnt']
$$;
-- More complex example with multiple parameters
CREATE FUNCTION search_products(
category TEXT,
min_price NUMERIC,
max_price NUMERIC
)
RETURNS JSON
LANGUAGE plpython3u
AS $$
import plpy
plan = plpy.prepare("""
SELECT id, name, price
FROM products
WHERE category = $1
AND price BETWEEN $2 AND $3
ORDER BY price
""", ["TEXT", "NUMERIC", "NUMERIC"])
result = plan.execute([category, min_price, max_price])
return [dict(row) for row in result]
$$;
```
#### Quote Functions for Safe SQL Construction
Always use quote functions when building dynamic SQL to prevent SQL injection:
```sql
CREATE FUNCTION safe_insert(table_name TEXT, col_name TEXT, value TEXT)
RETURNS VOID
LANGUAGE plpython3u
AS $$
import plpy
# quote_ident for identifiers (table/column names)
# quote_literal for values
# quote_nullable for values that might be NULL
safe_table = plpy.quote_ident(table_name)
safe_col = plpy.quote_ident(col_name)
safe_value = plpy.quote_nullable(value)
query = f"INSERT INTO {safe_table} ({safe_col}) VALUES ({safe_value})"
plpy.execute(query)
$$;
```
| Function | Purpose | Example |
|----------|---------|---------|
| `plpy.quote_literal(val)` | Quote a value as SQL literal | `'hello'` becomes `'hello'` |
| `plpy.quote_nullable(val)` | Like quote_literal, but None becomes NULL | `None` becomes `NULL` |
| `plpy.quote_ident(name)` | Quote an identifier if needed | `user name` becomes `"user name"` |
#### Logging Functions
```sql
CREATE FUNCTION process_with_logging(data JSON)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
import plpy
plpy.debug("Starting process_with_logging")
plpy.info(f"Processing data: {data}")
try:
# Processing logic here
result = data.get('value', 0) * 2
plpy.notice(f"Processing complete, result: {result}")
return f"Success: {result}"
except Exception as e:
plpy.warning(f"Processing failed: {e}")
plpy.error(f"Fatal error: {e}") # This raises an exception
$$;
```
| Function | Log Level | Notes |
|----------|-----------|-------|
| `plpy.debug(msg)` | DEBUG | For detailed debugging |
| `plpy.info(msg)` | INFO | Informational messages |
| `plpy.notice(msg)` | NOTICE | Important notices |
| `plpy.warning(msg)` | WARNING | Warning conditions |
| `plpy.log(msg)` | LOG | General logging |
| `plpy.error(msg)` | ERROR | Raises an exception |
| `plpy.fatal(msg)` | FATAL | Logs and raises exception |
#### Subtransactions
Use subtransactions for partial rollback on errors:
```sql
CREATE FUNCTION transfer_funds(from_account INTEGER, to_account INTEGER, amount NUMERIC)
RETURNS TEXT
LANGUAGE plpython3u
AS $$
import plpy
try:
with plpy.subtransaction():
# Debit from source account
plpy.execute(f"""
UPDATE accounts
SET balance = balance - {amount}
WHERE id = {from_account}
""")
# Check for negative balance
result = plpy.execute(f"""
SELECT balance FROM accounts WHERE id = {from_account}
""")
if result[0]['balance'] < 0:
raise Exception("Insufficient funds")
# Credit to destination account
plpy.execute(f"""
UPDATE accounts
SET balance = balance + {amount}
WHERE id = {to_account}
""")
return "Transfer successful"
except Exception as e:
# Subtransaction rolled back automatically
return f"Transfer failed: {e}"
$$;
```
### 19.6 Trigger Functions in PL/Python
PL/Python can be used to write trigger functions that respond to database events.
#### Creating a Trigger Function
Trigger functions access context through the `TD` dictionary:
```sql
-- Create a trigger function
CREATE FUNCTION audit_changes()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import plpy
import json
# TD dictionary contains trigger context
event = TD['event'] # INSERT, UPDATE, DELETE
when = TD['when'] # BEFORE, AFTER
level = TD['level'] # ROW, STATEMENT
table_name = TD['table_name']
old_row = TD['old'] # Previous values (UPDATE, DELETE)
new_row = TD['new'] # New values (INSERT, UPDATE)
if event == 'INSERT':
plpy.execute(f"""
INSERT INTO audit_log (table_name, action, new_data)
VALUES (
{plpy.quote_literal(table_name)},
'INSERT',
{plpy.quote_literal(json.dumps(new_row))}
)
""")
elif event == 'UPDATE':
plpy.execute(f"""
INSERT INTO audit_log (table_name, action, old_data, new_data)
VALUES (
{plpy.quote_literal(table_name)},
'UPDATE',
{plpy.quote_literal(json.dumps(old_row))},
{plpy.quote_literal(json.dumps(new_row))}
)
""")
elif event == 'DELETE':
plpy.execute(f"""
INSERT INTO audit_log (table_name, action, old_data)
VALUES (
{plpy.quote_literal(table_name)},
'DELETE',
{plpy.quote_literal(json.dumps(old_row))}
)
""")
return "OK"
$$;
-- Attach the trigger to a table
CREATE TRIGGER users_audit
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW
EXECUTE FUNCTION audit_changes();
```
#### TD Dictionary Reference
| Key | Description | Available When |
|-----|-------------|----------------|
| `TD['event']` | 'INSERT', 'UPDATE', 'DELETE', 'TRUNCATE' | Always |
| `TD['when']` | 'BEFORE', 'AFTER', 'INSTEAD OF' | Always |
| `TD['level']` | 'ROW', 'STATEMENT' | Always |
| `TD['table_name']` | Name of the table | Always |
| `TD['table_schema']` | Schema of the table | Always |
| `TD['trigger_name']` | Name of the trigger | Always |
| `TD['old']` | Old row values (dict) | UPDATE, DELETE |
| `TD['new']` | New row values (dict) | INSERT, UPDATE |
| `TD['args']` | Trigger arguments (list) | Always |
#### Trigger Return Values
For BEFORE triggers, the return value controls the operation:
```sql
CREATE FUNCTION validate_and_transform()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import plpy
# Only for BEFORE triggers
if TD['when'] != 'BEFORE':
return "OK"
if TD['event'] == 'INSERT' or TD['event'] == 'UPDATE':
new_row = TD['new']
# Validate email
email = new_row.get('email', '')
if not '@' in email:
plpy.error("Invalid email address")
# Transform: lowercase email
TD['new']['email'] = email.lower()
# Transform: auto-set updated_at
TD['new']['updated_at'] = None # Will use DEFAULT
# Return MODIFY to apply changes from TD['new']
return "MODIFY"
if TD['event'] == 'DELETE':
# Check if deletion is allowed
if TD['old'].get('protected', False):
return "SKIP" # Prevent the deletion
return "OK"
$$;
```
| Return Value | Effect |
|--------------|--------|
| `None` or `"OK"` | Continue normally |
| `"SKIP"` | Skip this row (BEFORE ROW only) |
| `"MODIFY"` | Apply changes from `TD['new']` (BEFORE only) |
### 19.7 Security Model
#### Allowed Python Modules
For security, PL/Python restricts which Python modules can be imported. The following modules are allowed:
**Mathematical Operations:**
- `math`, `cmath`, `decimal`, `fractions`, `statistics`, `random`, `numbers`
**Data Structures:**
- `collections`, `heapq`, `bisect`, `array`, `weakref`, `types`
**Functional Programming:**
- `itertools`, `functools`, `operator`
**Text Processing:**
- `string`, `re`, `textwrap`, `unicodedata`, `difflib`
**Date and Time:**
- `datetime`, `calendar`, `zoneinfo`
**Data Formats:**
- `json`, `csv`, `struct`, `base64`, `binascii`, `codecs`
**Hashing:**
- `hashlib`, `hmac`
**Identifiers:**
- `uuid`
**Type System:**
- `typing`, `dataclasses`, `enum`, `abc`
**Utilities:**
- `copy`, `pprint`, `reprlib`, `contextlib`
**Database Interface:**
- `plpy`
#### Blocked Modules
The following module categories are blocked:
- **Filesystem access:** `os`, `pathlib`, `shutil`, `glob`, `tempfile`
- **Network access:** `socket`, `http`, `urllib`, `ssl`
- **Process execution:** `subprocess`, `multiprocessing`, `threading`
- **System introspection:** `sys` (partially), `inspect`, `ctypes`
- **Code execution:** `importlib`, `marshal`, `shelve`
```sql
-- This will fail with ImportError
CREATE FUNCTION try_filesystem()
RETURNS TEXT
LANGUAGE plpython3u
AS $$
import os # ImportError: Module 'os' is not allowed in user-defined functions
return os.getcwd()
$$;
```
#### Resource Limits
PL/Python functions have built-in resource limits to prevent runaway execution:
| Resource | Default Limit | Description |
|----------|--------------|-------------|
| Execution time | 30 seconds | Maximum runtime |
| Memory | 64 MB | Maximum memory usage |
| Database queries | 1000 | Max queries via plpy.execute() |
| Recursion depth | 100 | Maximum call stack depth |
```sql
-- This will be terminated after 30 seconds
CREATE FUNCTION infinite_loop()
RETURNS INTEGER
LANGUAGE plpython3u
AS $$
while True:
pass
$$;
-- Error: execution timeout exceeded
SELECT infinite_loop();
```
### 19.8 Managing Functions
#### Viewing Functions
```sql
-- List all user-defined functions
SELECT * FROM _sys.functions;
-- Query specific function
SELECT
schema_name,
function_name,
return_type,
language,
volatility
FROM _sys.functions
WHERE function_name = 'my_function';
```
#### Dropping Functions
```sql
-- Drop a function (exact signature required)
DROP FUNCTION calculate_discount(NUMERIC, NUMERIC, BOOLEAN);
-- Drop if exists (no error if missing)
DROP FUNCTION IF EXISTS my_function(INTEGER);
-- Drop without specifying parameters (if name is unique)
DROP FUNCTION my_function;
```
### 19.9 Best Practices
#### 1. Always Use Quote Functions
Never concatenate user input directly into SQL strings:
```sql
-- BAD: SQL injection vulnerability
CREATE FUNCTION bad_search(term TEXT)
RETURNS JSON
LANGUAGE plpython3u
AS $$
import plpy
# DANGEROUS: Direct string concatenation
result = plpy.execute(f"SELECT * FROM products WHERE name = '{term}'")
return [dict(r) for r in result]
$$;
-- GOOD: Safe query construction
CREATE FUNCTION good_search(term TEXT)
RETURNS JSON
LANGUAGE plpython3u
AS $$
import plpy
# SAFE: Using quote_literal
result = plpy.execute(
f"SELECT * FROM products WHERE name = {plpy.quote_literal(term)}"
)
return [dict(r) for r in result]
$$;
```
#### 2. Handle NULL Values Appropriately
```sql
-- Explicit NULL handling
CREATE FUNCTION safe_process(val INTEGER)
RETURNS INTEGER
LANGUAGE plpython3u
AS $$
if val is None:
return 0 # Or return None if NULL should propagate
return val * 2
$$;
-- Or use STRICT for automatic NULL handling
CREATE FUNCTION safe_process_strict(val INTEGER)
RETURNS INTEGER
LANGUAGE plpython3u
STRICT
AS $$ return val * 2 $$;
```
#### 3. Use Appropriate Volatility
```sql
-- Pure calculation - IMMUTABLE
CREATE FUNCTION celsius_to_fahrenheit(c DOUBLE PRECISION)
RETURNS DOUBLE PRECISION
LANGUAGE plpython3u
IMMUTABLE
AS $$ return c * 9/5 + 32 $$;
-- Reads database - STABLE
CREATE FUNCTION get_user_preference(user_id INTEGER, key TEXT)
RETURNS TEXT
LANGUAGE plpython3u
STABLE
AS $$
import plpy
result = plpy.execute(f"""
SELECT value FROM user_preferences
WHERE user_id = {user_id} AND key = {plpy.quote_literal(key)}
""")
return result[0]['value'] if len(result) > 0 else None
$$;
-- Has side effects - VOLATILE (default)
CREATE FUNCTION log_access(resource TEXT)
RETURNS VOID
LANGUAGE plpython3u
VOLATILE
AS $$
import plpy
plpy.execute(f"""
INSERT INTO access_log (resource, accessed_at)
VALUES ({plpy.quote_literal(resource)}, NOW())
""")
$$;
```
#### 4. Use Prepared Statements for Repeated Queries
```sql
CREATE FUNCTION batch_process(ids INTEGER[])
RETURNS INTEGER
LANGUAGE plpython3u
AS $$
import plpy
# Prepare once
plan = plpy.prepare(
"UPDATE items SET processed = true WHERE id = $1",
["INTEGER"]
)
count = 0
for item_id in ids:
plan.execute([item_id])
count += 1
return count
$$;
```
#### 5. Keep Functions Focused
```sql
-- BAD: Function does too many things
CREATE FUNCTION do_everything(user_id INTEGER)
RETURNS JSON
LANGUAGE plpython3u
AS $$
# Validates user, updates profile, sends email, logs action...
# This is too complex and hard to test
$$;
-- GOOD: Separate concerns into focused functions
CREATE FUNCTION validate_user(user_id INTEGER) RETURNS BOOLEAN...
CREATE FUNCTION update_user_profile(user_id INTEGER, data JSON) RETURNS VOID...
CREATE FUNCTION log_user_action(user_id INTEGER, action TEXT) RETURNS VOID...
```
#### 6. Use Decimal for Financial Calculations
```sql
CREATE FUNCTION calculate_total(
subtotal NUMERIC,
tax_rate NUMERIC,
discount NUMERIC
)
RETURNS NUMERIC
LANGUAGE plpython3u
AS $$
from decimal import Decimal, ROUND_HALF_UP
# Convert to Decimal for precision
sub = Decimal(str(subtotal))
tax = Decimal(str(tax_rate))
disc = Decimal(str(discount))
# Calculate with exact decimal arithmetic
total = sub * (1 + tax) * (1 - disc)
# Round to cents
return total.quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)
$$;
```
### 19.10 Complete Examples
#### Example 1: Text Analysis Function
```sql
CREATE FUNCTION analyze_text(content TEXT)
RETURNS JSON
LANGUAGE plpython3u
AS $$
import re
from collections import Counter
if content is None:
return None
# Word count
words = re.findall(r'\b\w+\b', content.lower())
word_count = len(words)
# Character count (excluding spaces)
char_count = len(content.replace(' ', ''))
# Sentence count
sentences = re.split(r'[.!?]+', content)
sentence_count = len([s for s in sentences if s.strip()])
# Most common words
word_freq = Counter(words)
top_words = word_freq.most_common(5)
# Average word length
avg_word_len = sum(len(w) for w in words) / len(words) if words else 0
return {
'word_count': word_count,
'char_count': char_count,
'sentence_count': sentence_count,
'avg_word_length': round(avg_word_len, 2),
'top_words': [{'word': w, 'count': c} for w, c in top_words]
}
$$;
SELECT analyze_text('The quick brown fox jumps over the lazy dog. The dog was not amused.');
```
#### Example 2: Data Validation Function
```sql
CREATE FUNCTION validate_user_registration(
username TEXT,
email TEXT,
password TEXT,
age INTEGER
)
RETURNS JSON
LANGUAGE plpython3u
AS $$
import re
errors = []
warnings = []
# Username validation
if not username:
errors.append("Username is required")
elif len(username) < 3:
errors.append("Username must be at least 3 characters")
elif len(username) > 50:
errors.append("Username cannot exceed 50 characters")
elif not re.match(r'^[a-zA-Z0-9_]+$', username):
errors.append("Username can only contain letters, numbers, and underscores")
# Email validation
if not email:
errors.append("Email is required")
elif not re.match(r'^[^@]+@[^@]+\.[^@]+$', email):
errors.append("Invalid email format")
# Password validation
if not password:
errors.append("Password is required")
else:
if len(password) < 8:
errors.append("Password must be at least 8 characters")
if not re.search(r'[A-Z]', password):
warnings.append("Password should contain uppercase letters")
if not re.search(r'[0-9]', password):
warnings.append("Password should contain numbers")
if not re.search(r'[!@#$%^&*]', password):
warnings.append("Password should contain special characters")
# Age validation
if age is not None:
if age < 13:
errors.append("Users must be at least 13 years old")
elif age > 120:
errors.append("Please enter a valid age")
return {
'valid': len(errors) == 0,
'errors': errors,
'warnings': warnings
}
$$;
SELECT validate_user_registration('john_doe', 'john@example.com', 'SecurePass123!', 25);
```
#### Example 3: Audit Trigger Function
```sql
-- Create audit table
CREATE TABLE audit_trail (
id SERIAL PRIMARY KEY,
table_name TEXT NOT NULL,
operation TEXT NOT NULL,
old_values JSON,
new_values JSON,
changed_by TEXT,
changed_at TIMESTAMPTZ DEFAULT NOW()
);
-- Create comprehensive audit trigger function
CREATE FUNCTION comprehensive_audit()
RETURNS TRIGGER
LANGUAGE plpython3u
AS $$
import plpy
import json
event = TD['event']
table_name = TD['table_name']
old_values = None
new_values = None
if event in ('UPDATE', 'DELETE'):
old_values = json.dumps(TD['old'])
if event in ('INSERT', 'UPDATE'):
new_values = json.dumps(TD['new'])
# Get current user
user_result = plpy.execute("SELECT current_user AS username")
current_user = user_result[0]['username']
plpy.execute(f"""
INSERT INTO audit_trail (table_name, operation, old_values, new_values, changed_by)
VALUES (
{plpy.quote_literal(table_name)},
{plpy.quote_literal(event)},
{plpy.quote_nullable(old_values)},
{plpy.quote_nullable(new_values)},
{plpy.quote_literal(current_user)}
)
""")
plpy.notice(f"Audit: {event} on {table_name}")
return "OK"
$$;
-- Apply to tables
CREATE TRIGGER audit_users
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW EXECUTE FUNCTION comprehensive_audit();
CREATE TRIGGER audit_orders
AFTER INSERT OR UPDATE OR DELETE ON orders
FOR EACH ROW EXECUTE FUNCTION comprehensive_audit();
```
#### Example 4: Set-Returning Function
```sql
-- Function that generates date ranges
CREATE FUNCTION generate_date_series(
start_date DATE,
end_date DATE,
interval_days INTEGER
)
RETURNS SETOF JSON
LANGUAGE plpython3u
AS $$
from datetime import datetime, timedelta
if start_date is None or end_date is None:
return
# Convert to datetime
current = datetime.strptime(str(start_date), '%Y-%m-%d')
end = datetime.strptime(str(end_date), '%Y-%m-%d')
delta = timedelta(days=interval_days or 1)
results = []
while current <= end:
results.append({
'date': current.strftime('%Y-%m-%d'),
'day_of_week': current.strftime('%A'),
'week_number': current.isocalendar()[1],
'is_weekend': current.weekday() >= 5
})
current += delta
return results
$$;
SELECT * FROM generate_date_series('2024-01-01', '2024-01-07', 1);
```
---
## Chapter 20: Security - Protecting Your Data
Database security has multiple layers: authentication (who are you?), authorization (what can you do?), and row-level security (which specific rows can you access?).
### Security as Defense in Depth
A single security mechanism is a single point of failure. Effective database security combines multiple layers:
**Layer 1 - Network**: Limit who can connect at all. Use firewalls, private networks, and SSL/TLS encryption.
**Layer 2 - Authentication**: Verify identity through passwords, certificates, or external authentication systems.
**Layer 3 - Authorization**: Control what authenticated users can do at the database, schema, and table level.
**Layer 4 - Row-Level Security**: Restrict which rows users can see or modify, even within tables they can access.
**Layer 5 - Encryption**: Protect data at rest and in transit from unauthorized access even if other layers fail.
Each layer adds protection if another fails. A compromised password is less dangerous if the attacker cannot reach the database server. A misconfigured permission is less harmful if row-level policies restrict access.
### The Principle of Least Privilege
The most important security principle is simple: give users the minimum permissions they need to do their job, nothing more.
This means:
- Application users should not be superusers
- Read-only reporting should use roles that cannot write
- Each application should have its own credentials (not shared accounts)
- Permissions should be granted on specific tables, not entire schemas when possible
Overly permissive access is the root cause of most security incidents. When a vulnerability is exploited, the damage is limited to what the compromised account can access.
### 20.1 Roles and Users
In Cognica, users and roles are the same concept - a role that can log in is a user. The primary difference is that `CREATE USER` is equivalent to `CREATE ROLE ... LOGIN`.
#### CREATE ROLE Syntax
```sql
CREATE ROLE role_name
[ [ WITH ] option [ ... ] ]
-- Available options:
SUPERUSER | NOSUPERUSER
| INHERIT | NOINHERIT
| CREATEROLE | NOCREATEROLE
| CREATEDB | NOCREATEDB
| LOGIN | NOLOGIN
| REPLICATION | NOREPLICATION
| BYPASSRLS | NOBYPASSRLS
| CONNECTION LIMIT connlimit
| PASSWORD 'password' | PASSWORD NULL
| VALID UNTIL 'timestamp'
| IN ROLE role_name [, ...]
| ROLE role_name [, ...]
| ADMIN role_name [, ...]
```
#### Creating Roles
```sql
-- Create a role that can login (a user)
CREATE ROLE alice LOGIN PASSWORD 'secure_password';
-- Create a role that cannot login (a group)
CREATE ROLE analysts;
-- Create a superuser (has all permissions - use sparingly!)
CREATE ROLE admin LOGIN PASSWORD 'very_secure' SUPERUSER;
-- Role with specific attributes
CREATE ROLE app_user LOGIN PASSWORD 'pwd'
CONNECTION LIMIT 10 -- Max simultaneous connections
VALID UNTIL '2025-01-01'; -- Password expiration
-- Create a role that is member of other roles
CREATE ROLE junior_developer
IN ROLE developers, readers;
-- Create a role and immediately add members
CREATE ROLE team_leads
ROLE alice, bob, charlie;
-- Create a role with admin members (they can grant this role to others)
CREATE ROLE dba_team
ADMIN alice, bob;
```
#### CREATE USER Syntax
`CREATE USER` is equivalent to `CREATE ROLE ... LOGIN`:
```sql
-- Create a basic user with password
CREATE USER alice WITH PASSWORD 'secure_password_123';
-- Create a user with password expiration
CREATE USER contractor
PASSWORD 'temp_pass'
VALID UNTIL '2025-12-31 23:59:59';
-- Create a user with connection limit
CREATE USER api_service
PASSWORD 'service_password'
CONNECTION LIMIT 10;
-- Create a user that bypasses row-level security (for ETL)
CREATE USER etl_user
BYPASSRLS
PASSWORD 'etl_password';
```
#### ALTER ROLE
Modify the attributes of an existing role:
```sql
-- Change password
ALTER ROLE alice PASSWORD 'new_secure_password';
-- Remove password (trust authentication only)
ALTER ROLE alice PASSWORD NULL;
-- Set password expiration
ALTER ROLE alice VALID UNTIL '2025-06-30';
-- Remove password expiration
ALTER ROLE alice VALID UNTIL 'infinity';
-- Grant superuser privilege
ALTER ROLE alice SUPERUSER;
-- Revoke superuser privilege
ALTER ROLE alice NOSUPERUSER;
-- Enable RLS bypass
ALTER ROLE etl_user BYPASSRLS;
-- Change connection limit
ALTER ROLE api_service CONNECTION LIMIT 50;
-- Remove connection limit
ALTER ROLE api_service CONNECTION LIMIT -1;
-- Rename a role
ALTER ROLE old_name RENAME TO new_name;
-- Set default search_path for a role
ALTER ROLE developer SET search_path TO myschema, public;
```
#### DROP ROLE
Remove a database role:
```sql
-- Drop a single role
DROP ROLE analysts;
-- Drop multiple roles
DROP ROLE junior_developer, intern;
-- Drop role only if it exists (no error if not found)
DROP ROLE IF EXISTS temporary_role;
```
**Important:** Before dropping a role, you must:
1. Remove the role from all role memberships
2. Reassign or drop all objects owned by the role
3. Revoke all privileges granted to the role
#### Role Attributes Reference
| Attribute | Default | Description |
|-----------|---------|-------------|
| `SUPERUSER` | `NOSUPERUSER` | Bypasses all permission checks except login |
| `INHERIT` | `INHERIT` | Automatically inherits privileges from member roles |
| `CREATEROLE` | `NOCREATEROLE` | Can create, alter, and drop other roles |
| `CREATEDB` | `NOCREATEDB` | Can create new databases |
| `LOGIN` | `NOLOGIN` | Can establish database sessions |
| `REPLICATION` | `NOREPLICATION` | Can initiate streaming replication |
| `BYPASSRLS` | `NOBYPASSRLS` | Bypasses row-level security policies |
| `CONNECTION LIMIT` | `-1` (unlimited) | Maximum concurrent connections |
| `PASSWORD` | `NULL` | Authentication password (SCRAM-SHA-256) |
| `VALID UNTIL` | `NULL` (never expires) | Password expiration timestamp |
#### Role Membership and Inheritance
Role membership allows organizing roles into hierarchies where child roles can inherit privileges from parent roles:
```sql
-- Create group roles
CREATE ROLE readonly;
CREATE ROLE readwrite;
-- Grant permissions to groups
GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO readwrite;
-- Make readwrite include readonly permissions
GRANT readonly TO readwrite;
-- Assign users to groups
GRANT readonly TO alice;
GRANT readwrite TO bob;
-- Alice can SELECT
-- Bob can SELECT, INSERT, UPDATE, DELETE (inherited readonly + own)
```
#### Membership Grant Options
When granting role membership, you can control how the membership behaves:
```sql
-- Basic membership grant
GRANT developers TO alice;
-- Grant multiple roles to multiple members
GRANT readers, writers TO alice, bob, charlie;
-- Grant with admin option (member can grant this role to others)
GRANT team_lead TO alice WITH ADMIN OPTION;
-- Grant without inheritance (member must SET ROLE to use privileges)
GRANT admin_role TO alice WITH INHERIT FALSE;
-- Grant without SET option (member cannot SET ROLE to this role)
GRANT restricted_role TO alice WITH SET FALSE;
```
**Membership Options:**
| Option | Default | Description |
|--------|---------|-------------|
| `ADMIN OPTION` | `FALSE` | Member can grant this role to other roles |
| `INHERIT OPTION` | `TRUE` | Member automatically inherits privileges |
| `SET OPTION` | `TRUE` | Member can use `SET ROLE` to become this role |
#### Revoking Role Membership
```sql
-- Basic membership revoke
REVOKE developers FROM alice;
-- Revoke only the admin option (keep membership)
REVOKE ADMIN OPTION FOR team_lead FROM alice;
-- Revoke with cascade (also revokes from roles that got it via alice)
REVOKE developers FROM alice CASCADE;
```
### 20.2 Authentication
#### Password Management
```sql
-- On user creation
CREATE USER alice PASSWORD 'secure_password';
-- Later modification
ALTER USER alice PASSWORD 'new_secure_password';
-- Remove password (trust authentication only)
ALTER USER alice PASSWORD NULL;
```
**Password requirements:**
- Passwords are stored using SCRAM-SHA-256 hashing
- Minimum password length depends on server configuration
- Passwords are case-sensitive
#### SCRAM-SHA-256
Cognica uses SCRAM-SHA-256 (Salted Challenge Response Authentication Mechanism) for password authentication, which is the PostgreSQL 14+ default.
**Benefits:**
- Password is never sent in plaintext
- Server stores only a hash, not the actual password
- Resistant to replay attacks
- Channel binding support for man-in-the-middle protection
#### Password Expiration
```sql
-- Expire at specific time
ALTER USER contractor VALID UNTIL '2025-12-31 23:59:59';
-- Expire immediately (forces password change on next login)
ALTER USER alice VALID UNTIL 'now';
-- Remove expiration
ALTER USER alice VALID UNTIL 'infinity';
```
**Checking expiration:**
```sql
SELECT rolname, rolvaliduntil
FROM pg_roles
WHERE rolvaliduntil IS NOT NULL
AND rolvaliduntil < CURRENT_TIMESTAMP + INTERVAL '30 days';
```
### 20.3 Session Management
Session management controls how users switch between roles during a database session.
#### SET ROLE
Changes the current role for the session:
```sql
-- Switch to another role (must be member of that role)
SET ROLE admin_role;
-- Check current role
SELECT current_role;
-- Switch back to session role
SET ROLE NONE;
-- or
RESET ROLE;
```
**Requirements:**
- You must be a member of the target role
- The membership must have `SET OPTION` (default is TRUE)
```sql
-- alice is member of developers
GRANT developers TO alice;
-- alice can switch
SET ROLE developers; -- Works
-- alice is member of restricted without SET option
GRANT restricted TO alice WITH SET FALSE;
-- alice cannot switch
SET ROLE restricted; -- ERROR: permission denied
```
#### Session vs Current Role
| Function | Returns | Description |
|----------|---------|-------------|
| `session_user` | Original authenticated role | Never changes during session |
| `current_user` | Currently active role | Changes with SET ROLE |
| `current_role` | Same as current_user | Alias for current_user |
**Example:**
```sql
-- Login as alice
SELECT session_user, current_user, current_role;
-- alice, alice, alice
SET ROLE developers;
SELECT session_user, current_user, current_role;
-- alice, developers, developers
RESET ROLE;
SELECT session_user, current_user, current_role;
-- alice, alice, alice
```
**Use Cases:**
```sql
-- RLS policy using session_user (tracks who actually logged in)
CREATE POLICY audit_trail ON changes
FOR INSERT
WITH CHECK (created_by = session_user);
-- RLS policy using current_user (respects SET ROLE)
CREATE POLICY role_based ON documents
FOR SELECT
USING (allowed_role = current_role);
```
### 20.4 Privileges: Who Can Do What
#### Table Privileges
```sql
-- Grant specific privileges
GRANT SELECT ON products TO alice;
GRANT SELECT, INSERT, UPDATE ON orders TO bob;
GRANT ALL PRIVILEGES ON customers TO admin_role;
-- Grant to all users
GRANT SELECT ON public_data TO PUBLIC;
-- Grant on all tables in a schema
GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly_role;
-- Revoke privileges
REVOKE INSERT ON orders FROM bob;
REVOKE ALL PRIVILEGES ON customers FROM alice;
-- Revoke from PUBLIC
REVOKE SELECT ON internal_logs FROM PUBLIC;
-- Revoke on all tables in schema
REVOKE ALL ON ALL TABLES IN SCHEMA hr FROM contractors;
```
#### Privilege Types Reference
| Privilege | Applicable To | Allows |
|-----------|---------------|--------|
| `SELECT` | Tables, Views, Sequences | Read data from the object |
| `INSERT` | Tables | Insert new rows |
| `UPDATE` | Tables, Sequences | Modify existing rows / advance sequence |
| `DELETE` | Tables | Delete rows |
| `TRUNCATE` | Tables | Empty the table (faster than DELETE) |
| `REFERENCES` | Tables | Create foreign key references |
| `TRIGGER` | Tables | Create triggers on the table |
**Privilege Combinations for Common Operations:**
| Operation | Required Privileges |
|-----------|---------------------|
| `SELECT * FROM t` | SELECT on t |
| `SELECT a, b FROM t` | SELECT on t (or SELECT on columns a, b) |
| `INSERT INTO t VALUES (...)` | INSERT on t |
| `INSERT INTO t (a, b) VALUES (...)` | INSERT on t (or INSERT on columns a, b) |
| `UPDATE t SET a = 1` | UPDATE on t (or UPDATE on column a), SELECT if WHERE clause |
| `UPDATE t SET a = 1 WHERE b = 2` | UPDATE on column a, SELECT on column b |
| `DELETE FROM t` | DELETE on t |
| `DELETE FROM t WHERE a = 1` | DELETE on t, SELECT on column a |
| `TRUNCATE t` | TRUNCATE on t |
#### Column-Level Privileges
```sql
-- Grant access to specific columns only
GRANT SELECT (id, name, email) ON users TO support_team;
-- support_team can see id, name, email but NOT password_hash, salary, etc.
GRANT UPDATE (status) ON orders TO shipping_team;
-- shipping_team can only update the status column
-- Grant multiple privileges on different columns
GRANT SELECT (id, name), UPDATE (status) ON tasks TO workers;
-- Combination of table and column privileges
GRANT SELECT ON orders TO analysts; -- Full table access
GRANT SELECT (order_id, amount) ON orders TO limited_view; -- Partial access
-- Grant INSERT on specific columns (for tables with defaults)
GRANT INSERT (name, email) ON users TO registration_service;
```
**Column Privilege Behavior:**
1. **Table privilege grants access to all columns:**
```sql
GRANT SELECT ON users TO alice;
-- alice can: SELECT * FROM users;
-- alice can: SELECT id, name, email FROM users;
```
2. **Column privileges grant access to specific columns only:**
```sql
GRANT SELECT (id, name) ON users TO bob;
-- bob can: SELECT id, name FROM users;
-- bob CANNOT: SELECT * FROM users; (lacks email, etc.)
-- bob CANNOT: SELECT email FROM users;
```
3. **Column privileges are additive:**
```sql
GRANT SELECT (id) ON users TO charlie;
GRANT SELECT (name) ON users TO charlie;
-- charlie can: SELECT id, name FROM users;
```
4. **UPDATE requires SELECT for WHERE clause:**
```sql
GRANT UPDATE (status) ON orders TO worker;
-- worker can: UPDATE orders SET status = 'done';
-- worker CANNOT: UPDATE orders SET status = 'done' WHERE id = 1;
-- (lacks SELECT on id)
GRANT UPDATE (status), SELECT (id) ON orders TO worker;
-- worker can: UPDATE orders SET status = 'done' WHERE id = 1;
```
#### WITH GRANT OPTION
When `WITH GRANT OPTION` is specified, the grantee can grant the same privilege to others:
```sql
-- alice can now grant SELECT to other roles
GRANT SELECT ON reports TO alice WITH GRANT OPTION;
-- alice grants to bob
SET ROLE alice;
GRANT SELECT ON reports TO bob; -- This works
-- bob tries to grant to charlie (fails - bob doesn't have grant option)
SET ROLE bob;
GRANT SELECT ON reports TO charlie; -- ERROR: permission denied
```
**Revoking Grant Option:**
```sql
-- Remove only the grant option, alice keeps SELECT
REVOKE GRANT OPTION FOR SELECT ON reports FROM alice;
-- Remove privilege entirely with cascade
-- (also removes from roles alice granted to)
REVOKE SELECT ON reports FROM alice CASCADE;
```
#### Schema Privileges
```sql
-- Create a schema for sensitive data
CREATE SCHEMA finance;
-- Restrict access to the schema
GRANT USAGE ON SCHEMA finance TO finance_team;
GRANT ALL ON ALL TABLES IN SCHEMA finance TO finance_team;
-- Others can't even see tables in this schema
```
#### Default Privileges
Default privileges automatically apply to objects created in the future:
```sql
-- All future tables created by current user get SELECT for analysts
ALTER DEFAULT PRIVILEGES
GRANT SELECT ON TABLES TO analysts;
-- All future tables in schema 'reports' get SELECT for reporting_role
ALTER DEFAULT PRIVILEGES IN SCHEMA reports
GRANT SELECT ON TABLES TO reporting_role;
-- All future tables created by alice in public schema
ALTER DEFAULT PRIVILEGES FOR ROLE alice IN SCHEMA public
GRANT SELECT, INSERT ON TABLES TO developers;
-- Remove default privilege
ALTER DEFAULT PRIVILEGES IN SCHEMA public
REVOKE SELECT ON TABLES FROM PUBLIC;
-- Setup for application: app_owner creates tables, app_user accesses them
ALTER DEFAULT PRIVILEGES FOR ROLE app_owner
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user;
```
**Schema-Specific Defaults:**
```sql
-- Global default (all schemas)
ALTER DEFAULT PRIVILEGES
GRANT SELECT ON TABLES TO global_reader;
-- Schema-specific (only 'analytics' schema)
ALTER DEFAULT PRIVILEGES IN SCHEMA analytics
GRANT SELECT ON TABLES TO analytics_team;
-- The analytics_team gets SELECT on:
-- - Tables created in 'analytics' schema (from schema-specific default)
-- They DO NOT automatically get SELECT on tables in other schemas
```
### 20.5 Row-Level Security (RLS)
RLS lets you control which rows each user can see or modify. This is powerful for multi-tenant applications. When enabled, queries automatically filter rows based on policies.
#### Enabling RLS
```sql
-- Enable RLS on a table
ALTER TABLE documents ENABLE ROW LEVEL SECURITY;
-- Disable RLS on a table
ALTER TABLE documents DISABLE ROW LEVEL SECURITY;
-- Force RLS even for table owners
ALTER TABLE documents FORCE ROW LEVEL SECURITY;
-- Don't force RLS for table owner
ALTER TABLE documents NO FORCE ROW LEVEL SECURITY;
```
**Important:** After enabling RLS, if no policies are defined, no rows are visible to non-superusers (except the table owner).
#### CREATE POLICY Syntax
```sql
CREATE POLICY name ON table_name
[ AS { PERMISSIVE | RESTRICTIVE } ]
[ FOR { ALL | SELECT | INSERT | UPDATE | DELETE } ]
[ TO { role_name | PUBLIC | CURRENT_ROLE | CURRENT_USER | SESSION_USER } [, ...] ]
[ USING ( using_expression ) ]
[ WITH CHECK ( check_expression ) ]
```
#### Creating Policies
```sql
-- Users can only see their own documents
CREATE POLICY user_documents ON documents
FOR SELECT
USING (owner_id = current_user_id());
-- Users can only modify their own documents
CREATE POLICY user_modify_documents ON documents
FOR ALL -- SELECT, INSERT, UPDATE, DELETE
USING (owner_id = current_user_id()) -- For existing rows
WITH CHECK (owner_id = current_user_id()); -- For new/modified rows
-- Users can only insert records for themselves
CREATE POLICY insert_own ON user_data
FOR INSERT
TO PUBLIC
WITH CHECK (user_id = current_user_id());
-- Users can update their own non-locked records
CREATE POLICY update_own ON documents
FOR UPDATE
TO PUBLIC
USING (owner_id = current_user_id() AND NOT is_locked)
WITH CHECK (owner_id = current_user_id());
```
#### ALTER POLICY and DROP POLICY
```sql
-- Change the USING expression
ALTER POLICY user_isolation ON orders
USING (user_id = current_user_id() OR is_public = true);
-- Change target roles
ALTER POLICY manager_access ON orders
TO managers, executives;
-- Rename policy
ALTER POLICY old_policy_name ON orders RENAME TO new_policy_name;
-- Drop a policy
DROP POLICY user_isolation ON orders;
-- Drop if exists
DROP POLICY IF EXISTS deprecated_policy ON old_table;
```
#### Policy Types: PERMISSIVE vs RESTRICTIVE
**PERMISSIVE (default):**
- Multiple permissive policies are combined with OR
- A row is visible if ANY permissive policy allows it
**RESTRICTIVE:**
- Multiple restrictive policies are combined with AND
- A row is visible only if ALL restrictive policies allow it
**Combined behavior:**
```
(permissive_1 OR permissive_2 OR ...) AND restrictive_1 AND restrictive_2 AND ...
```
**Example:**
```sql
-- Permissive: users can see own documents OR public documents
CREATE POLICY own_or_public ON documents
AS PERMISSIVE
FOR SELECT
USING (owner_id = current_user_id() OR is_public = true);
-- Permissive: admins can see everything
CREATE POLICY admin_access ON documents
AS PERMISSIVE
FOR SELECT
TO admins
USING (true);
-- Restrictive: only non-deleted documents
CREATE POLICY not_deleted ON documents
AS RESTRICTIVE
FOR SELECT
USING (deleted_at IS NULL);
-- Result for regular users:
-- (owner_id = current_user_id() OR is_public = true) AND deleted_at IS NULL
-- Result for admins:
-- (owner_id = current_user_id() OR is_public = true OR true) AND deleted_at IS NULL
-- Simplifies to: deleted_at IS NULL (admins see all non-deleted)
-- Restrictive policy: only during business hours
CREATE POLICY business_hours ON sensitive_ops
AS RESTRICTIVE
FOR ALL
TO PUBLIC
USING (
EXTRACT(HOUR FROM CURRENT_TIMESTAMP) BETWEEN 9 AND 17
AND EXTRACT(DOW FROM CURRENT_TIMESTAMP) BETWEEN 1 AND 5
);
```
#### Policy Commands
| Command | Applies To | Description |
|---------|------------|-------------|
| `ALL` | SELECT, INSERT, UPDATE, DELETE | Policy applies to all commands |
| `SELECT` | SELECT, UPDATE (for WHERE), DELETE (for WHERE) | Controls which rows can be read |
| `INSERT` | INSERT | Controls which rows can be inserted |
| `UPDATE` | UPDATE | Controls which rows can be updated |
| `DELETE` | DELETE | Controls which rows can be deleted |
**Per-Command Policies:**
```sql
-- Different policies for different operations
CREATE POLICY read_all ON orders
FOR SELECT
USING (true); -- Everyone can read
CREATE POLICY insert_own ON orders
FOR INSERT
WITH CHECK (user_id = current_user_id()); -- Can only insert own
CREATE POLICY update_own ON orders
FOR UPDATE
USING (user_id = current_user_id()) -- Can only see own for update
WITH CHECK (user_id = current_user_id()); -- Can only update to own
CREATE POLICY delete_own ON orders
FOR DELETE
USING (user_id = current_user_id()); -- Can only delete own
```
#### USING vs WITH CHECK
| Clause | Purpose | Used For |
|--------|---------|----------|
| `USING` | Filter existing rows | SELECT, UPDATE (which rows), DELETE |
| `WITH CHECK` | Validate new/modified rows | INSERT, UPDATE (new values) |
**USING Expression:**
- Applied to existing rows before they are returned or modified
- For SELECT: filters which rows are visible
- For UPDATE/DELETE: filters which rows can be affected
**WITH CHECK Expression:**
- Applied to new row values
- For INSERT: validates the new row
- For UPDATE: validates the updated row values
- If omitted, defaults to the USING expression
**Example:**
```sql
-- Users can see and modify their own records
-- But cannot change ownership
CREATE POLICY ownership ON records
FOR ALL
USING (owner_id = current_user_id()) -- Can only see own
WITH CHECK (owner_id = current_user_id()); -- Cannot change owner
-- This prevents:
-- UPDATE records SET owner_id = other_user WHERE ...
-- Because WITH CHECK would fail
```
#### RLS Bypass Rules
Certain roles can bypass RLS:
| Role Type | Bypass RLS? | Notes |
|-----------|-------------|-------|
| Superuser | Always | Cannot be restricted |
| `BYPASSRLS` role | Always | Explicitly granted bypass |
| Table owner | By default | Unless `FORCE ROW LEVEL SECURITY` is set |
| Regular users | Never | Must pass all policies |
**Examples:**
```sql
-- Create user that bypasses RLS (for ETL processes)
CREATE USER etl_processor BYPASSRLS PASSWORD 'etl_pass';
-- Force owner to also obey RLS
ALTER TABLE sensitive_data FORCE ROW LEVEL SECURITY;
-- Now even the owner must pass policies
-- (except superusers, who always bypass)
```
#### Multi-Tenant Example
```sql
-- Setup: Table with tenant_id
CREATE TABLE customer_data (
id SERIAL PRIMARY KEY,
tenant_id INTEGER NOT NULL,
data JSONB
);
-- Enable RLS
ALTER TABLE customer_data ENABLE ROW LEVEL SECURITY;
-- Policy: Users can only see their tenant's data
CREATE POLICY tenant_isolation ON customer_data
FOR ALL
USING (tenant_id = current_setting('app.tenant_id')::INTEGER)
WITH CHECK (tenant_id = current_setting('app.tenant_id')::INTEGER);
-- Application sets tenant context at connection start:
-- SET app.tenant_id = '42';
-- All subsequent queries automatically filtered to tenant 42
```
#### Policy Examples
```sql
-- Managers can see employees in their department
CREATE POLICY managers_view_dept ON employees
FOR SELECT
USING (
department_id IN (
SELECT department_id FROM managers
WHERE user_id = current_user_id()
)
);
-- Admins can see everything
CREATE POLICY admin_all_access ON employees
FOR ALL
TO admin_role
USING (true); -- No restriction
-- Time-based access: Only see records from last 30 days
CREATE POLICY recent_only ON audit_logs
FOR SELECT
USING (created_at > NOW() - INTERVAL '30 days');
```
### 20.6 Database Management
#### CREATE DATABASE
Create a new database with workspace-based isolation. Each database operates as an independent workspace where tables, views, and other objects are fully isolated from other databases.
```sql
-- Create a new database
CREATE DATABASE analytics;
-- Create only if it doesn't already exist
CREATE DATABASE IF NOT EXISTS staging;
-- Create with options
CREATE DATABASE production
OWNER admin_user
ENCODING 'UTF8';
```
**Supported Options:**
| Option | Description |
|--------|-------------|
| `OWNER` | Role that owns the database |
| `ENCODING` | Character encoding (default: UTF8) |
| `TEMPLATE` | Template database to copy from |
| `LC_COLLATE` | Collation order |
| `LC_CTYPE` | Character classification |
| `CONNECTION LIMIT` | Maximum concurrent connections |
| `IF NOT EXISTS` | Skip if database already exists |
**Workspace Isolation:**
Databases in Cognica provide strong isolation. Tables in different databases are completely independent, even if they share the same name:
```sql
-- Connect to database 'analytics'
-- \c analytics
CREATE TABLE events (id BIGINT, name TEXT);
-- Connect to database 'production'
-- \c production
CREATE TABLE events (id BIGINT, name TEXT, priority INT);
-- This is a completely separate table from analytics.events
```
**Checking Available Databases:**
```sql
-- List all databases (via system catalog)
SELECT datname FROM pg_database;
-- Get current database name
SELECT current_catalog;
-- or
SELECT current_database();
```
#### DROP DATABASE
```sql
-- Drop a database
DROP DATABASE analytics;
-- Drop only if it exists
DROP DATABASE IF EXISTS staging;
```
**Requirements:**
- You must have the `CREATEDB` privilege or be a superuser
- You cannot drop the database you are currently connected to
- All active connections to the target database must be closed first
### 20.7 System Functions
#### Role Information Functions
| Function | Return Type | Description |
|----------|-------------|-------------|
| `current_user` | name | Current active role name |
| `current_role` | name | Same as current_user |
| `session_user` | name | Original authenticated role |
| `pg_get_userbyid(oid)` | name | Role name for given OID |
**Examples:**
```sql
-- Get current user information
SELECT current_user, session_user;
-- Get role name from OID
SELECT pg_get_userbyid(10); -- Returns 'admin' (bootstrap superuser)
-- Use in queries
SELECT * FROM orders WHERE created_by = current_user;
```
#### Privilege Checking Functions
| Function | Return Type | Description |
|----------|-------------|-------------|
| `has_table_privilege(table, privilege)` | boolean | Check current user's privilege |
| `has_table_privilege(user, table, privilege)` | boolean | Check specific user's privilege |
| `has_column_privilege(table, column, privilege)` | boolean | Check column privilege |
| `has_column_privilege(user, table, column, privilege)` | boolean | Check specific user's column privilege |
**Examples:**
```sql
-- Check if current user can SELECT from orders
SELECT has_table_privilege('orders', 'SELECT');
-- Check if alice can INSERT into users
SELECT has_table_privilege('alice', 'users', 'INSERT');
-- Check multiple privileges
SELECT has_table_privilege('orders', 'SELECT, UPDATE');
-- Check column privilege
SELECT has_column_privilege('users', 'email', 'SELECT');
```
#### Session and Database Functions
| Function | Return Type | Description |
|----------|-------------|-------------|
| `current_catalog` | name | Current database name |
| `current_database()` | name | Current database name (alias) |
| `current_schema()` | name | Current schema name |
| `pg_backend_pid()` | integer | Process ID of current session |
| `pg_cancel_backend(pid)` | boolean | Cancel a running query by process ID |
| `version()` | text | Cognica version string |
| `pg_typeof(expression)` | regtype | Returns the data type of an expression |
| `to_regtype(type_name)` | regtype | Converts a type name string to regtype OID (returns NULL for invalid types) |
| `gen_random_uuid()` | uuid | Generates a random UUID v4 |
| `pg_encoding_to_char(encoding_id)` | name | Encoding ID to encoding name |
**Examples:**
```sql
-- Get current database context
SELECT current_catalog, current_schema(), pg_backend_pid();
-- Generate UUIDs for primary keys
INSERT INTO events (id, name) VALUES (gen_random_uuid(), 'signup');
-- Validate type names programmatically
SELECT to_regtype('integer'); -- Returns 'integer'
SELECT to_regtype('nonexistent'); -- Returns NULL
-- Get the type of an expression
SELECT pg_typeof(42); -- Returns 'integer'
SELECT pg_typeof(3.14); -- Returns 'numeric'
SELECT pg_typeof('hello'::text); -- Returns 'text'
-- Cancel a long-running query from another session
SELECT pg_cancel_backend(12345);
-- Returns true if cancellation signal was sent successfully
```
**Query Cancellation:**
The `pg_cancel_backend()` function sends a cancellation signal to a running query identified by its process ID. This is useful for terminating long-running or runaway queries without disconnecting the client session. The cancelled query receives a SQLSTATE 57014 error (`query_canceled`).
```sql
-- Find long-running queries and cancel them
SELECT pg_backend_pid(), current_catalog;
-- In another session:
SELECT pg_cancel_backend(pid_from_above);
```
### 20.8 System Catalog Views
#### pg_catalog Views
**pg_roles - All database roles:**
```sql
SELECT rolname, rolsuper, rolinherit, rolcreaterole, rolcreatedb,
rolcanlogin, rolreplication, rolbypassrls, rolconnlimit
FROM pg_roles;
```
| Column | Type | Description |
|--------|------|-------------|
| `rolname` | name | Role name |
| `rolsuper` | boolean | Is superuser |
| `rolinherit` | boolean | Inherits privileges |
| `rolcreaterole` | boolean | Can create roles |
| `rolcreatedb` | boolean | Can create databases |
| `rolcanlogin` | boolean | Can log in |
| `rolreplication` | boolean | Can do replication |
| `rolbypassrls` | boolean | Bypasses RLS |
| `rolconnlimit` | integer | Connection limit |
| `rolvaliduntil` | timestamptz | Password expiration |
**pg_user - Login roles only:**
```sql
SELECT usename, usesysid, usecreatedb, usesuper
FROM pg_user;
```
**pg_auth_members - Role memberships:**
```sql
SELECT r.rolname AS role, m.rolname AS member,
am.admin_option, am.inherit_option, am.set_option
FROM pg_auth_members am
JOIN pg_roles r ON r.oid = am.roleid
JOIN pg_roles m ON m.oid = am.member;
```
#### information_schema Views
**applicable_roles - Roles applicable to current user:**
```sql
SELECT * FROM information_schema.applicable_roles;
```
**role_table_grants - Table privileges by role:**
```sql
SELECT grantor, grantee, table_name, privilege_type, is_grantable
FROM information_schema.role_table_grants
WHERE grantee = 'alice';
```
**table_privileges - All table privileges:**
```sql
SELECT grantor, grantee, table_schema, table_name, privilege_type
FROM information_schema.table_privileges
WHERE table_schema = 'public';
```
**column_privileges - Column privileges:**
```sql
SELECT grantor, grantee, table_name, column_name, privilege_type
FROM information_schema.column_privileges
WHERE grantee = current_user;
```
### 20.9 Common Patterns
#### Multi-Tenant Application
```sql
-- Setup
CREATE TABLE tenants (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL
);
CREATE TABLE tenant_data (
id SERIAL PRIMARY KEY,
tenant_id INTEGER REFERENCES tenants(id),
data JSONB
);
-- Enable RLS
ALTER TABLE tenant_data ENABLE ROW LEVEL SECURITY;
-- Function to get current tenant
CREATE FUNCTION current_tenant_id() RETURNS INTEGER AS $$
SELECT current_setting('app.tenant_id')::INTEGER
$$ LANGUAGE SQL SECURITY DEFINER;
-- Policy: users can only see their tenant's data
CREATE POLICY tenant_isolation ON tenant_data
FOR ALL
USING (tenant_id = current_tenant_id())
WITH CHECK (tenant_id = current_tenant_id());
```
#### Audit Trail with RLS
```sql
-- Audit table
CREATE TABLE audit_log (
id SERIAL PRIMARY KEY,
table_name TEXT,
action TEXT,
old_data JSONB,
new_data JSONB,
user_name TEXT DEFAULT session_user,
timestamp TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);
-- RLS: users can only see their own audit entries
ALTER TABLE audit_log ENABLE ROW LEVEL SECURITY;
CREATE POLICY own_audit ON audit_log
FOR SELECT
USING (user_name = session_user);
-- Admins can see all
CREATE POLICY admin_audit ON audit_log
FOR SELECT
TO admin_role
USING (true);
```
#### Department-Based Access Control
```sql
-- Setup
CREATE TABLE departments (
id SERIAL PRIMARY KEY,
name TEXT
);
CREATE TABLE user_departments (
user_id TEXT,
department_id INTEGER REFERENCES departments(id)
);
CREATE TABLE department_data (
id SERIAL PRIMARY KEY,
department_id INTEGER REFERENCES departments(id),
data JSONB
);
-- Enable RLS
ALTER TABLE department_data ENABLE ROW LEVEL SECURITY;
-- Policy: users can only see their departments' data
CREATE POLICY dept_access ON department_data
FOR ALL
USING (
department_id IN (
SELECT department_id FROM user_departments
WHERE user_id = current_user
)
);
```
### 20.10 Security Best Practices
#### Principle of Least Privilege
```sql
-- BAD: Give everyone full access
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO app_role;
-- GOOD: Grant only what's needed
GRANT SELECT ON products TO web_app;
GRANT SELECT, INSERT ON orders TO web_app;
GRANT SELECT, UPDATE (status) ON orders TO order_processor;
```
#### Use Roles for Applications
```sql
-- Create a dedicated role for each application
CREATE ROLE web_backend LOGIN PASSWORD 'secret1';
CREATE ROLE mobile_api LOGIN PASSWORD 'secret2';
CREATE ROLE analytics_etl LOGIN PASSWORD 'secret3';
-- Grant appropriate permissions to each
GRANT SELECT, INSERT, UPDATE ON orders TO web_backend;
GRANT SELECT ON orders TO mobile_api; -- Read-only
GRANT SELECT ON ALL TABLES IN SCHEMA public TO analytics_etl;
```
#### Audit Sensitive Operations
```sql
-- Create audit table
CREATE TABLE security_audit (
id SERIAL PRIMARY KEY,
event_type TEXT,
table_name TEXT,
user_name TEXT,
timestamp TIMESTAMPTZ DEFAULT NOW(),
details JSONB
);
-- Use triggers to log sensitive operations
CREATE OR REPLACE FUNCTION audit_sensitive_access()
RETURNS TRIGGER AS $$
BEGIN
INSERT INTO security_audit (event_type, table_name, user_name, details)
VALUES (TG_OP, TG_TABLE_NAME, current_user,
jsonb_build_object('row_id', NEW.id));
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER audit_salary_changes
AFTER UPDATE OF salary ON employees
FOR EACH ROW EXECUTE FUNCTION audit_sensitive_access();
```
#### Regular Privilege Audits
```sql
-- Find roles with superuser privilege
SELECT rolname FROM pg_roles WHERE rolsuper;
-- Find roles that can bypass RLS
SELECT rolname FROM pg_roles WHERE rolbypassrls OR rolsuper;
-- Find roles with CREATEROLE (can create other roles)
SELECT rolname FROM pg_roles WHERE rolcreaterole;
-- Find all privileges on a specific table
SELECT grantee, privilege_type, is_grantable
FROM information_schema.table_privileges
WHERE table_name = 'sensitive_table';
```
### 20.11 Security Troubleshooting
#### Common Issues
**"Permission denied" errors:**
```sql
-- Check if user has the required privilege
SELECT has_table_privilege('alice', 'orders', 'SELECT');
-- View all privileges on a table
SELECT grantee, privilege_type
FROM information_schema.table_privileges
WHERE table_name = 'orders';
-- Check role memberships
SELECT r.rolname AS role, m.rolname AS member
FROM pg_auth_members am
JOIN pg_roles r ON r.oid = am.roleid
JOIN pg_roles m ON m.oid = am.member
WHERE m.rolname = 'alice';
```
**RLS blocking all rows:**
```sql
-- Check if RLS is enabled
SELECT relname, relrowsecurity, relforcerowsecurity
FROM pg_class
WHERE relname = 'my_table';
-- List all policies on a table
SELECT polname, polcmd, polpermissive, polroles, polqual
FROM pg_policy
WHERE polrelid = 'my_table'::regclass;
-- Temporarily disable RLS (superuser only, for debugging)
ALTER TABLE my_table DISABLE ROW LEVEL SECURITY;
```
**Session/role confusion:**
```sql
-- See who you actually are
SELECT session_user, current_user, current_role;
-- See all roles you can become
SELECT * FROM information_schema.applicable_roles;
-- Reset to your original role
RESET ROLE;
```
**Password authentication failed:**
```sql
-- Check if password is expired
SELECT rolname, rolvaliduntil
FROM pg_roles
WHERE rolname = 'alice';
-- Check if role can login
SELECT rolname, rolcanlogin
FROM pg_roles
WHERE rolname = 'alice';
-- Reset password
ALTER USER alice PASSWORD 'new_password';
```
---
## Chapter 21: Data Federation - Unified Access to Distributed Data
Data federation enables querying data across multiple sources as if they were a single database. Instead of moving data into Cognica, you bring the query to the data. This chapter explains why federation matters, how Cognica implements it, and when to use it.
### Why Data Federation Matters
Modern enterprises rarely have all their data in one place. Data lives in:
- Legacy databases (PostgreSQL, MySQL, Oracle)
- Cloud data warehouses (Snowflake, BigQuery, Redshift)
- Data lakes (Delta Lake, Apache Iceberg on S3/GCS)
- Operational databases across regions
- Partner and vendor systems
Traditional approaches require ETL pipelines to copy data into a central warehouse. This creates problems:
| Problem | Impact |
|---------|--------|
| Data staleness | ETL runs periodically; queries see outdated data |
| Storage duplication | Same data stored multiple times, increasing costs |
| Pipeline maintenance | ETL jobs fail, schemas change, pipelines break |
| Governance complexity | Data copies multiply compliance burden |
| Time to insight | New data sources require new pipelines |
Data federation solves these problems by querying data in place. Cognica sends queries to remote systems, retrieves results, and combines them with local data - all in a single SQL statement.
### The Business Case for Federation
**1. Faster Time to Value**
Without federation:
```
Week 1-2: Design ETL pipeline
Week 3-4: Implement and test pipeline
Week 5: Deploy and schedule
Week 6+: Maintain forever
```
With federation:
```sql
-- Day 1: Query remote data immediately
CREATE FOREIGN TABLE remote_customers ()
SERVER flight_sql_server
OPTIONS (source_type 'cognica', host 'us-west.cognica.example.com', ...);
SELECT * FROM remote_customers WHERE region = 'west';
```
**2. Real-Time Data Access**
Federation queries see current data, not yesterday's ETL snapshot:
```sql
-- Combine local orders with real-time inventory from warehouse system
SELECT o.order_id, o.product_id, i.current_stock
FROM orders o
JOIN warehouse_inventory i ON o.product_id = i.product_id
WHERE o.status = 'pending'
AND i.current_stock < o.quantity;
```
**3. Data Sovereignty Compliance**
Some data cannot leave its jurisdiction. Federation queries data in place:
```sql
-- Query EU customer data from EU datacenter without copying to US
SELECT COUNT(*) as eu_customers, AVG(lifetime_value)
FROM eu_datacenter.customers
WHERE country IN ('DE', 'FR', 'IT');
```
**4. Reduced Infrastructure Costs**
No need to provision storage for copies of data that already exists elsewhere:
- Query 10TB data lake without importing 10TB
- Access archived data on cold storage without restoration
- Use partner APIs without local caching
### Cognica's Federation Architecture
Cognica implements federation through Foreign Tables with three specialized backends:
| Backend | Supported Sources | Best For |
|---------|-------------------|----------|
| **file_server** | Parquet, CSV, JSON, Arrow IPC, ORC | Data lakes, file-based analytics. Direct access via Apache Arrow with predicate pushdown for Parquet/ORC. |
| **duckdb_server** | PostgreSQL, MySQL, SQLite, DuckDB, Delta Lake, Iceberg | Relational databases and modern table formats. Connects via DuckDB extensions with SQL pushdown. |
| **flight_sql_server** | Cognica, ClickHouse, DataFusion, Dremio | High-performance analytics. 10-100x faster than JDBC/ODBC. Native protocol for Cognica-to-Cognica federation. |
### Cognica-to-Cognica Federation
One of Cognica's unique capabilities is federating across multiple Cognica instances. This enables:
**Multi-Region Architectures**
```sql
-- Create connections to regional Cognica instances
CREATE FOREIGN TABLE us_orders ()
SERVER flight_sql_server
OPTIONS (
source_type 'cognica',
host 'us-east.example.com',
port '8815',
query 'SELECT * FROM orders'
);
CREATE FOREIGN TABLE eu_orders ()
SERVER flight_sql_server
OPTIONS (
source_type 'cognica',
host 'eu-west.example.com',
port '8815',
query 'SELECT * FROM orders'
);
CREATE FOREIGN TABLE apac_orders ()
SERVER flight_sql_server
OPTIONS (
source_type 'cognica',
host 'ap-seoul.example.com',
port '8815',
query 'SELECT * FROM orders'
);
-- Global query across all regions
SELECT region, COUNT(*) as order_count, SUM(total) as revenue
FROM (
SELECT 'US' as region, * FROM us_orders
UNION ALL
SELECT 'EU' as region, * FROM eu_orders
UNION ALL
SELECT 'APAC' as region, * FROM apac_orders
) global_orders
WHERE order_date >= CURRENT_DATE - INTERVAL '7 days'
GROUP BY region;
```
**Distributed Cognica Clusters**
For horizontal scaling, deploy specialized Cognica instances:
```sql
-- Analytics cluster handles heavy aggregations
CREATE FOREIGN TABLE analytics_summary ()
SERVER flight_sql_server
OPTIONS (
source_type 'cognica',
host 'analytics.internal',
query 'SELECT date, product_category, SUM(revenue) as total
FROM sales GROUP BY 1, 2'
);
-- OLTP cluster handles transactional data
CREATE FOREIGN TABLE current_inventory ()
SERVER flight_sql_server
OPTIONS (
source_type 'cognica',
host 'oltp.internal',
query 'SELECT * FROM inventory WHERE quantity > 0'
);
-- Combine for operational analytics
SELECT a.product_category, a.total as last_month_revenue,
SUM(i.quantity) as current_stock
FROM analytics_summary a
JOIN current_inventory i ON a.product_category = i.category
WHERE a.date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY 1, 2;
```
**Development and Staging Environments**
Query production data from development without copying:
```sql
-- Development environment queries production (read-only)
CREATE FOREIGN TABLE prod_users ()
SERVER flight_sql_server
OPTIONS (
source_type 'cognica',
host 'prod.internal',
query 'SELECT id, email, created_at FROM users' -- Exclude sensitive columns
);
-- Test against real data patterns
SELECT DATE_TRUNC('month', created_at) as month, COUNT(*)
FROM prod_users
GROUP BY 1
ORDER BY 1;
```
### Federation Use Cases
#### Use Case 1: Data Mesh Implementation
In a data mesh architecture, each domain owns its data. Federation enables cross-domain queries:
```sql
-- Sales domain owns customer data
CREATE FOREIGN TABLE sales.customers ()
SERVER flight_sql_server
OPTIONS (source_type 'cognica', host 'sales.mesh.internal', ...);
-- Marketing domain owns campaign data
CREATE FOREIGN TABLE marketing.campaigns ()
SERVER flight_sql_server
OPTIONS (source_type 'cognica', host 'marketing.mesh.internal', ...);
-- Finance domain owns billing data
CREATE FOREIGN TABLE finance.invoices ()
SERVER duckdb_server
OPTIONS (source_type 'postgres', host 'finance-db.internal', ...);
-- Cross-domain analytics query
SELECT
c.customer_segment,
COUNT(DISTINCT c.customer_id) as customers,
COUNT(DISTINCT m.campaign_id) as campaigns_reached,
SUM(f.amount) as total_revenue
FROM sales.customers c
LEFT JOIN marketing.campaigns m ON c.customer_id = m.customer_id
LEFT JOIN finance.invoices f ON c.customer_id = f.customer_id
WHERE f.invoice_date >= '2025-01-01'
GROUP BY 1;
```
#### Use Case 2: Hybrid Cloud Analytics
Query data across cloud providers and on-premises:
```sql
-- On-premises legacy database
CREATE FOREIGN TABLE legacy_erp ()
SERVER duckdb_server
OPTIONS (source_type 'postgres', host '10.0.1.50', ...);
-- AWS S3 data lake
CREATE FOREIGN TABLE s3_events ()
SERVER file_server
OPTIONS (path 's3://analytics-bucket/events/*.parquet', format 'parquet');
-- Google Cloud Cognica instance
CREATE FOREIGN TABLE gcp_ml_scores ()
SERVER flight_sql_server
OPTIONS (source_type 'cognica', host 'ml.gcp.example.com', ...);
-- Unified view across all sources
SELECT
e.customer_id,
l.customer_name,
m.propensity_score
FROM s3_events e
JOIN legacy_erp l ON e.customer_id = l.id
JOIN gcp_ml_scores m ON e.customer_id = m.customer_id
WHERE e.event_type = 'purchase'
AND e.event_date = CURRENT_DATE;
```
#### Use Case 3: Zero-Copy Data Sharing
Share data with partners without copying:
```sql
-- Partner creates foreign table pointing to your Cognica
-- (on partner's Cognica instance)
CREATE FOREIGN TABLE supplier_inventory ()
SERVER flight_sql_server
OPTIONS (
source_type 'cognica',
host 'partner-api.supplier.com',
query 'SELECT sku, quantity, warehouse FROM shared.inventory'
);
-- Partner queries your data in real-time
SELECT sku, quantity
FROM supplier_inventory
WHERE warehouse = 'east-coast';
```
#### Use Case 4: Gradual Migration
Migrate from legacy database while maintaining operations:
```sql
-- Phase 1: Create foreign table to legacy system
CREATE FOREIGN TABLE legacy_orders ()
SERVER duckdb_server
OPTIONS (source_type 'postgres', host 'legacy-db', ...);
-- Phase 2: Create local table for new data
CREATE TABLE orders (
id BIGINT PRIMARY KEY,
customer_id BIGINT,
order_date TIMESTAMP,
total DECIMAL(10,2)
);
-- Phase 3: Create unified view
CREATE VIEW all_orders AS
SELECT * FROM orders
WHERE order_date >= '2025-01-01' -- New orders in Cognica
UNION ALL
SELECT * FROM legacy_orders
WHERE order_date < '2025-01-01'; -- Historical in legacy
-- Applications query the view - migration is transparent
SELECT * FROM all_orders WHERE customer_id = 12345;
-- Phase 4: Gradually migrate historical data
-- Phase 5: Drop foreign table when migration complete
```
### Performance Considerations
Federation involves network round-trips. Optimize for performance:
**1. Push Predicates to Remote Systems**
```sql
-- Good: Filter pushed to remote
SELECT * FROM remote_orders WHERE order_date = '2025-01-15';
-- Bad: All data transferred, filtered locally
SELECT * FROM remote_orders WHERE UPPER(status) = 'PENDING';
```
**2. Minimize Data Transfer**
```sql
-- Good: Select only needed columns
SELECT customer_id, total FROM remote_orders;
-- Bad: Transfer entire rows
SELECT * FROM remote_orders;
```
**3. Aggregate Remotely When Possible**
```sql
-- Good: Aggregation in remote query
CREATE FOREIGN TABLE daily_totals ()
SERVER flight_sql_server
OPTIONS (
query 'SELECT DATE(order_date), SUM(total) FROM orders GROUP BY 1'
);
-- Bad: Transfer all rows, aggregate locally
SELECT DATE(order_date), SUM(total) FROM remote_orders GROUP BY 1;
```
**4. Cache Frequently Accessed Remote Data**
```sql
-- Create materialized view for frequently queried remote data
CREATE MATERIALIZED VIEW customer_cache AS
SELECT * FROM remote_customers;
-- Refresh periodically
REFRESH MATERIALIZED VIEW customer_cache;
-- Query cache instead of remote
SELECT * FROM customer_cache WHERE region = 'west';
```
### Security in Federated Environments
**1. Credential Management**
Never embed credentials in foreign table definitions for production:
```sql
-- Development only - credentials in definition
CREATE FOREIGN TABLE dev_data ()
SERVER duckdb_server
OPTIONS (source_type 'postgres', password 'secret', ...);
-- Production - use environment variables or secrets manager
-- Credentials configured at server level, not table level
```
**2. Query Restrictions**
Use the `query` option to restrict what remote data is accessible:
```sql
-- Expose only non-sensitive columns
CREATE FOREIGN TABLE safe_customers ()
SERVER flight_sql_server
OPTIONS (
query 'SELECT id, name, city, state FROM customers' -- No SSN, no email
);
```
**3. Network Security**
- Use TLS for all remote connections (`tls_enabled 'true'`)
- Restrict network access with firewalls
- Use private endpoints for cloud connections
**4. Audit Trail**
Foreign table queries are logged like local queries. Monitor for unusual access patterns.
### 21.1 Federation Best Practices
**Architecture**:
- Use federation for analytics and exploration
- Import frequently-accessed data for performance-critical paths
- Design foreign tables around common query patterns
**Naming Conventions**:
```sql
-- Include source information in table names
CREATE FOREIGN TABLE ext_postgres_customers ... -- External PostgreSQL
CREATE FOREIGN TABLE ext_s3_events ... -- External S3
CREATE FOREIGN TABLE remote_us_orders ... -- Remote Cognica (US)
```
**Documentation**:
- Document each foreign table's source, refresh characteristics, and limitations
- Note which columns are available vs. the full source schema
- Track SLAs for remote system availability
**Monitoring**:
- Monitor federation query latency separately from local queries
- Alert on remote system unavailability
- Track data freshness for cached/materialized remote data
**Governance**:
- Catalog all foreign tables and their sources
- Review foreign table access as part of security audits
- Establish ownership for each federated data source
---
## Chapter 22: Foreign Tables - Querying External Files
Foreign tables let you query external data files directly using SQL, without first importing the data. Cognica provides a streamlined syntax using predefined servers for different data source types.
### When to Use Foreign Tables vs. Regular Tables
The decision between importing data and using foreign tables involves trade-offs:
**Use Foreign Tables When:**
- Data is too large to import or impractical to duplicate
- Data changes frequently in the source system
- You need to query data once or infrequently
- You are exploring data before deciding whether to import it
- Data must remain in place (regulatory, architectural, or organizational constraints)
**Import Into Regular Tables When:**
- Query performance is critical (regular tables are faster)
- You query the same data repeatedly
- You need indexes, constraints, or triggers on the data
- You need to join frequently with other database tables
- Data consistency with your database transactions matters
**Performance Expectations**: Foreign tables execute queries against external files, which is inherently slower than querying indexed, local data. Expect foreign table queries to be 10-100x slower than equivalent queries on imported data. This is acceptable for analytics, exploration, and ETL, but not for high-performance transactional workloads.
### 22.1 Understanding Foreign Tables
#### The Three-Backend Architecture
Cognica implements External Virtual Tables through three distinct backends, each optimized for different types of data sources:
**The Arrow Backend** (`file_server`) excels at reading columnar file formats like Parquet, ORC, and Arrow IPC. Apache Arrow provides highly optimized readers for these formats, with support for predicate pushdown directly into the file format's metadata and statistics. When you query a Parquet file, Arrow can skip entire row groups that don't match your filter conditions without reading the underlying data. The Arrow backend also handles partitioned datasets natively, understanding Hive-style partition layouts and applying partition pruning automatically.
**The DuckDB Backend** (`duckdb_server`) handles database connectivity and data lake formats. DuckDB's extension ecosystem provides battle-tested connectors for PostgreSQL, MySQL, and SQLite, as well as readers for Delta Lake and Apache Iceberg. By embedding DuckDB as a query processing engine, Cognica gains access to this entire ecosystem while maintaining a consistent SQL interface.
**The Flight SQL Backend** (`flight_sql_server`) connects to databases and analytics platforms that expose the Arrow Flight SQL protocol. Arrow Flight SQL is a high-performance wire protocol built on Apache Arrow that enables columnar data transfer, achieving 10-100x better throughput than traditional ODBC/JDBC for analytical workloads. This backend is ideal for connecting to ClickHouse, other Cognica instances, Apache DataFusion/Ballista, and Dremio.
| Backend | Server Name | Best For | Data Sources |
|---------|-------------|----------|--------------|
| Arrow | `file_server` | File formats | Parquet, CSV, JSON, Arrow IPC, ORC |
| DuckDB | `duckdb_server` | Databases, Data Lakes | PostgreSQL, MySQL, SQLite, Delta Lake, Iceberg, Cognica (via PostgreSQL protocol) |
| Flight SQL | `flight_sql_server` | High-performance analytics | ClickHouse, Cognica, DataFusion, Dremio |
**Connecting to Cognica from Another Cognica Instance:**
Cognica can be accessed through two different backends:
| Method | Backend | Protocol | Best For |
|--------|---------|----------|----------|
| PostgreSQL Protocol | `duckdb_server` | Row-based | Transactional queries, small result sets, simpler setup |
| Flight SQL Protocol | `flight_sql_server` | Columnar (Arrow) | Analytical queries, large scans, high throughput |
For high-volume analytical workloads or when transferring large result sets, prefer `flight_sql_server`. For simple queries or when minimal setup is preferred, `duckdb_server` with PostgreSQL source works well.
#### How Foreign Tables Work
Foreign tables create a SQL interface to external files. When you query a foreign table:
1. Cognica reads the file(s) from the specified path
2. The schema is automatically inferred from the file contents
3. Predicates (WHERE conditions) are pushed down to reduce I/O
4. Results stream through the query engine like regular table data
This approach lets you query terabytes of external data without importing anything into the database.
#### Query Optimization
Cognica optimizes queries against external sources through several mechanisms:
**Predicate Pushdown** ensures that filter conditions in your WHERE clauses are pushed down to the external data source whenever possible. Rather than fetching an entire table and filtering locally, Cognica instructs the remote system to perform the filtering, dramatically reducing network transfer and processing time.
**Projection Pushdown** works similarly for column selection. When your query only references specific columns, Cognica requests only those columns from the external source, further reducing data transfer overhead.
**Partition Pruning** applies to partitioned datasets. If the data is organized by date (e.g., `year=2024/month=12/`), queries filtering on partition columns skip entire directories without scanning their contents.
#### The file_server
Cognica provides a built-in `file_server` for file-based foreign tables. You don't need to create extensions or configure servers - simply reference `file_server` in your CREATE FOREIGN TABLE statement.
**Use Cases**:
- Analyze log files without importing
- Query CSV exports from other systems
- Work with Parquet data lakes
- ETL pipeline source data
- One-time data exploration
### 22.2 Supported File Formats
| Format | format Value | Best For | Features |
|--------|--------------|----------|----------|
| Parquet | `parquet` | Analytics, data lakes | Columnar, compressed, predicate pushdown |
| Arrow IPC | `arrow` | Fast interchange | Zero-copy reads, schema in header |
| CSV | `csv` | Simple data exchange | Human-readable, universal |
| ORC | `orc` | Hadoop ecosystem | Columnar, optimized for Hive |
| JSON | `json` | Semi-structured data | JSON Lines format, flexible schema |
### 22.3 CREATE FOREIGN TABLE Syntax
```sql
CREATE FOREIGN TABLE [IF NOT EXISTS] table_name ()
SERVER file_server
OPTIONS (
path 'source_path',
format 'format_type',
[partitioning 'partitioning_type']
);
```
**Key Points**:
- The column list is **empty `()`** - Cognica automatically infers the schema from the source files
- `path` supports local paths, glob patterns, and cloud storage URIs (S3, GCS, Azure, HDFS)
- `format` specifies the file format: `parquet`, `csv`, `json`, `arrow`, `orc`
- `partitioning` is optional: `hive` for Hive-style partitioning, `directory`, or `none` (default)
### 22.4 Examples by Format
#### Parquet Files
Parquet is the recommended format for analytical workloads due to columnar storage and predicate pushdown:
```sql
-- Single Parquet file
CREATE FOREIGN TABLE sales_2024 ()
SERVER file_server
OPTIONS (path '/data/sales/2024.parquet', format 'parquet');
-- Directory of Parquet files with glob pattern
CREATE FOREIGN TABLE all_sales ()
SERVER file_server
OPTIONS (path '/data/sales/*.parquet', format 'parquet');
-- Amazon S3 with Hive partitioning (year=YYYY/month=MM directories)
CREATE FOREIGN TABLE s3_events ()
SERVER file_server
OPTIONS (
path 's3://analytics-bucket/events/',
format 'parquet',
partitioning 'hive'
);
-- Google Cloud Storage
CREATE FOREIGN TABLE gcs_metrics ()
SERVER file_server
OPTIONS (path 'gs://metrics-bucket/data/*.parquet', format 'parquet');
-- Azure Blob Storage
CREATE FOREIGN TABLE azure_data ()
SERVER file_server
OPTIONS (
path 'abfs://container@account.dfs.core.windows.net/data/',
format 'parquet'
);
-- Query with predicate pushdown (only reads matching row groups)
SELECT region, SUM(amount) AS total
FROM s3_events
WHERE year = 2024 AND month >= 10
GROUP BY region;
```
#### CSV Files
```sql
-- Basic CSV with inferred schema
CREATE FOREIGN TABLE customer_import ()
SERVER file_server
OPTIONS (path '/imports/customers.csv', format 'csv');
-- Multiple CSV files with glob pattern
CREATE FOREIGN TABLE all_logs ()
SERVER file_server
OPTIONS (path '/var/log/app/*.csv', format 'csv');
-- Query CSV data
SELECT country, COUNT(*) AS customer_count
FROM customer_import
GROUP BY country
ORDER BY customer_count DESC;
```
#### JSON Files (JSON Lines Format)
```sql
-- JSON Lines format (one JSON object per line)
CREATE FOREIGN TABLE json_logs ()
SERVER file_server
OPTIONS (path '/data/logs/*.json', format 'json');
-- Query JSON data
SELECT
DATE_TRUNC('hour', timestamp) AS hour,
level,
COUNT(*) AS count
FROM json_logs
WHERE level IN ('ERROR', 'WARN')
GROUP BY 1, 2
ORDER BY hour;
```
#### Arrow IPC Files
```sql
-- Arrow IPC format (fastest for Arrow-native workloads)
CREATE FOREIGN TABLE arrow_data ()
SERVER file_server
OPTIONS (path '/data/exchange/*.arrow', format 'arrow');
```
#### ORC Files
```sql
-- ORC files (Hadoop ecosystem)
CREATE FOREIGN TABLE hadoop_data ()
SERVER file_server
OPTIONS (path '/hdfs/warehouse/table/*.orc', format 'orc');
```
### 22.5 Cloud Storage Support
Cognica supports all major cloud storage providers through Arrow's filesystem abstraction:
| Storage | URI Scheme | Example Path |
|---------|------------|--------------|
| Local filesystem | Plain path or `file://` | `/data/sales/*.parquet` |
| Amazon S3 | `s3://` or `s3a://` | `s3://bucket/path/*.parquet` |
| Google Cloud Storage | `gs://` or `gcs://` | `gs://bucket/path/` |
| Azure Blob Storage | `abfs://` or `abfss://` | `abfs://container@account.dfs.core.windows.net/` |
| Hadoop HDFS | `hdfs://` or `viewfs://` | `hdfs://namenode:8020/path/` |
**Authentication** is handled through standard environment variables and credential files:
- **S3**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `~/.aws/credentials`
- **GCS**: `GOOGLE_APPLICATION_CREDENTIALS`, application default credentials
- **Azure**: `AZURE_STORAGE_CONNECTION_STRING`, Azure AD tokens
### 22.6 Working with Foreign Tables
#### Joining with Local Tables
```sql
-- Enrich file data with database data
SELECT
f.transaction_id,
f.amount,
c.name AS customer_name,
c.email
FROM file_transactions f
JOIN customers c ON f.customer_id = c.id
WHERE f.amount > 1000;
```
#### Importing Data
```sql
-- Import filtered data into a regular table
INSERT INTO transactions (id, customer_id, amount, date)
SELECT id, customer_id, amount, date
FROM file_transactions
WHERE date >= '2024-01-01';
-- Create a new table from foreign table query
CREATE TABLE q4_sales AS
SELECT * FROM file_sales WHERE sale_date >= '2024-10-01';
```
#### Write Operations
Foreign tables support INSERT, UPDATE, and DELETE operations:
```sql
-- INSERT appends to a new file
INSERT INTO archive_logs (timestamp, level, message)
VALUES (NOW(), 'INFO', 'Application started');
-- UPDATE uses copy-on-write (rewrites affected files)
UPDATE logs SET level = 'WARNING' WHERE level = 'WARN';
-- DELETE uses copy-on-write (rewrites files without deleted rows)
DELETE FROM logs WHERE timestamp < '2023-01-01';
```
### 22.7 Managing Foreign Tables
```sql
-- Drop a foreign table (doesn't delete the underlying files)
DROP FOREIGN TABLE logs;
-- Drop only if exists (no error if not found)
DROP FOREIGN TABLE IF EXISTS logs;
```
### 22.8 Best Practices
**Performance Considerations**:
- Foreign tables are slower than local tables for repeated queries
- **Parquet and ORC** are much faster than CSV for analytics (columnar, compressed, predicate pushdown)
- Filter early - WHERE clauses reduce data read from files
- Select only needed columns - columnar formats only read requested columns
**When to Use Foreign Tables**:
- One-time or infrequent analysis of external files
- Exploring data before deciding to import
- Ad-hoc queries on data lake files
- ETL pipeline source data
- Files that are updated externally
**When to Import Instead**:
- Frequently queried data (import and add indexes)
- Data that needs transaction support
- Data requiring referential integrity constraints
---
## Chapter 23: Foreign Tables - DuckDB Backend
Foreign tables can connect to external database systems and modern data lake formats using DuckDB as a backend engine. Cognica can query PostgreSQL, MySQL, SQLite databases and Delta Lake/Iceberg tables as if they were local tables.
For connecting to Arrow Flight SQL-compatible databases (ClickHouse, other Cognica instances, DataFusion, Dremio), see [Chapter 19: Foreign Tables - Arrow Flight SQL](#chapter-19-foreign-tables---arrow-flight-sql).
### The Promise and Challenges of Federated Queries
Federated queries let you write SQL that spans multiple databases as if they were one system. This is powerful but involves inherent challenges:
**What Works Well:**
- Reading data from external systems without duplication
- Combining data from multiple sources in a single query
- Accessing legacy systems through standard SQL
- One-time migrations and data exploration
**What To Be Cautious About:**
- **Network latency**: Every query crosses the network, adding latency
- **Query pushdown limitations**: Not all queries can be pushed to the remote system; sometimes data must be pulled locally for processing
- **Transaction boundaries**: Joins across databases do not share a transaction; you may see inconsistent snapshots
- **Schema drift**: If the external schema changes, your foreign table definition may become stale
**When Federated Queries Are Not the Right Choice:**
- High-frequency transactional queries (too slow)
- Data requiring strict consistency guarantees across sources
- Performance-critical paths in your application
Think of federated queries as a bridge, not a destination. They are excellent for analysis, integration, and migration, but production systems typically benefit from consolidated data.
### 23.1 Understanding the duckdb_server
Cognica provides a built-in `duckdb_server` for connecting to external databases and data lake formats. Unlike `file_server` which reads files directly via Arrow, `duckdb_server` leverages DuckDB's extension ecosystem to connect to:
- **Relational databases**: PostgreSQL, MySQL, SQLite
- **Data lake formats**: Delta Lake, Apache Iceberg
- **DuckDB files**: Native DuckDB database files
DuckDB automatically handles SQL pushdown, connection management, and type conversion.
**Key Benefits**:
- Query external databases without data duplication
- SQL queries are pushed to external database engines for optimal performance
- Delta Lake and Iceberg support includes transaction logs and time travel
- Federated queries can join multiple external sources
### 23.2 CREATE FOREIGN TABLE Syntax
```sql
CREATE FOREIGN TABLE [IF NOT EXISTS] [schema_name.]table_name ()
SERVER duckdb_server
OPTIONS (
source 'source_type',
connection 'connection_string',
table 'source_table_or_path',
[option = value, ...]
);
```
**Key Points**:
- The column list is **empty `()`** - schema is automatically inferred from the source
- `source` specifies the data source type: `postgresql`, `mysql`, `sqlite`, `delta`, `iceberg`, `duckdb`
- `connection` provides connection credentials for database sources
- `table` identifies the specific table name, or `path` for file-based sources
**Supported Source Types**:
| source Value | Data Source | Description |
|--------------|-------------|-------------|
| `postgresql` or `postgres` | PostgreSQL database | Push SQL queries to PostgreSQL |
| `mysql` | MySQL database | Push SQL queries to MySQL |
| `sqlite` | SQLite database | Query SQLite database files |
| `delta` or `deltalake` | Delta Lake | ACID transactions, time travel |
| `iceberg` | Apache Iceberg | Schema evolution, time travel |
| `duckdb` | DuckDB file | Native DuckDB database files |
### 23.3 Connecting to PostgreSQL
PostgreSQL connections support full SQL pushdown - queries are executed on the PostgreSQL server:
```sql
-- Create foreign table connected to PostgreSQL
CREATE FOREIGN TABLE customers ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=db.example.com port=5432 dbname=production user=reader password=secret',
table 'public.customers'
);
-- Query as if it were a local table (SQL is pushed to PostgreSQL)
SELECT * FROM customers WHERE country = 'USA';
-- Join with local data
SELECT
c.name,
c.email,
COUNT(o.id) AS order_count
FROM customers c
JOIN orders o ON c.id = o.customer_id
GROUP BY c.name, c.email;
-- Aggregation query (executed on PostgreSQL)
SELECT country, COUNT(*) AS customer_count
FROM customers
GROUP BY country
ORDER BY customer_count DESC;
```
**Connection String Format**:
```
host=hostname port=5432 dbname=database user=username password=secret
```
#### Connecting to Cognica via PostgreSQL Protocol
Cognica supports the PostgreSQL wire protocol, so you can connect to other Cognica instances using the PostgreSQL source type:
```sql
-- Connect to another Cognica instance via PostgreSQL protocol
CREATE FOREIGN TABLE remote_cognica_users ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=remote-cognica.example.com port=5432 dbname=mydb user=reader password=secret',
table 'public.users'
);
```
**When to use PostgreSQL vs Flight SQL for Cognica-to-Cognica connections**:
| Aspect | PostgreSQL Protocol | Flight SQL Protocol |
|--------|---------------------|---------------------|
| Data format | Row-based | Columnar (Apache Arrow) |
| Best for | Transactional queries, small result sets | Analytical queries, large scans |
| Serialization | PostgreSQL text/binary | Zero-copy Arrow buffers |
| Network efficiency | Standard | Higher for large datasets |
| Setup complexity | Simpler (standard port 5432) | Requires Flight SQL endpoint |
For high-volume analytical workloads or when transferring large result sets, consider using the `flight_sql_server` instead (see [Chapter 18](#chapter-18-foreign-tables---arrow-flight-sql)).
### 23.4 Connecting to MySQL
```sql
-- Create foreign table connected to MySQL
CREATE FOREIGN TABLE inventory ()
SERVER duckdb_server
OPTIONS (
source 'mysql',
connection 'host=mysql.internal port=3306 database=warehouse user=reader password=secret',
table 'inventory.stock_levels'
);
-- Query MySQL data (SQL pushed to MySQL)
SELECT category, COUNT(*) AS product_count, AVG(price) AS avg_price
FROM inventory
WHERE stock_quantity > 0
GROUP BY category
ORDER BY product_count DESC;
```
**Connection String Format**:
```
host=hostname port=3306 database=dbname user=username password=secret
```
### 23.5 Connecting to SQLite
```sql
-- Create foreign table from SQLite file
CREATE FOREIGN TABLE local_cache ()
SERVER duckdb_server
OPTIONS (
source 'sqlite',
path '/data/cache.db',
table 'cached_results'
);
-- Query SQLite data
SELECT * FROM local_cache WHERE expires_at > NOW();
```
### 23.6 Reading Delta Lake Tables
Delta Lake provides ACID transactions and time travel for data lakes:
```sql
-- Delta Lake table on S3
CREATE FOREIGN TABLE transactions ()
SERVER duckdb_server
OPTIONS (
source 'delta',
path 's3://data-lake/bronze/transactions'
);
-- Delta Lake table on local filesystem
CREATE FOREIGN TABLE local_delta ()
SERVER duckdb_server
OPTIONS (
source 'delta',
path '/data/delta-tables/events'
);
-- Query Delta Lake data
SELECT
DATE_TRUNC('day', event_time) AS day,
COUNT(*) AS event_count,
COUNT(DISTINCT user_id) AS unique_users
FROM transactions
WHERE event_time >= '2025-01-01'
GROUP BY 1
ORDER BY 1;
```
#### Delta Lake Time Travel
Query historical versions of your data:
```sql
-- Query a specific version number
SELECT * FROM transactions VERSION AS OF 5;
-- Query data at a specific point in time
SELECT * FROM transactions TIMESTAMP AS OF '2024-01-01';
-- Compare current data with historical version
SELECT
current.product_id,
current.amount AS current_amount,
historical.amount AS jan_amount,
current.amount - historical.amount AS change
FROM transactions current
JOIN transactions TIMESTAMP AS OF '2024-01-01' historical
ON current.transaction_id = historical.transaction_id;
```
### 23.7 Reading Apache Iceberg Tables
Apache Iceberg provides schema evolution and time travel:
```sql
-- Iceberg table on S3
CREATE FOREIGN TABLE events ()
SERVER duckdb_server
OPTIONS (
source 'iceberg',
path 's3://data-lake/events'
);
-- Query Iceberg data
SELECT event_type, COUNT(*) AS count
FROM events
WHERE timestamp >= '2025-01-01'
GROUP BY event_type;
```
### 23.8 Query Pushdown Optimization
DuckDB optimizes queries by pushing operations to external sources:
```sql
-- This query:
SELECT name, email FROM customers WHERE country = 'USA' AND active = true;
-- Is optimized to send to PostgreSQL:
-- SELECT name, email FROM public.customers WHERE country = 'USA' AND active = true
-- Only matching rows are transferred over the network
```
**What Gets Pushed Down**:
- WHERE conditions (equality, comparisons, IN, BETWEEN)
- Column selection (SELECT list projection)
- LIMIT and OFFSET clauses
- ORDER BY clauses
- Aggregations (COUNT, SUM, AVG, MIN, MAX)
**What Runs Locally**:
- Complex expressions not supported by the remote system
- Joins between foreign tables from different sources
- Joins between foreign and local tables
- Window functions (usually)
### 23.9 Federated Queries
Join data from multiple external sources in a single query:
```sql
-- Create foreign tables from different sources
CREATE FOREIGN TABLE pg_customers ()
SERVER duckdb_server
OPTIONS (source 'postgresql', connection '...', table 'customers');
CREATE FOREIGN TABLE mysql_products ()
SERVER duckdb_server
OPTIONS (source 'mysql', connection '...', table 'products');
CREATE FOREIGN TABLE delta_transactions ()
SERVER duckdb_server
OPTIONS (source 'delta', path 's3://bucket/transactions');
-- Cross-system analytical query
SELECT
c.country,
p.category,
SUM(t.quantity) AS total_quantity,
SUM(t.amount) AS total_revenue
FROM delta_transactions t -- Delta Lake
JOIN pg_customers c ON t.customer_id = c.id -- PostgreSQL
JOIN mysql_products p ON t.product_id = p.id -- MySQL
GROUP BY c.country, p.category
ORDER BY total_revenue DESC;
```
DuckDB's query optimizer plans the federated query, pushing predicates to sources where possible and choosing optimal join strategies.
### 23.10 Managing Foreign Tables
```sql
-- Drop a foreign table
DROP FOREIGN TABLE customers;
-- Drop only if exists (no error if not found)
DROP FOREIGN TABLE IF EXISTS customers;
```
### 23.11 Write Support
Write operations have limited support depending on the source:
| Source | INSERT | UPDATE | DELETE |
|--------|--------|--------|--------|
| PostgreSQL | Yes | Yes | Yes |
| MySQL | Yes | Yes | Yes |
| SQLite | Yes | Yes | Yes |
| Delta Lake | No (read-only) | No | No |
| Iceberg | No (read-only) | No | No |
| DuckDB file | Yes | Yes | Yes |
For file writes to cloud storage, use `file_server` foreign tables which support INSERT, UPDATE, DELETE through copy-on-write semantics.
### 23.12 Best Practices
**Performance**:
- Filter early to minimize data transfer over the network
- Use foreign tables for occasional queries; import frequently-accessed data
- DuckDB caches database attachments within a session to avoid redundant connections
- Use EXPLAIN to verify query pushdown is working
**Security**:
- Use read-only database accounts for external connections
- Credentials are stored in the foreign table definition - manage securely
- Consider SSL connections for sensitive data
**Reliability**:
- Foreign tables depend on external system availability
- Queries fail if the external source is unreachable
- Have fallback plans for when external systems are down
- Consider creating regular tables from foreign table snapshots for critical reports
---
## Chapter 24: Foreign Tables - Arrow Flight SQL
Cognica provides `flight_sql_server` for connecting to databases and analytics platforms that expose the Arrow Flight SQL protocol. Arrow Flight SQL is a high-performance wire protocol built on Apache Arrow that enables columnar data transfer, achieving 10-100x better throughput than traditional ODBC/JDBC for analytical workloads.
For connecting to traditional databases (PostgreSQL, MySQL, SQLite) and data lake formats (Delta Lake, Iceberg), see [Chapter 18: Foreign Tables - DuckDB Backend](#chapter-18-foreign-tables---duckdb-backend).
### 24.1 Understanding the flight_sql_server
#### What Is Arrow Flight SQL?
Arrow Flight SQL is a protocol that combines:
- **Apache Arrow**: Columnar memory format for efficient analytics
- **Arrow Flight**: High-performance data transfer over gRPC
- **SQL Semantics**: Standard database operations (queries, metadata, transactions)
Unlike row-by-row serialization in traditional database protocols, Flight SQL transfers data in Arrow's columnar format, enabling zero-copy reads and efficient processing.
#### When to Use flight_sql_server vs duckdb_server
| Use Case | Recommended Server | Reason |
|----------|-------------------|--------|
| PostgreSQL, MySQL | `duckdb_server` | Native protocol support, full SQL pushdown |
| Delta Lake, Iceberg | `duckdb_server` | DuckDB extension ecosystem |
| **ClickHouse** | `flight_sql_server` | Native Flight SQL support (v21.8+) |
| **Another Cognica instance** | `flight_sql_server` | Optimized for Cognica-to-Cognica federation |
| **Apache DataFusion/Ballista** | `flight_sql_server` | Primary query interface |
| **Dremio** | `flight_sql_server` | High-performance Flight SQL endpoint |
| Other Flight SQL servers | `flight_sql_server` | Standard protocol support |
#### Supported Flight SQL Sources
| Source Type | Description | Key Features |
|-------------|-------------|--------------|
| `cognica` | Another Cognica instance | Full SQL compatibility, optimal for federation |
| `clickhouse` | ClickHouse analytics database | Time-series analytics, columnar storage |
| `datafusion` | Apache DataFusion or Ballista | Distributed query execution |
| `dremio` | Dremio data platform | Data lake analytics, semantic layer |
| `generic` | Any Flight SQL-compatible server | Standard Flight SQL protocol |
### 24.2 CREATE FOREIGN TABLE Syntax
```sql
CREATE FOREIGN TABLE [IF NOT EXISTS] [schema_name.]table_name ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc://host:port',
source 'source_type',
catalog 'catalog_name',
schema 'schema_name',
table 'table_name',
[authentication options...],
[connection options...]
);
```
**Key Points**:
- The column list is **empty `()`** - schema is automatically inferred from the remote server
- `endpoint` specifies the Flight SQL server address (`grpc://` or `grpc+tls://`)
- `source` identifies the server type for SQL dialect adaptation
- Remote table coordinates use `catalog`, `schema`, and `table` options
#### Connection Options Reference
| Option | Description | Example |
|--------|-------------|---------|
| `endpoint` | Flight SQL server address | `grpc://host:9100` or `grpc+tls://host:31337` |
| `source` | Server type | `cognica`, `clickhouse`, `datafusion`, `dremio`, `generic` |
| `catalog` | Remote catalog name | `default` |
| `schema` | Remote schema name | `public` |
| `table` | Remote table name | `customers` |
| `token` | Bearer authentication token | `eyJhbGciOiJI...` |
| `username` | Basic auth username | `reader` |
| `password` | Basic auth password | `secret` |
| `tls_enabled` | Enable TLS encryption | `true` or `false` |
| `tls_root_certs` | Custom CA certificates (PEM) | Path or inline PEM |
| `timeout_ms` | Query timeout in milliseconds | `30000` |
### 24.3 Connecting to Cognica Instances
Connect to a remote Cognica instance for multi-region federation:
```sql
-- Connect to APAC region Cognica instance
CREATE FOREIGN TABLE apac_customers ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc+tls://apac.cognica.example.com:31337',
source 'cognica',
catalog 'default',
schema 'public',
table 'customers',
token 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...',
tls_enabled 'true'
);
-- Query remote customers as if they were local
SELECT customer_id, name, email
FROM apac_customers
WHERE region = 'APAC'
ORDER BY created_at DESC
LIMIT 100;
```
#### Flight SQL vs PostgreSQL Protocol
Cognica supports both Flight SQL and PostgreSQL wire protocols. You can also connect to Cognica using `duckdb_server` with the PostgreSQL source type (see [Section 16.3](#163-connecting-to-postgresql)).
**Choose Flight SQL when:**
- Transferring large result sets (millions of rows)
- Running analytical queries with full table scans
- Performance is critical (zero-copy Arrow buffers)
- You need columnar data format for downstream analytics
**Choose PostgreSQL protocol when:**
- Running transactional queries with small result sets
- Simpler setup is preferred (standard PostgreSQL client libraries)
- Existing PostgreSQL tooling must be reused
- Network environment restricts gRPC traffic
### 24.4 Connecting to ClickHouse
ClickHouse exposes Flight SQL natively since version 21.8:
```sql
-- Connect to ClickHouse time-series database
CREATE FOREIGN TABLE clickhouse_events ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc://clickhouse.internal:9100',
source 'clickhouse',
catalog 'default',
table 'system.events',
username 'reader',
password 'secret'
);
-- Query ClickHouse analytics data
SELECT
DATE_TRUNC('hour', event_time) AS hour,
event_type,
COUNT(*) AS event_count
FROM clickhouse_events
WHERE event_date >= '2025-01-01'
GROUP BY 1, 2
ORDER BY hour;
```
### 24.5 Connecting to DataFusion
Query distributed DataFusion/Ballista clusters:
```sql
-- Connect to Ballista distributed query engine
CREATE FOREIGN TABLE ballista_metrics ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc://ballista-scheduler:50050',
source 'datafusion',
table 'aggregated_metrics'
);
-- Leverage distributed compute for heavy aggregations
SELECT
region,
metric_name,
AVG(value) AS avg_value,
MAX(value) AS max_value
FROM ballista_metrics
WHERE timestamp >= NOW() - INTERVAL '1 day'
GROUP BY region, metric_name;
```
### 24.6 Connecting to Dremio
Access Dremio's semantic layer and data lake analytics:
```sql
-- Connect to Dremio data platform
CREATE FOREIGN TABLE dremio_sales ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc+tls://dremio.cloud.example.com:32010',
source 'dremio',
catalog 'Samples',
schema 'samples.dremio.com',
table 'NYC-taxi-trips',
token 'dremio_pat_xxx',
tls_enabled 'true'
);
-- Query NYC taxi data through Dremio
SELECT
passenger_count,
COUNT(*) AS trip_count,
AVG(trip_distance) AS avg_distance
FROM dremio_sales
WHERE pickup_datetime >= '2024-01-01'
GROUP BY passenger_count
ORDER BY trip_count DESC;
```
### 24.7 Generic Flight SQL Servers
Connect to any Flight SQL-compatible server:
```sql
-- Connect to a generic Flight SQL endpoint
CREATE FOREIGN TABLE generic_data ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc://flight-server:8815',
source 'generic',
table 'my_table',
timeout_ms '60000'
);
```
### 24.8 Query Examples
Once created, Flight SQL foreign tables work like any other table in your queries.
#### Simple Queries
```sql
-- Basic SELECT with filters (pushed to remote server)
SELECT customer_id, name, email
FROM apac_customers
WHERE status = 'active' AND created_at > '2024-01-01'
ORDER BY name
LIMIT 1000;
-- Aggregation (executed on remote Flight SQL server)
SELECT
country,
COUNT(*) AS customer_count,
SUM(total_purchases) AS total_revenue
FROM apac_customers
GROUP BY country
HAVING COUNT(*) > 100
ORDER BY total_revenue DESC;
```
#### Cross-Instance Federation
Join data from multiple Cognica instances:
```sql
-- Join local orders with remote customer data
SELECT
c.name AS customer_name,
c.email,
COUNT(o.order_id) AS order_count,
SUM(o.total) AS total_spent
FROM local_orders o
JOIN apac_customers c ON o.customer_id = c.customer_id
WHERE o.order_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY c.customer_id, c.name, c.email
HAVING COUNT(o.order_id) > 5
ORDER BY total_spent DESC;
```
#### Combining Flight SQL with DuckDB Sources
Mix Flight SQL and DuckDB foreign tables in the same query:
```sql
-- Create both types of foreign tables
CREATE FOREIGN TABLE clickhouse_events ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc://clickhouse:9100',
source 'clickhouse',
table 'events'
);
CREATE FOREIGN TABLE postgres_users ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=pg.internal dbname=app user=reader password=secret',
table 'users'
);
-- Join ClickHouse analytics with PostgreSQL user data
SELECT
u.username,
u.email,
COUNT(e.event_id) AS event_count,
MAX(e.event_time) AS last_activity
FROM clickhouse_events e
JOIN postgres_users u ON e.user_id = u.id
WHERE e.event_time >= NOW() - INTERVAL '7 days'
GROUP BY u.id, u.username, u.email
ORDER BY event_count DESC
LIMIT 100;
```
#### Subqueries with Remote Data
```sql
-- Find local products that sold well in remote regions
SELECT p.*
FROM local_products p
WHERE p.product_id IN (
SELECT DISTINCT product_id
FROM apac_orders
WHERE order_total > 1000
AND order_date >= '2025-01-01'
);
```
### 24.9 Query Pushdown Optimization
Flight SQL optimizes queries by pushing operations to the remote server:
```sql
-- This query:
SELECT name, email FROM apac_customers WHERE country = 'South Korea' AND status = 'active';
-- Sends to the remote Flight SQL server:
-- SELECT "name", "email" FROM "public"."customers" WHERE "country" = 'South Korea' AND "status" = 'active'
-- Only matching rows are transferred over the network
```
**What Gets Pushed to Remote Server**:
- WHERE conditions (equality, comparisons, IN, BETWEEN, LIKE)
- Column projection (SELECT list)
- ORDER BY and LIMIT clauses
- Aggregations (COUNT, SUM, AVG, MIN, MAX, etc.)
- GROUP BY and HAVING clauses
**What Executes Locally**:
- Joins between Flight SQL tables from different endpoints
- Joins between Flight SQL and local/DuckDB tables
- Complex expressions not supported by the remote system
- Window functions (in most cases)
### 24.10 Managing Foreign Tables
```sql
-- Drop a foreign table
DROP FOREIGN TABLE apac_customers;
-- Drop only if exists (no error if not found)
DROP FOREIGN TABLE IF EXISTS clickhouse_events;
```
Flight SQL foreign tables are currently read-only. Write operations (INSERT, UPDATE, DELETE) are not supported. To modify data in remote Flight SQL databases, connect directly to the remote system.
### 24.11 Best Practices
**Performance**:
- Filter early to minimize data transfer over the network
- Use foreign tables for occasional queries; import frequently-accessed data
- Flight SQL uses connection pooling for efficient client reuse
- Use EXPLAIN to verify query pushdown is working
- Prefer `grpc+tls://` endpoints for production deployments
- Set appropriate `timeout_ms` values based on expected query complexity
**Security**:
- Use token-based authentication (`token` option) when available
- Enable TLS encryption (`tls_enabled = 'true'`) for production deployments
- Use read-only service accounts on remote Flight SQL servers
- Credentials are stored in the foreign table definition - manage securely
- Consider using custom CA certificates (`tls_root_certs`) for internal PKI
**Reliability**:
- Flight SQL connections include automatic retry logic
- Set appropriate timeout values for network latency
- Foreign tables depend on remote server availability
- Consider creating regular tables from foreign table snapshots for critical reports
- Monitor connection pool statistics for capacity planning
### 24.12 Troubleshooting Foreign Tables
This section addresses common issues across all foreign table types.
#### Connection Issues
**Error: Connection refused**
The database server is not accepting connections from the Cognica server.
Solutions:
1. Verify the hostname and port are correct
2. Check that the database server is running
3. Verify firewall rules allow the connection
4. Check if SSL is required but not configured
**Error: Authentication failed**
The provided credentials are invalid or insufficient.
Solutions:
1. Verify username and password are correct
2. Check that the user has appropriate permissions
3. For PostgreSQL, verify `pg_hba.conf` allows the connection method
4. Check if password has special characters that need escaping
**Error: SSL required but not configured**
The database server requires SSL but the connection string doesn't enable it.
Solution:
```sql
-- Add sslmode to connection string for PostgreSQL
connection 'host=db.example.com sslmode=require ...'
-- Use grpc+tls:// for Flight SQL
endpoint 'grpc+tls://host:port'
```
#### Extension Issues (duckdb_server)
**Error: Extension "postgres" not found**
The required DuckDB extension is not installed.
Solutions:
1. Verify internet connectivity for automatic extension download
2. For air-gapped environments, pre-install extensions manually
3. Check DuckDB extension repository availability
**Error: Extension version mismatch**
The installed extension is incompatible with the DuckDB version.
Solution: Update the extension by reinstalling it. Extensions are typically updated when DuckDB is updated.
#### Cloud Storage Issues (file_server)
**Error: Access Denied to S3**
AWS credentials lack permission to access the bucket or object.
Solutions:
1. Verify IAM policy includes `s3:GetObject` and `s3:ListBucket`
2. Check bucket policy allows access from the server
3. Verify credentials are configured correctly
4. For cross-account access, check trust relationships
**Error: Invalid endpoint**
The cloud storage endpoint is unreachable or misconfigured.
Solutions:
1. Verify the endpoint URL is correct
2. Check network connectivity to the endpoint
3. For S3-compatible storage, verify `url_style_path` setting
#### Schema Issues
**Error: Column not found**
The query references a column that doesn't exist in the cached schema.
Solutions:
1. Verify the column name is spelled correctly
2. Check if the column was recently removed from the source
3. Refresh the schema: `ALTER FOREIGN TABLE table_name REFRESH SCHEMA`
**Error: Type mismatch**
A value cannot be converted to the expected type.
Solutions:
1. Check the type mapping between source and Cognica
2. Use explicit casts where necessary
3. Verify the source column type hasn't changed
#### Performance Issues
**Slow queries with no visible cause**
Predicates may not be pushing down, causing full table scans.
Solutions:
1. Use EXPLAIN to verify predicate pushdown
2. Simplify filter conditions to enable pushdown
3. Add indexes to the external source on filtered columns
**Memory errors on large results**
Query results exceed available memory.
Solutions:
1. Add filters to reduce result set size
2. Select fewer columns
3. Use pagination with LIMIT and OFFSET
4. Increase available memory or use streaming cursors
### 24.13 Real-World Examples
#### Example 1: E-commerce Analytics Platform
An e-commerce company needs to analyze customer behavior by combining transactional data from PostgreSQL with clickstream data from Delta Lake.
**Setup**:
```sql
-- PostgreSQL: Transactional data
CREATE FOREIGN TABLE pg_orders ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=orders-db.internal port=5432 dbname=ecommerce user=analytics password=secret sslmode=require',
table 'public.orders'
);
CREATE FOREIGN TABLE pg_customers ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=crm-db.internal port=5432 dbname=crm user=analytics password=secret sslmode=require',
table 'public.customers'
);
-- Delta Lake: Clickstream data
CREATE FOREIGN TABLE delta_clickstream ()
SERVER duckdb_server
OPTIONS (
source 'delta',
path 's3://data-lake/clickstream/events'
);
```
**Analysis Query**: Find customers who viewed products multiple times but haven't purchased:
```sql
WITH product_views AS (
SELECT
user_id,
product_id,
COUNT(*) AS view_count,
MAX(event_timestamp) AS last_view
FROM delta_clickstream
WHERE event_type = 'product_view'
AND event_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY user_id, product_id
HAVING COUNT(*) >= 3
),
recent_purchases AS (
SELECT DISTINCT customer_id, product_id
FROM pg_orders
WHERE order_date >= CURRENT_DATE - INTERVAL '30 days'
)
SELECT
c.email,
c.name,
pv.product_id,
pv.view_count,
pv.last_view
FROM product_views pv
JOIN pg_customers c ON pv.user_id = c.id
LEFT JOIN recent_purchases rp
ON pv.user_id = rp.customer_id AND pv.product_id = rp.product_id
WHERE rp.customer_id IS NULL
ORDER BY pv.view_count DESC
LIMIT 1000;
```
This query demonstrates joining three external sources (two PostgreSQL tables and one Delta Lake table) to identify potential customers for retargeting campaigns.
#### Example 2: Multi-Region Data Federation
A global company maintains separate databases in each region for compliance. They need unified reporting across all regions using Flight SQL.
**Setup**:
```sql
-- US Region Database (via Flight SQL)
CREATE FOREIGN TABLE us_sales ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc+tls://us-cognica.company.com:31337',
source 'cognica',
catalog 'sales',
schema 'public',
table 'sales',
token 'eyJhbGciOiJSUzI1NiIs...'
);
-- EU Region Database (via Flight SQL)
CREATE FOREIGN TABLE eu_sales ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc+tls://eu-cognica.company.com:31337',
source 'cognica',
catalog 'sales',
schema 'public',
table 'sales',
token 'eyJhbGciOiJSUzI1NiIs...'
);
-- APAC Region Database (via Flight SQL)
CREATE FOREIGN TABLE apac_sales ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc+tls://apac-cognica.company.com:31337',
source 'cognica',
catalog 'sales',
schema 'public',
table 'sales',
token 'eyJhbGciOiJSUzI1NiIs...'
);
```
**Global Report Query**:
```sql
SELECT
'US' AS region,
DATE_TRUNC('month', sale_date) AS month,
COUNT(*) AS transactions,
SUM(amount) AS revenue
FROM us_sales
WHERE sale_date >= '2024-01-01'
GROUP BY DATE_TRUNC('month', sale_date)
UNION ALL
SELECT
'EU' AS region,
DATE_TRUNC('month', sale_date) AS month,
COUNT(*) AS transactions,
SUM(amount) AS revenue
FROM eu_sales
WHERE sale_date >= '2024-01-01'
GROUP BY DATE_TRUNC('month', sale_date)
UNION ALL
SELECT
'APAC' AS region,
DATE_TRUNC('month', sale_date) AS month,
COUNT(*) AS transactions,
SUM(amount) AS revenue
FROM apac_sales
WHERE sale_date >= '2024-01-01'
GROUP BY DATE_TRUNC('month', sale_date)
ORDER BY month, region;
```
#### Example 3: Real-Time Dashboard with Historical Context
A dashboard application needs to show real-time metrics alongside historical trends from the data lake.
**Setup**:
```sql
-- Real-time data from PostgreSQL
CREATE FOREIGN TABLE pg_transactions ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=txn-db.internal dbname=transactions user=dashboard password=secret',
table 'public.transactions'
);
-- Historical aggregates from Delta Lake
CREATE FOREIGN TABLE delta_daily_metrics ()
SERVER duckdb_server
OPTIONS (
source 'delta',
path 's3://analytics/daily_metrics'
);
```
**Dashboard Query**: Compare today's metrics to the 30-day average:
```sql
WITH today_metrics AS (
SELECT
COUNT(*) AS transaction_count,
SUM(amount) AS total_amount,
AVG(amount) AS avg_amount
FROM pg_transactions
WHERE created_at >= CURRENT_DATE
AND created_at < CURRENT_DATE + INTERVAL '1 day'
),
historical_avg AS (
SELECT
AVG(transaction_count) AS avg_daily_transactions,
AVG(total_amount) AS avg_daily_amount,
AVG(avg_transaction_amount) AS avg_transaction_amount
FROM delta_daily_metrics
WHERE metric_date >= CURRENT_DATE - INTERVAL '30 days'
AND metric_date < CURRENT_DATE
)
SELECT
t.transaction_count AS today_transactions,
h.avg_daily_transactions AS avg_30day_transactions,
ROUND(100.0 * t.transaction_count / h.avg_daily_transactions - 100, 1) AS transactions_vs_avg_pct,
t.total_amount AS today_revenue,
h.avg_daily_amount AS avg_30day_revenue,
ROUND(100.0 * t.total_amount / h.avg_daily_amount - 100, 1) AS revenue_vs_avg_pct
FROM today_metrics t, historical_avg h;
```
#### Example 4: Hybrid Analytics with ClickHouse
Combine operational data from PostgreSQL with high-performance analytics from ClickHouse.
**Setup**:
```sql
-- ClickHouse: High-volume event data
CREATE FOREIGN TABLE ch_events ()
SERVER flight_sql_server
OPTIONS (
endpoint 'grpc://clickhouse.internal:9100',
source 'clickhouse',
table 'default.events',
username 'analytics',
password 'secret'
);
-- PostgreSQL: User metadata
CREATE FOREIGN TABLE pg_users ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=users-db.internal dbname=app user=reader password=secret',
table 'public.users'
);
-- Local: Product catalog
-- (using native Cognica collection for frequently-accessed reference data)
```
**Analytics Query**: Find power users by combining behavioral data with profile information:
```sql
WITH user_activity AS (
SELECT
user_id,
COUNT(*) AS event_count,
COUNT(DISTINCT DATE_TRUNC('day', event_time)) AS active_days,
MAX(event_time) AS last_activity
FROM ch_events
WHERE event_time >= NOW() - INTERVAL '90 days'
GROUP BY user_id
HAVING COUNT(*) >= 100
)
SELECT
u.email,
u.name,
u.signup_date,
a.event_count,
a.active_days,
ROUND(a.event_count::NUMERIC / a.active_days, 2) AS events_per_day,
a.last_activity
FROM user_activity a
JOIN pg_users u ON a.user_id = u.id
WHERE u.status = 'active'
ORDER BY a.event_count DESC
LIMIT 100;
```
This pattern leverages each system's strengths: ClickHouse for high-volume event aggregation, PostgreSQL for user metadata, and Cognica for query federation and local data.
---
## Chapter 25: COPY Operations - Bulk Data Import and Export
The COPY command provides high-performance bulk data transfer between Cognica and files. It is the standard approach for loading large datasets, migrating between databases, and creating backups.
### COPY vs. INSERT: Understanding the Performance Gap
Individual INSERT statements have overhead: parsing, planning, constraint checking, index updates, and transaction management per row. For interactive use, this overhead is negligible. For millions of rows, it becomes prohibitive.
COPY uses an optimized code path:
- Single parse and plan for the entire operation
- Batched constraint and trigger processing
- Efficient bulk index updates
- Minimal transaction logging overhead
The performance difference is dramatic: COPY can load 100,000 rows/second or more, while individual INSERTs typically handle 1,000-10,000 rows/second. For a 10 million row dataset, this means minutes versus hours.
### Error Handling Philosophy
COPY follows an all-or-nothing approach by default. If any row fails to parse or violates a constraint, the entire operation fails and no data is loaded. This behavior protects data integrity but requires clean input data.
For dirty data, you have options:
1. **Pre-validate**: Clean data before loading with external tools
2. **Staging tables**: Load into a table without constraints, then transform and move
3. **Error tolerance settings**: Some formats support skipping bad rows (with careful use)
### When to Use COPY
| Scenario | Use COPY | Use INSERT |
|----------|----------|------------|
| Loading >10,000 rows | Yes | No |
| Interactive single-row inserts | No | Yes |
| Data migration/backup | Yes | No |
| Application writes | No | Yes |
| ETL pipelines | Yes | No |
| Must capture per-row errors | No | Yes |
### 25.1 Understanding COPY
COPY operates in two directions:
- **COPY TO**: Export data from a table to a file
- **COPY FROM**: Import data from a file to a table
Performance advantage: COPY can load millions of rows per second, while individual INSERTs might handle thousands.
### 25.2 Exporting Data with COPY TO
#### Basic Export
```sql
-- Export entire table to CSV
COPY customers TO '/exports/customers.csv' WITH (FORMAT csv, HEADER true);
-- Export query results
COPY (SELECT * FROM orders WHERE created_at > '2024-01-01')
TO '/exports/recent_orders.csv'
WITH (FORMAT csv, HEADER true);
```
#### Export Formats
```sql
-- CSV (most common for interchange)
COPY products TO '/exports/products.csv'
WITH (
FORMAT csv,
HEADER true,
DELIMITER ',',
QUOTE '"',
ESCAPE '"',
NULL '' -- Empty string for NULLs
);
-- TSV (Tab-separated)
COPY logs TO '/exports/logs.tsv'
WITH (
FORMAT csv,
DELIMITER E'\t', -- Tab character
HEADER true
);
-- Binary (fastest, but not human-readable)
COPY large_table TO '/exports/data.bin'
WITH (FORMAT binary);
-- Parquet (columnar, compressed, great for analytics)
COPY analytics_data TO '/exports/data.parquet'
WITH (FORMAT parquet);
-- JSON Lines
COPY events TO '/exports/events.jsonl'
WITH (FORMAT json);
-- Arrow IPC (fastest for Arrow-native workflows, zero-copy reads)
COPY analytics_data TO '/exports/data.arrow'
WITH (FORMAT ipc);
-- ORC (Optimized Row Columnar, great for Hadoop/Spark ecosystems)
COPY warehouse_data TO '/exports/data.orc'
WITH (FORMAT orc);
```
#### Compressed Export
```sql
-- Automatically compress based on extension
COPY large_table TO '/exports/data.csv.gz'
WITH (FORMAT csv, HEADER true);
-- Creates a gzip-compressed file
COPY huge_table TO '/exports/data.csv.zst'
WITH (FORMAT csv, HEADER true);
-- Creates a zstd-compressed file (smaller, faster)
```
#### Partitioned Export
Cognica supports partitioned export using Apache Arrow Dataset, writing data into
directory structures organized by partition column values. This is essential for
data lake workflows and interoperability with Spark, Trino, and Hive.
```sql
-- Hive-style partitioning (key=value directory structure)
-- Produces: /exports/orders/year=2024/month=01/part-0.parquet
-- /exports/orders/year=2024/month=02/part-0.parquet
COPY (
SELECT *, EXTRACT(YEAR FROM created_at) AS year,
EXTRACT(MONTH FROM created_at) AS month
FROM orders
)
TO '/exports/orders/'
WITH (
FORMAT parquet,
PARTITIONING 'hive',
PARTITION_BY (year, month)
);
-- Directory-based partitioning (value-only directory names)
-- Produces: /exports/logs/2024/01/part-0.parquet
-- /exports/logs/2024/02/part-0.parquet
COPY (
SELECT *, EXTRACT(YEAR FROM ts) AS year,
EXTRACT(MONTH FROM ts) AS month
FROM logs
)
TO '/exports/logs/'
WITH (
FORMAT parquet,
PARTITIONING 'directory',
PARTITION_BY (year, month)
);
```
**Partitioning schemes:**
| Scheme | Directory Structure | Use Case |
|--------|-------------------|----------|
| `hive` | `column=value/` directories | Spark, Trino, Hive interoperability |
| `directory` | Value-only directories | Simpler structure, custom tooling |
### 25.3 Importing Data with COPY FROM
#### Basic Import
```sql
-- Import CSV file
COPY customers FROM '/imports/customers.csv'
WITH (FORMAT csv, HEADER true);
-- Import specific columns
COPY customers (name, email, created_at)
FROM '/imports/partial_customers.csv'
WITH (FORMAT csv, HEADER true);
```
#### Import Options
```sql
-- Handle various CSV formats
COPY products FROM '/imports/products.csv'
WITH (
FORMAT csv,
HEADER true,
DELIMITER ';', -- Semicolon-delimited
QUOTE '''', -- Single quote for quoting
NULL 'N/A', -- How NULLs are represented
ENCODING 'UTF8'
);
-- Import binary file (must match COPY TO binary format)
COPY large_table FROM '/imports/data.bin'
WITH (FORMAT binary);
-- Import Parquet
COPY analytics FROM '/imports/data.parquet'
WITH (FORMAT parquet);
-- Import Arrow IPC
COPY analytics FROM '/imports/data.arrow'
WITH (FORMAT ipc);
-- Import ORC
COPY warehouse_data FROM '/imports/data.orc'
WITH (FORMAT orc);
-- Import compressed file (auto-detected from extension)
COPY logs FROM '/imports/logs.csv.gz'
WITH (FORMAT csv, HEADER true);
-- Auto-detect format from file extension
-- Cognica infers the format (CSV, Parquet, JSON, IPC, ORC)
-- based on the file extension, so FORMAT can be omitted:
COPY analytics FROM '/imports/data.parquet';
-- Equivalent to: WITH (FORMAT parquet)
```
#### Error Handling
```sql
-- Skip malformed rows (log errors instead of failing)
COPY products FROM '/imports/messy_data.csv'
WITH (
FORMAT csv,
HEADER true,
ON_ERROR 'ignore' -- Skip bad rows
);
-- Limit errors before failing
COPY products FROM '/imports/data.csv'
WITH (
FORMAT csv,
HEADER true,
ON_ERROR 'ignore',
MAX_ERRORS 100 -- Fail if more than 100 errors
);
```
### 25.4 Streaming COPY (Client-Side Data)
For programmatic data loading, you can stream data directly:
```sql
-- In psql or applications:
\copy customers FROM STDIN WITH (FORMAT csv, HEADER true)
-- Then paste or pipe data...
-- Export to stdout for piping
\copy (SELECT * FROM orders) TO STDOUT WITH (FORMAT csv)
```
### 25.5 Performance Tips
#### Large Imports
```sql
-- Disable indexes during bulk load
-- (much faster, rebuild after)
DROP INDEX idx_products_name;
COPY products FROM '/imports/millions_of_products.csv'
WITH (FORMAT csv, HEADER true);
CREATE INDEX idx_products_name ON products (name);
-- Use ANALYZE after large imports
ANALYZE products;
```
#### Parallel Import
For very large files, split them and import in parallel:
```bash
# Split large file
split -l 1000000 huge_file.csv part_
# Import parts in parallel (from application)
# Each part can be COPY'd independently
```
#### Format Performance Comparison
| Format | Import Speed | Export Speed | File Size | Human-Readable | Best For |
|--------|-------------|--------------|-----------|----------------|----------|
| Binary | Fastest | Fastest | Medium | No | Cognica-to-Cognica transfers |
| Arrow IPC | Fastest | Fastest | Medium | No | Arrow-native workflows, zero-copy |
| Parquet | Fast | Fast | Smallest | No | Analytics, data lakes, long-term storage |
| ORC | Fast | Fast | Smallest | No | Hadoop/Spark ecosystems |
| CSV | Medium | Medium | Large | Yes | Universal interchange |
| JSON | Slower | Slower | Largest | Yes | Web APIs, human inspection |
### 25.6 Common COPY Patterns
#### Daily Export for Backup
```sql
-- Export with date in filename
COPY orders TO '/backups/orders_2024-12-25.csv.gz'
WITH (FORMAT csv, HEADER true);
```
#### ETL Pipeline
```sql
-- Load staging table
COPY staging_customers FROM '/etl/customers_extract.csv'
WITH (FORMAT csv, HEADER true);
-- Transform and load to production
INSERT INTO customers (id, name, email, created_at)
SELECT
id,
TRIM(name),
LOWER(email),
COALESCE(created_at, NOW())
FROM staging_customers
ON CONFLICT (id) DO UPDATE SET
name = EXCLUDED.name,
email = EXCLUDED.email;
-- Clean up
TRUNCATE staging_customers;
```
#### Export for Analysis
```sql
-- Export for data science work
COPY (
SELECT
o.id,
o.created_at,
o.total,
c.country,
c.age_group
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.created_at >= '2024-01-01'
)
TO '/analysis/orders_with_customers.parquet'
WITH (FORMAT parquet);
-- Parquet preserves types and compresses well
```
### 25.7 Filtering Rows During Import (COPY FROM ... WHERE)
COPY FROM supports a WHERE clause that filters rows during import, preventing unwanted data from ever entering the table. This is more efficient than loading everything and deleting rows afterward, because filtering happens at the Arrow scan level before document conversion.
```sql
-- Import only active customers
COPY customers FROM '/imports/all_customers.csv'
WITH (FORMAT csv, HEADER true)
WHERE status = 'active';
-- Import only recent orders
COPY orders FROM '/imports/orders.parquet'
WITH (FORMAT parquet)
WHERE order_date >= '2024-01-01';
-- Combine multiple filter conditions
COPY products FROM '/imports/catalog.csv'
WITH (FORMAT csv, HEADER true)
WHERE category IN ('electronics', 'books') AND price > 0;
```
The WHERE clause is converted to an Arrow compute expression for filter pushdown. For Parquet files, this enables row group skipping -- entire groups of rows are skipped without being read if the column statistics indicate no rows match the predicate. For CSV and other formats, filtering happens during the scan without buffering the entire file.
**Supported filter expressions:**
| Expression | Example |
|------------|---------|
| Comparison | `price > 100`, `status = 'active'` |
| Logical AND/OR | `price > 100 AND category = 'books'` |
| IS NULL / IS NOT NULL | `email IS NOT NULL` |
| IN lists | `region IN ('US', 'EU', 'APAC')` |
| Type casts | `CAST(year AS INTEGER) >= 2024` |
| Function calls | `length(name) > 0` |
### 25.8 Multi-File Import (Glob Patterns and Directories)
Cognica can import data from multiple files in a single COPY command using glob patterns or directory paths. This is essential for data lake workflows where data is spread across many files.
#### Glob Pattern Import
```sql
-- Import all Parquet files matching a pattern
COPY events FROM '/data/events/*.parquet'
WITH (FORMAT parquet);
-- Import all CSV files from a specific month
COPY logs FROM '/data/logs/2024-01-*.csv'
WITH (FORMAT csv, HEADER true);
-- Import JSON Lines files matching a pattern
COPY metrics FROM '/data/metrics/server_*.jsonl'
WITH (FORMAT json);
```
#### Directory Import
When the path ends with `/`, Cognica treats it as a directory and imports all files of the specified format:
```sql
-- Import all Parquet files in a directory
COPY sales FROM '/data/sales/'
WITH (FORMAT parquet);
-- Import all CSV files in a directory
COPY users FROM '/data/users/'
WITH (FORMAT csv, HEADER true);
```
#### Recursive Directory Scanning
For nested directory structures, use the RECURSIVE option to scan subdirectories:
```sql
-- Import all Parquet files recursively
COPY events FROM '/data/events/'
WITH (FORMAT parquet, RECURSIVE true);
-- Recursive with Hive partitioning discovery
COPY orders FROM '/data/orders/'
WITH (FORMAT parquet, PARTITIONING 'hive', RECURSIVE true);
```
#### Schema Evolution
When importing multiple files, different files may have different schemas (columns added or removed over time, or type changes). Cognica handles this transparently through schema unification:
- All unique columns from all files are included in the unified schema
- Missing columns are filled with NULL values
- Compatible types are promoted (e.g., int32 to int64, float32 to float64)
- Incompatible types fall back to string representation
```sql
-- Files from 2023 have columns: id, name, email
-- Files from 2024 have columns: id, name, email, phone, verified
-- All files are imported with the union of all columns
COPY customers FROM '/data/customers/'
WITH (FORMAT parquet, RECURSIVE true);
-- Result: id, name, email, phone, verified
-- 2023 rows will have NULL for phone and verified
```
### 25.9 Format Auto-Detection
Cognica can automatically detect the file format from the file extension, eliminating the need to specify FORMAT explicitly:
```sql
-- Format is detected from the .parquet extension
COPY products FROM '/imports/products.parquet'
WITH (FORMAT auto);
-- Format is detected from the .csv extension
COPY users FROM '/imports/users.csv'
WITH (FORMAT auto, HEADER true);
-- Works with COPY TO as well
COPY orders TO '/exports/orders.arrow'
WITH (FORMAT auto);
```
**Auto-detection mapping:**
| Extension | Detected Format |
|-----------|----------------|
| `.parquet`, `.pq` | Parquet |
| `.csv` | CSV |
| `.tsv`, `.txt` | Text (tab-delimited) |
| `.json`, `.jsonl`, `.ndjson` | JSON Lines |
| `.arrow`, `.ipc`, `.feather` | Arrow IPC |
| `.orc` | ORC |
### 25.10 Cloud Storage Support
COPY operations support reading from and writing to cloud storage systems using URI schemes. This enables direct data exchange with data lakes without intermediate local copies.
```sql
-- Import from Amazon S3
COPY events FROM 's3://my-bucket/data/events.parquet'
WITH (FORMAT parquet);
-- Import from Google Cloud Storage
COPY logs FROM 'gs://my-bucket/logs/2024/'
WITH (FORMAT json, RECURSIVE true);
-- Import from Azure Blob Storage
COPY metrics FROM 'abfs://container@account/metrics/*.parquet'
WITH (FORMAT parquet);
-- Import from HDFS
COPY warehouse FROM 'hdfs://namenode:8020/data/warehouse.parquet'
WITH (FORMAT parquet);
-- Export to S3
COPY (SELECT * FROM orders WHERE region = 'US')
TO 's3://my-bucket/exports/us_orders.parquet'
WITH (FORMAT parquet);
```
**Supported URI schemes:**
| Scheme | Storage System |
|--------|---------------|
| `s3://`, `s3a://` | Amazon S3 |
| `gs://`, `gcs://` | Google Cloud Storage |
| `abfs://`, `abfss://` | Azure Blob Storage |
| `hdfs://`, `viewfs://` | Hadoop HDFS |
| `file://` | Local filesystem (explicit) |
Cloud credentials are configured through environment variables or cloud provider SDK configuration (e.g., `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` for S3).
### 25.11 Security Considerations
#### Path Validation
All file paths in COPY operations are validated against the server's configured root directory. Path traversal attempts (e.g., `../../etc/passwd`) are rejected before any file I/O occurs.
#### COPY FROM/TO PROGRAM
The `COPY FROM PROGRAM` and `COPY TO PROGRAM` syntax is recognized by the parser for PostgreSQL compatibility but is not yet supported. Attempts to use it return an error:
```sql
-- This is parsed but rejected at execution time
COPY logs FROM PROGRAM 'cat /var/log/syslog';
-- ERROR: COPY FROM PROGRAM not yet supported
```
---
## Quick Reference
### Data Types
| Type | Description |
|------|-------------|
| INTEGER / BIGINT | 64-bit signed integer |
| REAL / DOUBLE PRECISION | 64-bit floating point |
| NUMERIC / DECIMAL | Numeric (stored as double) |
| TEXT / VARCHAR | Variable-length text |
| BOOLEAN | TRUE / FALSE / NULL |
| TIMESTAMP / TIMESTAMPTZ | Date and time (with/without timezone) |
| DATE / TIME | Date-only / time-only |
| INTERVAL | Time duration |
| JSON / JSONB | JSON document |
| UUID | 128-bit universally unique identifier |
| BYTEA | Binary data |
| ENUM | Custom value set (CREATE TYPE ... AS ENUM) |
| int4range, int8range, ... | Range types (integer, numeric, timestamp, date) |
| ARRAY | Array of any type (e.g., INTEGER[], TEXT[]) |
### Operators
| Operator | Description |
|----------|-------------|
| +, -, *, / | Arithmetic |
| % | Modulo |
| ^ | Power / exponentiation |
| &, \|, #, ~, <<, >> | Bitwise AND, OR, XOR, NOT, shift left/right |
| =, <>, <, >, <=, >= | Comparison |
| AND, OR, NOT | Logical |
| LIKE, ILIKE | Pattern matching |
| SIMILAR TO | Regex-style pattern matching |
| ~ / ~* / !~ / !~* | POSIX regex (case-sensitive / insensitive) |
| IN, NOT IN | Set membership |
| BETWEEN | Range test |
| IS NULL, IS NOT NULL | NULL test |
| @> (range/array) | Contains |
| <@ (range/array) | Contained by |
| && (range/array) | Overlap |
| \|\| (array) | Array concatenation |
| @@ | Full-text search match |
| <-> | Vector distance (L2) |
### Aggregate Functions
| Function | Description |
|----------|-------------|
| COUNT(*) | Count all rows |
| COUNT(col) | Count non-NULL values |
| SUM(col) | Sum of values |
| AVG(col) | Average (ignores NULLs) |
| MIN(col) | Minimum value |
| MAX(col) | Maximum value |
| STRING_AGG(col, sep) | Concatenate strings |
| ARRAY_AGG(col) | Collect into array |
| JSON_AGG(col) | Collect into JSON array |
| BOOL_AND(col) | TRUE if all values are TRUE |
| BOOL_OR(col) | TRUE if any value is TRUE |
| BIT_AND(col) | Bitwise AND of all values |
| BIT_OR(col) | Bitwise OR of all values |
| ANY_VALUE(col) | Arbitrary value from group |
| COVAR_POP(y, x) | Population covariance |
| COVAR_SAMP(y, x) | Sample covariance |
### Window Functions
| Function | Description |
|----------|-------------|
| ROW_NUMBER() | Unique sequential number |
| RANK() | Rank with gaps for ties |
| DENSE_RANK() | Rank without gaps |
| NTILE(n) | Divide into n groups |
| LAG(col, n) | Value n rows before |
| LEAD(col, n) | Value n rows after |
| FIRST_VALUE(col) | First value in frame |
| LAST_VALUE(col) | Last value in frame |
| NTH_VALUE(col, n) | Nth value in frame |
| PERCENT_RANK() | Relative rank (0 to 1) |
| CUME_DIST() | Cumulative distribution |
| SUM/AVG/etc OVER() | Aggregate as window |
### Key Functions
| Category | Functions |
|----------|-----------|
| String | LENGTH, UPPER, LOWER, TRIM, SUBSTRING, REPLACE, CONCAT, SPLIT_PART |
| Math | ABS, CEIL, FLOOR, ROUND, SQRT, POWER, LOG, LN, MOD, RANDOM |
| Date/Time | NOW, CURRENT_DATE, EXTRACT, DATE_TRUNC, AGE, DATE_PART |
| Conditional | COALESCE, NULLIF, GREATEST, LEAST, CASE...WHEN...END |
| JSON | ->, ->>, #>, jsonb_extract_path, jsonb_set, jsonb_each |
| Array | array_length, cardinality, array_position, array_append, unnest |
| Range | int4range, numrange, isempty, lower, upper, lower_inc, upper_inc |
| Sequence | nextval, currval, setval, lastval |
### COPY Quick Reference
| Operation | Syntax |
|-----------|--------|
| Export CSV | COPY table TO '/path.csv' WITH (FORMAT csv, HEADER true) |
| Import CSV | COPY table FROM '/path.csv' WITH (FORMAT csv, HEADER true) |
| Export Parquet | COPY table TO '/path.parquet' WITH (FORMAT parquet) |
| Import Parquet | COPY table FROM '/path.parquet' WITH (FORMAT parquet) |
| With filter | COPY table FROM '/path.csv' WITH (FORMAT csv) WHERE col > 0 |
| Glob import | COPY table FROM '/data/*.parquet' WITH (FORMAT parquet) |
| Directory import | COPY table FROM '/dir/' WITH (FORMAT parquet, RECURSIVE true) |
| Auto-detect format | COPY table FROM '/path.parquet' WITH (FORMAT auto) |
| Cloud storage | COPY table FROM 's3://bucket/data.parquet' WITH (FORMAT parquet) |
| Partitioned export | COPY (query) TO '/dir/' WITH (FORMAT parquet, PARTITIONING 'hive') |
### Transaction Quick Reference
| Command | Description |
|---------|-------------|
| BEGIN | Start transaction |
| COMMIT | Save changes |
| ROLLBACK | Discard changes |
| SAVEPOINT name | Create savepoint |
| ROLLBACK TO name | Rollback to savepoint |
| SET TRANSACTION ISOLATION LEVEL ... | Set isolation (READ COMMITTED, REPEATABLE READ, SERIALIZABLE) |
| PREPARE TRANSACTION 'id' | Prepare for two-phase commit |
| COMMIT PREPARED 'id' | Commit a prepared transaction |
| ROLLBACK PREPARED 'id' | Rollback a prepared transaction |
### DDL Quick Reference
| Command | Description |
|---------|-------------|
| CREATE TABLE | Create a new table |
| ALTER TABLE | Modify table structure |
| DROP TABLE | Remove a table |
| CREATE INDEX | Create an index (B-tree, FTS, HNSW) |
| DROP INDEX | Remove an index |
| CREATE VIEW | Create a view |
| CREATE MATERIALIZED VIEW | Create a materialized view |
| REFRESH MATERIALIZED VIEW | Refresh materialized view data |
| CREATE SCHEMA | Create a new schema |
| ALTER SCHEMA | Rename or change owner of a schema |
| DROP SCHEMA | Remove a schema |
| CREATE SEQUENCE | Create an auto-increment sequence |
| CREATE TYPE ... AS ENUM | Create an enumerated type |
| TRUNCATE | Remove all rows from a table |
### Common Patterns
**Pagination:**
```sql
SELECT * FROM items ORDER BY id LIMIT 20 OFFSET 40;
```
**Upsert:**
```sql
INSERT INTO t (id, val) VALUES (1, 'new')
ON CONFLICT (id) DO UPDATE SET val = EXCLUDED.val;
```
**Running Total:**
```sql
SELECT *, SUM(amount) OVER (ORDER BY date) FROM transactions;
```
**Top N Per Group:**
```sql
WITH ranked AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) AS rn
FROM products
)
SELECT * FROM ranked WHERE rn <= 3;
```
**LATERAL Join (Top N per row):**
```sql
SELECT d.name, t.title
FROM departments d
CROSS JOIN LATERAL (
SELECT title FROM employees WHERE dept_id = d.id ORDER BY salary DESC LIMIT 3
) t;
```
**Range Overlap:**
```sql
SELECT * FROM reservations
WHERE reservation_period && tsrange('2024-06-01', '2024-06-30');
```
---
## Appendix: Execution Engine Architecture
This appendix provides a detailed overview of Cognica's internal execution engine architecture. Understanding these components helps advanced users optimize their queries and provides insight into how SQL statements are transformed into results.
### A.1 CVM (Cognica Virtual Machine)
The CVM is Cognica's bytecode-based query execution engine. When you execute a SQL query, it is compiled into CVM bytecode and executed by the interpreter.
#### Architecture Overview
```mermaid
flowchart TB
subgraph Compilation["Compilation Pipeline"]
SQL["SQL Query"] --> Parser["Parser
(libpg_query)"]
Parser --> AST["Abstract Syntax Tree"]
AST --> Lowering["IR Lowering"]
Lowering --> IR["Intermediate Representation"]
IR --> Optimizer["Optimizer
(CSE, Constant Folding)"]
Optimizer --> Codegen["Bytecode Generator"]
Codegen --> Bytecode["CVM Bytecode Module"]
end
subgraph Execution["Execution Engine"]
Bytecode --> Interpreter["Scalar Interpreter"]
Bytecode --> VecInterp["Vectorized Interpreter"]
Bytecode --> JIT["JIT Compiler"]
end
```
#### Instruction Set
The CVM uses a register-based architecture with 16 general-purpose registers (R0-R15) and 8 floating-point registers (F0-F7). The instruction set includes:
| Category | Examples | Description |
|----------|----------|-------------|
| Data Movement | MOVE, LOAD_CONSTANT, LOAD_PARAM | Register and constant operations |
| Arithmetic | ADD_INT64, SUB_F64, MUL, DIV, MOD | Integer and floating-point math |
| Comparison | CMP_EQ, CMP_LT, CMP_GT | Comparison with result in register |
| String | STRING_LENGTH, STRING_CONCAT, LIKE | String manipulation |
| Document | GET_FIELD, SET_FIELD, DOCUMENT_NEW | JSON document operations |
| Cursor | CURSOR_OPEN, CURSOR_NEXT, CURSOR_CLOSE | Table iteration |
| Aggregation | AGG_INIT, AGG_ACCUMULATE, AGG_FINALIZE | GROUP BY operations |
| Window | WIN_BUF_NEW, WIN_COMPUTE | Window function support |
| Control Flow | JUMP, JUMP_FALSE, CALL, RETURN | Branching and function calls |
#### Optimization Levels
```sql
-- Optimization is automatic, but understanding levels helps debugging
-- O0: No optimization (fastest compilation)
-- O1: Basic (constant folding, dead code elimination) - DEFAULT
-- O2: Standard (+ CSE, strength reduction)
-- O3: Aggressive (+ inlining, register coalescing)
```
#### Viewing Bytecode
Use the `--show-bytecode` flag to inspect compiled bytecode:
```bash
bin/cognica db query sql "SELECT name FROM users WHERE age > 21" --show-bytecode
```
### A.2 JIT Compilation
Cognica includes a copy-and-patch JIT compiler that generates native machine code for hot code paths, providing significant performance improvements for compute-intensive queries.
#### JIT Architecture
The JIT uses a **tiered compilation** strategy:
```mermaid
flowchart LR
subgraph Tier0["Tier 0: Interpreter"]
Bytecode["CVM Bytecode"] --> Interp["Interpreter
+ Profiling"]
end
subgraph Tier1["Tier 1: Baseline JIT"]
Interp -->|"100 invocations"| BaseJIT["Baseline JIT
(~1ms/KB)"]
end
subgraph Tier2["Tier 2: Optimized JIT"]
BaseJIT -->|"1000 invocations"| OptJIT["Optimized JIT
(~10ms/KB)"]
end
BaseJIT --> Native1["Native Code
(3-5x faster)"]
OptJIT --> Native2["Native Code
(5-10x faster)"]
```
#### Supported Architectures
| Architecture | Status | File |
|--------------|--------|------|
| x86-64 | Fully Supported | `stencils_x86_64.cpp` |
| ARM64/AArch64 | Fully Supported | `stencils_aarch64.cpp` |
#### JIT-Compiled Operations
- Arithmetic operations (integer and floating-point)
- Comparison and logical operations
- Field access with inline caching
- Type conversions
- Aggregate functions (COUNT, SUM, MIN, MAX)
- String operations (length, contains, starts_with)
#### Performance Characteristics
| Workload | Interpreter | Baseline JIT | Optimized JIT |
|----------|-------------|--------------|---------------|
| Arithmetic loops | 1x | 3-5x | 5-10x |
| Expression evaluation | 1x | 2-3x | 3-4x |
| Aggregate operations | 1x | 2-3x | 3-5x |
| Field access heavy | 1x | 1.5-2x | 2-3x |
### A.3 Apache Arrow and Acero Integration
Cognica integrates Apache Arrow's Acero execution engine for columnar processing, particularly beneficial for analytical queries on large datasets.
#### When Acero Is Used
```mermaid
flowchart TB
Query["SQL Query"] --> Analyzer["Query Analyzer"]
Analyzer -->|"Foreign Table
+ Aggregates/Limits"| Acero["Acero Execution"]
Analyzer -->|"Regular Tables"| CVM["CVM Execution"]
Analyzer -->|"Complex JOINs"| Hybrid["Hybrid Execution"]
subgraph AceroPath["Acero Execution Path"]
Acero --> Scanner["Arrow Scanner
(Parquet/CSV)"]
Scanner --> Filter["Filter Pushdown"]
Filter --> Project["Column Pruning"]
Project --> Exec["Vectorized Execution"]
end
```
#### Acero Optimizations
| Optimization | Benefit | When Applied |
|--------------|---------|--------------|
| Column Pruning | 22x faster than post-scan | Foreign table queries with SELECT subset |
| Partition Pruning | Skips irrelevant files | Queries with partition key filters |
| Filter Pushdown | Reduces data scanned | WHERE clauses on foreign tables |
| Parallel Decoding | Multi-core utilization | Large Parquet files |
| Early Termination | Stops at LIMIT | LIMIT/OFFSET queries |
#### Data Flow
```mermaid
flowchart LR
Source["External Source
(Parquet/CSV)"] --> Scanner["Arrow Scanner"]
Scanner --> RecordBatch["RecordBatch
(Columnar)"]
RecordBatch --> Acero["Acero Pipeline"]
Acero --> ExecBatch["ExecBatch"]
ExecBatch --> Converter["Document Converter"]
Converter --> Documents["Result Documents"]
```
### A.4 DuckDB Integration
DuckDB is integrated as a specialized backend for foreign tables, enabling federated queries across external databases and data lake formats.
#### Supported Data Sources
| Source Type | Extension | Use Case |
|-------------|-----------|----------|
| PostgreSQL | `postgres` | Query remote PostgreSQL databases |
| MySQL | `mysql` | Query remote MySQL databases |
| SQLite | `sqlite` | Query local SQLite files |
| Delta Lake | `delta` | Query Delta Lake tables on S3/local |
| Apache Iceberg | `iceberg` | Query Iceberg tables |
| DuckDB Files | native | Query existing DuckDB databases |
#### Query Execution Flow
```mermaid
flowchart TB
Query["SELECT * FROM foreign_table
WHERE condition"] --> Provider["DuckDB Cursor Provider"]
Provider --> ExtLoad["Load Extension
(postgres/delta/etc)"]
ExtLoad --> Attach["ATTACH Database
(if needed)"]
Attach --> Translate["Translate to DuckDB SQL"]
Translate --> Execute["Execute Query"]
Execute --> Stream["Stream Results
(2048 rows/chunk)"]
Stream --> Convert["Convert to Documents"]
Convert --> Result["Return to Cognica"]
```
#### Query Pushdown
DuckDB foreign tables support intelligent query pushdown:
```sql
-- This query pushes the entire WHERE clause to DuckDB
SELECT customer_id, order_total
FROM postgres_orders
WHERE order_date > '2024-01-01'
AND status = 'completed'
ORDER BY order_total DESC
LIMIT 100;
-- Internally translated to:
-- SELECT customer_id, order_total FROM remote_db.orders
-- WHERE order_date > '2024-01-01' AND status = 'completed'
-- ORDER BY order_total DESC LIMIT 100
```
#### Resource Management
| Setting | Default | Description |
|---------|---------|-------------|
| Memory Limit | 1GB | Per-session memory limit |
| Connection Pool | 4 | Max concurrent connections |
| Chunk Size | 2048 | Rows per streaming chunk |
### A.5 CompositeRow: Zero-Copy JOIN Architecture
CompositeRow is a lightweight abstraction that enables efficient multi-table JOINs by avoiding document materialization until absolutely necessary.
#### The Problem CompositeRow Solves
Traditional JOIN execution creates intermediate documents at each join level:
```
-- Without CompositeRow (expensive):
A JOIN B: Copy A + Copy B -> New Document AB
AB JOIN C: Copy AB + Copy C -> New Document ABC
-- Total: 5 document copies for 3-way join
```
CompositeRow maintains references instead:
```
-- With CompositeRow (efficient):
A JOIN B: Reference to A + Reference to B -> CompositeRow {A, B}
{A,B} JOIN C: Add Reference to C -> CompositeRow {A, B, C}
-- Total: 0 document copies until final output
```
#### CompositeRow Structure
```mermaid
flowchart TB
subgraph CompositeRow["CompositeRow"]
Slot0["Slot 0: AliasedDocument
alias='u', doc=users_doc*"]
Slot1["Slot 1: AliasedDocument
alias='o', doc=orders_doc*"]
Slot2["Slot 2: AliasedDocument
alias='p', doc=products_doc*"]
end
Slot0 --> UsersDoc["Users Document
{id: 1, name: 'Alice'}"]
Slot1 --> OrdersDoc["Orders Document
{id: 101, user_id: 1}"]
Slot2 --> ProductsDoc["Products Document
{id: 50, price: 29.99}"]
```
#### Field Resolution
CompositeRow supports both qualified and unqualified field access:
```sql
-- Qualified access (O(1) after alias match):
SELECT u.name, o.amount, p.price
FROM users u
JOIN orders o ON u.id = o.user_id
JOIN products p ON o.product_id = p.id;
-- Unqualified access (O(slots) scan):
SELECT name, amount, price -- Finds first matching field
FROM users u
JOIN orders o ON u.id = o.user_id
JOIN products p ON o.product_id = p.id;
```
#### Materialization
CompositeRow is only materialized to a concrete Document when needed:
| Operation | Materialization |
|-----------|----------------|
| WHERE evaluation | No - uses field references |
| JOIN condition | No - uses field references |
| ORDER BY | Depends on sort key source |
| Final SELECT output | Yes - creates output document |
| Subquery parameter | Yes - if passed to subquery |
#### CVM Opcodes for CompositeRow
| Opcode | Description | Complexity |
|--------|-------------|-----------|
| COMPOSITE_NEW | Create empty CompositeRow | O(1) |
| COMPOSITE_ADD_SLOT | Add document reference | O(1) |
| COMPOSITE_GET_FIELD | Get field by qualified name | O(slots) |
| COMPOSITE_GET_SLOT | Get field by slot index | O(1) |
| COMPOSITE_MATERIALIZE | Convert to Document | O(fields) |
| COMPOSITE_CLEAR | Reset for reuse | O(1) |
### A.6 Execution Path Selection
Cognica automatically selects the optimal execution path based on query characteristics:
```mermaid
flowchart TB
Query["SQL Query"] --> Analyzer["Query Analyzer"]
Analyzer --> Q1{"Foreign Table
Query?"}
Q1 -->|Yes| Q2{"DuckDB
Source?"}
Q1 -->|No| CVM["CVM Execution"]
Q2 -->|Yes| DuckDB["DuckDB Execution"]
Q2 -->|No| Q3{"Large Dataset
+ Aggregates?"}
Q3 -->|Yes| Acero["Acero Execution"]
Q3 -->|No| Arrow["Arrow Streaming"]
subgraph Paths["Execution Paths"]
CVM --> CVMDesc["Best for: Regular tables,
complex expressions,
PL/pgSQL"]
DuckDB --> DuckDesc["Best for: External DBs,
Delta Lake, Iceberg"]
Acero --> AceroDesc["Best for: Large Parquet,
analytical aggregates"]
Arrow --> ArrowDesc["Best for: Streaming
foreign table scans"]
end
```
#### Execution Path Summary
| Path | Best For | Key Benefit |
|------|----------|-------------|
| CVM (Scalar) | Regular queries, complex logic | Full SQL support, debugging |
| CVM (Vectorized) | Batch processing | SIMD operations on columns |
| JIT | Hot loops, aggregates | 5-10x faster execution |
| Acero | Large Parquet files | Columnar processing, pushdown |
| DuckDB | External databases, data lakes | Federation, extension ecosystem |
#### Query Hints
While Cognica automatically selects execution paths, you can observe the chosen path using EXPLAIN:
```sql
EXPLAIN SELECT * FROM large_parquet_table WHERE region = 'US';
-- Shows: Foreign Scan using Acero with filter pushdown
EXPLAIN SELECT COUNT(*), AVG(price) FROM products GROUP BY category;
-- Shows: CVM execution with hash aggregation
```
### A.7 Server-Side Prepared Statements
Cognica supports the PostgreSQL Extended Query Protocol, enabling server-side prepared statements for improved performance and security.
#### How Prepared Statements Work
```mermaid
flowchart LR
subgraph Parse["Parse Phase"]
SQL["SQL with $1, $2, ..."] --> Plan["Query Plan"]
end
subgraph Bind["Bind Phase"]
Plan --> Bound["Bound Plan"]
Params["Parameter Values"] --> Bound
end
subgraph Execute["Execute Phase"]
Bound --> Results["Query Results"]
end
Plan -->|"Cached"| Cache["Plan Cache"]
Cache -->|"Reuse"| Bound
```
When a client sends a parameterized query, it goes through three phases:
1. **Parse**: The SQL is parsed and planned once, with parameter placeholders (`$1`, `$2`, ...) left unresolved. The resulting plan is cached.
2. **Bind**: Parameter values are bound to the cached plan, creating an executable statement.
3. **Execute**: The bound statement is executed, and results are returned.
#### Benefits
| Benefit | Description |
|---------|-------------|
| **Performance** | Query parsing and planning happen once; subsequent executions reuse the cached plan |
| **SQL Injection Prevention** | Parameters are never interpolated into SQL text, eliminating injection attacks |
| **Binary Transfer** | Parameters and results can use binary format, reducing serialization overhead |
| **Type Safety** | Parameter types are validated at bind time |
#### Client Usage
Prepared statements are used automatically by most PostgreSQL drivers:
```python
# Python (psycopg2) - automatically uses prepared statements
cursor.execute("SELECT * FROM users WHERE id = %s AND status = %s", (42, 'active'))
# The driver translates this to:
# Parse: SELECT * FROM users WHERE id = $1 AND status = $2
# Bind: $1 = 42, $2 = 'active'
# Execute: returns results
```
```javascript
// Node.js (node-postgres) - parameterized queries
const result = await client.query(
'SELECT * FROM orders WHERE customer_id = $1 AND total > $2',
[customerId, minTotal]
);
```
```java
// JDBC - PreparedStatement
PreparedStatement stmt = conn.prepareStatement(
"SELECT * FROM products WHERE category = ? AND price < ?"
);
stmt.setString(1, "Electronics");
stmt.setDouble(2, 999.99);
ResultSet rs = stmt.executeQuery();
```
#### Plan Cache Invalidation
Cached plans are automatically invalidated when:
- The schema of referenced tables changes (columns added/removed/modified)
- Indexes referenced by the plan are created or dropped
- The session ends
This ensures plans always reflect the current schema without manual intervention.
### A.8 In-Flight Query Cancellation
Cognica supports cancelling running queries without disconnecting the client session, following the PostgreSQL cancellation protocol.
#### Cancellation Mechanism
```mermaid
flowchart TB
subgraph Session1["Session 1 (Running Query)"]
Query["Long-running query"] --> CVM["CVM Interpreter"]
CVM -->|"Checks token every ~16K instructions"| Token["CancellationToken"]
end
subgraph Session2["Session 2 (Requesting Cancel)"]
Cancel["pg_cancel_backend(pid)"] --> Token
end
Token -->|"Cancelled"| Error["SQLSTATE 57014
query_canceled"]
```
The cancellation system uses a lightweight `CancellationToken` that is propagated through the entire execution stack:
1. **CVM Interpreter**: Checks the token periodically (approximately every 16,384 instructions)
2. **Physical Plan Operators**: Check the token every ~1,000 rows processed
3. **Cursor Operations**: Check before each batch fetch
This design balances responsiveness (queries are cancelled promptly) with overhead (cancellation checks are infrequent enough to not affect normal throughput).
#### Usage
```sql
-- From another session, cancel a query by its process ID
SELECT pg_cancel_backend(12345);
-- Find running queries and their PIDs
SELECT pg_backend_pid(); -- Get your own PID
```
When a query is cancelled, the client receives an error with SQLSTATE code `57014` (query_canceled). The session remains connected and can continue executing new queries.
#### Client-Side Cancellation
PostgreSQL client libraries also support cancellation through the wire protocol's Cancel Request mechanism:
```python
# Python (psycopg2) - Ctrl+C triggers cancellation
import signal
# The driver sends a Cancel Request to the server
```
```bash
# psql - pressing Ctrl+C sends a cancel request
psql> SELECT * FROM huge_table; -- Press Ctrl+C to cancel
Cancel request sent
ERROR: cancelling statement due to user request
```
### A.9 Vectorized Execution
Cognica supports vectorized batch execution for improved throughput on analytical workloads. Instead of processing one row at a time (the traditional Volcano model), vectorized execution processes batches of rows, enabling:
- **SIMD Operations**: Arithmetic and comparison operations on columnar batches using hardware SIMD instructions (via xsimd for portable vectorization across x86-64 and ARM64)
- **Cache Efficiency**: Columnar batch processing improves CPU cache utilization
- **Reduced Interpretation Overhead**: Per-batch dispatch instead of per-row dispatch
Vectorized execution is automatically selected when beneficial, typically for:
- Aggregate queries over large result sets
- Scan-heavy analytical queries
- Batch processing pipelines
The CVM seamlessly switches between scalar (row-at-a-time) and vectorized (batch-at-a-time) execution within the same query, choosing the optimal mode for each operator.
---
*This manual documents Cognica Database SQL features. For the latest updates, visit the official documentation.*
---
# External Virtual Tables
## Overview
External Virtual Tables represent one of the most powerful features in Cognica Database, fundamentally changing how organizations can interact with their data ecosystem. Rather than requiring all data to be imported into Cognica before it can be queried, External Virtual Tables allow you to query data exactly where it lives — whether that is in a PostgreSQL database, a MySQL server, an S3-hosted Delta Lake table, or a collection of Parquet files on Azure Blob Storage.
This capability addresses a critical challenge in modern data architectures: data is rarely centralized in a single system. Organizations typically have transactional data in relational databases, historical data in data lakes, log files on cloud storage, and analytics results in various formats. Traditional approaches require building ETL pipelines to move this data into a single query engine, which introduces latency, increases storage costs, and creates synchronization challenges.
External Virtual Tables eliminate these problems by bringing the query to the data rather than the data to the query. When you execute a SQL statement against an External Virtual Table, Cognica translates your query into the appropriate format for the target system, executes it remotely, and streams the results back — all transparently, as if you were querying a native Cognica collection.
### Why External Virtual Tables Matter
The value proposition of External Virtual Tables extends beyond simple convenience. Consider a typical enterprise scenario: your customer data lives in PostgreSQL, your clickstream events are stored in Delta Lake on S3, and your product catalog is managed in MySQL. Without External Virtual Tables, answering a question like "Which premium customers viewed products but did not purchase in the last 30 days?" would require:
1. Exporting data from each source system
2. Loading it into a common data warehouse
3. Running the analytical query
4. Maintaining synchronization as source data changes
With External Virtual Tables, this becomes a single SQL query that joins data from all three sources in real-time, always reflecting the current state of each system.
### Key Capabilities
External Virtual Tables provide a rich set of capabilities designed for production use:
**Federated Query Execution** enables you to write SQL queries that seamlessly join data from multiple external sources with each other and with native Cognica collections. The query optimizer understands the capabilities of each source and generates efficient execution plans that minimize data movement.
**Predicate Pushdown** ensures that filter conditions in your WHERE clauses are pushed down to the external data source whenever possible. This is critical for performance — rather than fetching an entire table and filtering locally, Cognica instructs the remote system to perform the filtering, dramatically reducing network transfer and processing time.
**Projection Pushdown** works similarly for column selection. When your query only references specific columns, Cognica requests only those columns from the external source, further reducing data transfer overhead.
**Automatic Schema Inference** eliminates the need to manually define table schemas. When you create an External Virtual Table, Cognica connects to the source, inspects its schema, and automatically maps the source types to Cognica types. This schema is cached locally for performance but can be refreshed when the source schema changes.
**Extension Auto-Installation** handles the complexity of DuckDB extension management transparently. When you first query a PostgreSQL-backed virtual table, Cognica automatically downloads, installs, and loads the required `postgres` extension. This happens once and is cached for subsequent queries.
**Connection Pooling and Caching** optimize repeated access to external sources. Database connections are pooled and reused across queries, and database attachments are cached to avoid redundant authentication handshakes.
---
## Architecture
Understanding the architecture of External Virtual Tables helps you make informed decisions about when and how to use them. Cognica implements External Virtual Tables through three distinct backends, each optimized for different types of data sources.
### The Three-Backend Design
The decision to use three separate backends — Apache Arrow for file-based sources, DuckDB for database and data lake sources, and Arrow Flight SQL for remote SQL databases — reflects a fundamental architectural principle: use the right tool for each job.
**The Arrow Backend** excels at reading columnar file formats like Parquet, ORC, and Arrow IPC. Apache Arrow provides highly optimized readers for these formats, with support for predicate pushdown directly into the file format's metadata and statistics. When you query a Parquet file, Arrow can skip entire row groups that don't match your filter conditions without reading the underlying data. The Arrow backend also handles partitioned datasets natively, understanding Hive-style partition layouts and applying partition pruning automatically.
**The DuckDB Backend** handles database connectivity and data lake formats. DuckDB's extension ecosystem provides battle-tested connectors for PostgreSQL, MySQL, and SQLite, as well as readers for Delta Lake and Apache Iceberg. By embedding DuckDB as a query processing engine, Cognica gains access to this entire ecosystem while maintaining a consistent SQL interface. DuckDB runs as an in-memory instance within the Cognica process, eliminating inter-process communication overhead.
**The Flight SQL Backend** enables federated querying of remote databases that implement the Arrow Flight SQL protocol. This includes other Cognica instances, ClickHouse, Apache DataFusion (and Ballista clusters), Dremio, and any server exposing a generic Flight SQL endpoint. Flight SQL transmits data in Apache Arrow columnar format over gRPC, achieving high throughput with minimal serialization overhead. Each virtual table definition specifies a `grpc://` or `grpc+tls://` endpoint along with optional authentication credentials, and Cognica translates SQL queries into the target server's dialect before execution.
### Query Execution Flow
When you execute a query against an External Virtual Table, the execution follows a carefully orchestrated flow designed to maximize efficiency:
First, the SQL parser analyzes your query and identifies which tables are referenced. For each table, the query planner consults the virtual table registry to determine whether it is a native collection, an Arrow-backed external table, a DuckDB-backed external table, or a Flight SQL-backed external table.
For DuckDB-backed sources, the planner extracts filter conditions that can be pushed down and generates a DuckDB SQL query. This query is executed against DuckDB's in-memory instance, which in turn connects to the external source, executes the translated query, and streams results back.
For Flight SQL-backed sources, Cognica translates the query into the target server's SQL dialect, sends it over gRPC to the Flight SQL endpoint, and streams Arrow-formatted results back. The query translator handles dialect differences for ClickHouse, DataFusion, Dremio, and generic Flight SQL servers.
For Arrow-backed sources, the planner creates an Arrow dataset scanner with the appropriate filters and projections, which reads directly from the file system or cloud storage.
Results from external sources are materialized as Cognica cursors, which can then participate in joins with other external sources or native collections. The query executor handles the orchestration, applying any remaining operations that could not be pushed down to the sources.
### The Cursor Provider Chain
Cognica's query execution system uses a chain-of-responsibility pattern for cursor creation. When the executor needs to read from a table, it asks the cursor provider chain, which consists of multiple providers, each handling a specific type of table:
```mermaid
flowchart LR
A[SystemCatalogCursorProvider] --> B[FlightSQLCursorProvider]
B --> C[DuckDBCursorProvider]
C --> D[ExternalTableCursorProvider]
D --> E[TransactionCursorProvider]
```
The **SystemCatalogCursorProvider** handles queries against system catalogs like `_sys.virtual_tables` and `_sys.sequences`. These are internal metadata tables that describe the database structure itself.
The **FlightSQLCursorProvider** intercepts requests for tables that are registered as Flight SQL-backed virtual tables. It checks the virtual table registry, and if the requested table is a Cognica, ClickHouse, DataFusion, Dremio, or generic Flight SQL source, it translates the query into the appropriate dialect, sends it to the remote endpoint over gRPC, and returns a cursor over the Arrow-formatted results.
The **DuckDBCursorProvider** intercepts requests for tables that are registered as DuckDB-backed virtual tables. It checks the virtual table registry, and if the requested table is a PostgreSQL, MySQL, Delta Lake, or Iceberg source, it generates the appropriate DuckDB query and returns a cursor over the results.
The **ExternalTableCursorProvider** handles Arrow-backed file sources. It creates Arrow dataset scanners for Parquet, CSV, ORC, and other file formats.
The **TransactionCursorProvider** is the final link in the chain, handling native Cognica collections. If no upstream provider claims a table, it must be a native collection, and this provider creates a cursor using the transaction's snapshot of the collection.
This chain architecture provides clean separation of concerns and makes it easy to add new types of external sources in the future.
---
## Supported Data Sources
Cognica supports a diverse range of external data sources, carefully selected to cover the most common enterprise data management scenarios. Each source type has specific characteristics, capabilities, and configuration requirements that you should understand to use them effectively.
### Database Sources
Database sources allow you to query tables in external relational database management systems. This is particularly valuable for accessing operational data that cannot be easily replicated or for querying systems of record in real-time.
**PostgreSQL** is the most feature-rich database source, leveraging DuckDB's `postgres` extension. This extension uses PostgreSQL's native wire protocol to establish connections and execute queries. It supports the full range of PostgreSQL data types including arrays, JSON, and user-defined types. The extension handles connection management, query translation, and result streaming automatically.
PostgreSQL virtual tables are ideal for accessing transactional data, master data, and any information maintained in PostgreSQL-based systems. Common use cases include querying CRM databases, ERP systems, and custom applications built on PostgreSQL.
**MySQL** support is provided through DuckDB's `mysql` extension. Like the PostgreSQL extension, it uses the native MySQL protocol for efficient communication. MySQL virtual tables work well for accessing e-commerce platforms, content management systems, and legacy applications that use MySQL as their data store.
One important consideration with MySQL is character set handling. The extension assumes UTF-8 encoding, which matches MySQL's default in modern versions. If you are connecting to a database with different encoding settings, you may need to configure character set conversion on the MySQL server side.
**SQLite** is unique among database sources because it operates on local database files rather than network connections. This makes it useful for accessing embedded databases, application-specific data stores, and database backups. SQLite virtual tables are read-only by default, protecting the source file from accidental modifications.
SQLite support is built into DuckDB without requiring an additional extension, making it immediately available without any download or installation step.
### Flight SQL Sources
Flight SQL sources allow you to query remote databases that implement the Apache Arrow Flight SQL protocol. Flight SQL is a wire protocol built on gRPC that transmits query results in Apache Arrow columnar format, achieving high throughput with minimal serialization overhead.
**Cognica** instances can be queried as Flight SQL sources, enabling cross-cluster federated queries. This is useful for organizations that run multiple Cognica deployments across regions or environments and need to join data across them.
**ClickHouse** is supported through Flight SQL, providing access to ClickHouse's high-performance analytics engine. The query translator handles ClickHouse-specific SQL dialect differences, including function name mappings, type casting syntax, and identifier quoting conventions.
**Apache DataFusion** (and Ballista clusters) can serve as Flight SQL endpoints. DataFusion is an extensible query engine written in Rust, and Ballista is its distributed execution framework. Querying DataFusion through Flight SQL provides access to data managed by Rust-based analytics pipelines.
**Dremio** is a data lakehouse platform that exposes a Flight SQL interface. Connecting to Dremio through Flight SQL provides access to curated datasets, virtual datasets, and Dremio's query acceleration capabilities.
**Generic Flight SQL** servers are supported for any database or service that implements the Flight SQL protocol. This catch-all category ensures forward compatibility with new systems that adopt the standard.
### Data Lake Formats
Data lake formats represent a significant evolution in how organizations store and manage large-scale analytical data. Unlike traditional file formats, data lake formats provide ACID transaction semantics, schema evolution, and time travel capabilities on top of object storage.
**Delta Lake** is an open-source storage layer developed by Databricks that brings reliability to data lakes. Delta Lake stores data as Parquet files with a transaction log that tracks all changes. This transaction log enables atomic writes, schema enforcement, and the ability to query historical versions of the data.
When you create a Delta Lake virtual table, Cognica uses DuckDB's `delta` extension to read the transaction log and identify which Parquet files constitute the current version of the table. Filter conditions are evaluated against Delta Lake's file-level statistics, enabling efficient partition pruning and data skipping.
Delta Lake virtual tables are particularly valuable for accessing data produced by Spark-based ETL pipelines, streaming ingestion systems, and any process that writes to Delta format. The ability to query Delta tables directly eliminates the need to maintain separate exports or copies of the data.
**Apache Iceberg** is another open table format, originally developed by Netflix and now an Apache project. Iceberg takes a different approach to metadata management, storing schema, partitioning, and snapshot information in a manifest structure that scales to extremely large tables.
Iceberg's hidden partitioning is particularly powerful — partition transforms are applied automatically based on the table's partitioning spec, so you don't need to include partition columns in your queries. For example, if a table is partitioned by month based on an event_timestamp column, Iceberg automatically applies partition pruning when you filter on event_timestamp, even though you never explicitly mention the partition.
The `iceberg` extension in DuckDB provides read access to Iceberg tables, supporting snapshot isolation and the ability to query specific snapshots for time travel analysis.
### File Formats
For simpler use cases where full data lake capabilities are not required, Cognica supports direct access to common file formats through the Arrow backend.
**Parquet** is a columnar storage format that has become the de facto standard for analytical workloads. Parquet files include rich metadata with column statistics, enabling predicate pushdown and efficient data skipping. The Arrow backend's Parquet reader is highly optimized, supporting parallel column reads and vectorized decompression.
Parquet virtual tables excel at querying exported data, archived datasets, and any data stored in Parquet format without the overhead of a full data lake format.
**CSV** (Comma-Separated Values) remains ubiquitous despite its limitations. The Arrow backend provides robust CSV parsing with automatic type inference, configurable delimiters, quote handling, and header detection. CSV virtual tables are useful for querying data exports, log files, and data from systems that only support text-based formats.
**ORC** (Optimized Row Columnar) is another columnar format, originally developed for Apache Hive. ORC provides similar benefits to Parquet with some differences in compression and encoding strategies. ORC virtual tables provide access to data produced by Hive-based systems and older Hadoop ecosystems.
**JSON** and **Newline-Delimited JSON** (NDJSON) formats are supported for querying semi-structured data. Each line in an NDJSON file is parsed as a separate JSON object, making this format suitable for log files, API response archives, and event streams.
**Arrow IPC** files contain data serialized in Arrow's native format. This format has minimal parsing overhead since the data is already in Arrow's in-memory representation. Arrow IPC virtual tables are ideal for data produced by other Arrow-based systems or for high-performance data exchange.
---
## SQL Syntax
External Virtual Tables are created, managed, and dropped using SQL Data Definition Language (DDL) statements. Cognica follows PostgreSQL's Foreign Data Wrapper (FDW) syntax, which provides a familiar and standardized interface for defining external table mappings.
### Creating Foreign Tables
The `CREATE FOREIGN TABLE` statement establishes a mapping between a virtual table name in Cognica and an external data source. The statement has the following general form:
```sql
CREATE FOREIGN TABLE [IF NOT EXISTS] [schema_name.]table_name ()
SERVER server_name
OPTIONS (
option_name 'option_value'
[, ...]
);
```
The empty parentheses after the table name indicate that the schema will be inferred from the external source. Unlike regular `CREATE TABLE` statements, you do not specify column definitions — Cognica automatically discovers the schema by querying the source.
The `IF NOT EXISTS` clause prevents an error if a virtual table with the same name already exists. This is useful in scripts and automation where the table may have been created in a previous run.
The `schema_name` prefix is optional and defaults to `public`. Using schemas helps organize virtual tables, especially when you have many external sources. For example, you might use `ext_pg.customers` for PostgreSQL tables and `ext_delta.events` for Delta Lake tables.
The `SERVER` clause specifies which type of backend should handle this virtual table. Use `duckdb_server` for database sources and data lake formats, `flight_sql_server` for Arrow Flight SQL sources, and `file_server` for file-based sources.
The `OPTIONS` clause contains key-value pairs that configure the connection to the external source. The required and available options depend on the source type and are detailed in the Data Source Configuration section.
### Dropping Foreign Tables
When you no longer need an External Virtual Table, you can remove it with the `DROP FOREIGN TABLE` statement:
```sql
DROP FOREIGN TABLE [IF EXISTS] [schema_name.]table_name;
```
This statement removes the virtual table definition from Cognica's metadata. It does not affect the underlying data source — your PostgreSQL tables, Delta Lake files, or Parquet datasets remain unchanged.
The `IF EXISTS` clause suppresses the error that would normally occur if the table does not exist. This is useful in cleanup scripts and idempotent operations.
### Altering Foreign Tables
The `ALTER FOREIGN TABLE` statement allows you to modify an existing virtual table. Currently, the primary operation supported is schema refresh:
```sql
ALTER FOREIGN TABLE [schema_name.]table_name REFRESH SCHEMA;
```
This statement reconnects to the external source, retrieves the current schema, and updates the cached schema in Cognica. Schema refresh is necessary when columns have been added, removed, or modified in the external source.
Future versions may support additional alter operations, such as changing connection parameters or modifying options without recreating the table.
---
## Data Source Configuration
Each type of external data source requires specific configuration options. This section provides detailed guidance on configuring each supported source type, including required options, optional settings, and example configurations.
### PostgreSQL Configuration
PostgreSQL virtual tables connect to PostgreSQL database servers using the native PostgreSQL wire protocol. This provides efficient, low-overhead access to PostgreSQL data with full support for PostgreSQL's rich type system.
To create a PostgreSQL virtual table, you must provide a connection string that specifies how to connect to the database server, and optionally the specific table to expose.
```sql
CREATE FOREIGN TABLE pg_customers ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=db.example.com port=5432 dbname=production user=reader password=secret',
table 'public.customers'
);
```
The `source` option must be set to `'postgresql'` or `'postgres'` to indicate that this virtual table should use the PostgreSQL backend.
The `connection` option contains a PostgreSQL connection string in the standard libpq format. Each parameter is specified as `key=value` pairs separated by spaces:
- `host` specifies the hostname or IP address of the PostgreSQL server. This can be a DNS name that resolves to the database server, an IP address, or `localhost` for local connections.
- `port` specifies the TCP port on which PostgreSQL is listening. The default PostgreSQL port is 5432, but many deployments use different ports for security or multi-instance configurations.
- `dbname` specifies the name of the database to connect to. A single PostgreSQL server can host multiple databases, and you must specify which one contains the table you want to query.
- `user` specifies the PostgreSQL role (user) to authenticate as. This role must have SELECT permission on the tables you want to query.
- `password` specifies the password for authentication. For production deployments, consider using server-side credential configuration rather than embedding passwords in SQL statements.
- `sslmode` controls SSL/TLS encryption for the connection. Options include `disable` (no encryption), `require` (encrypted but no certificate verification), `verify-ca` (verify the server certificate is signed by a trusted CA), and `verify-full` (verify the certificate and hostname match). For security, production deployments should use `verify-full` whenever possible.
The `table` option specifies the fully-qualified name of the table to expose, in the format `schema.table`. If omitted, you will need to specify the table name in your queries using DuckDB's attached database syntax, which is less convenient for most use cases.
Additional options that may be useful:
- `read_only` can be set to `'true'` or `'false'` to control whether write operations are allowed. The default is `'true'`, and currently all virtual tables are read-only regardless of this setting.
**Connection Pooling Behavior**: When you first query a PostgreSQL virtual table, Cognica attaches the PostgreSQL database to its internal DuckDB instance using the `ATTACH` command. This attachment is cached, so subsequent queries reuse the existing connection without re-authenticating. The attachment cache is maintained for the lifetime of the DuckDB instance, typically the lifetime of the Cognica process.
**Type Mapping**: PostgreSQL types are mapped to Cognica types according to the following rules:
- Integer types (`smallint`, `integer`, `bigint`) map to `Int64`
- Floating-point types (`real`, `double precision`) map to `Double`
- Numeric types (`numeric`, `decimal`) map to `Double` (note: this may lose precision for very large or very precise values)
- Character types (`varchar`, `text`, `char`) map to `String`
- Boolean type maps to `Bool`
- Timestamp types (`timestamp`, `timestamptz`) map to `Int64` as epoch milliseconds
- Date type maps to `String` in ISO format (YYYY-MM-DD)
- JSON types (`json`, `jsonb`) map to `Object`
- Array types map to `Array`
### MySQL Configuration
MySQL virtual tables connect to MySQL database servers using the MySQL wire protocol. The configuration is similar to PostgreSQL but uses MySQL-specific connection parameters.
```sql
CREATE FOREIGN TABLE mysql_orders ()
SERVER duckdb_server
OPTIONS (
source 'mysql',
connection 'host=mysql.example.com port=3306 database=ecommerce user=reader password=secret',
table 'orders'
);
```
The `source` option must be set to `'mysql'` to indicate MySQL backend usage.
The `connection` option uses MySQL connection string format, which differs slightly from PostgreSQL:
- `host` specifies the MySQL server hostname or IP address.
- `port` specifies the TCP port. The default MySQL port is 3306.
- `database` (note: not `dbname`) specifies the database name to connect to.
- `user` specifies the MySQL user for authentication.
- `password` specifies the user's password.
The `table` option specifies the table name. MySQL does not have schemas in the PostgreSQL sense — all tables exist directly within a database.
**Character Set Considerations**: The MySQL extension assumes UTF-8 encoding (specifically `utf8mb4` in MySQL terminology). If your MySQL database uses a different character set, you may encounter encoding issues. The recommended approach is to ensure your MySQL database uses `utf8mb4` for all text columns.
**Type Mapping**: MySQL types are mapped as follows:
- Integer types (`TINYINT`, `SMALLINT`, `MEDIUMINT`, `INT`, `BIGINT`) map to `Int64`
- Floating-point types (`FLOAT`, `DOUBLE`) map to `Double`
- Decimal types (`DECIMAL`, `NUMERIC`) map to `Double`
- Character types (`VARCHAR`, `TEXT`, `CHAR`) map to `String`
- Boolean type (`BOOLEAN`, `TINYINT(1)`) maps to `Bool`
- Date and time types map to `String` or `Int64` depending on the specific type
- JSON type maps to `Object`
### SQLite Configuration
SQLite virtual tables access SQLite database files directly from the file system. This provides a simple way to query embedded databases, backup files, and application-specific data stores.
```sql
CREATE FOREIGN TABLE sqlite_logs ()
SERVER duckdb_server
OPTIONS (
source 'sqlite',
connection '/var/data/application.db',
table 'logs'
);
```
The `source` option must be set to `'sqlite'`.
The `connection` option specifies the path to the SQLite database file. This should be an absolute path to ensure consistent behavior regardless of the current working directory. The Cognica server process must have read access to this file.
The `table` option specifies the table name within the SQLite database.
**File Access**: Unlike network-based database sources, SQLite operates directly on files. This means:
- The file must be accessible from the Cognica server's file system
- File permissions must allow read access
- The file should not be actively written to by another process during queries (SQLite's locking mechanisms provide some protection, but concurrent access can cause issues)
**No Extension Required**: SQLite support is built into DuckDB's core, so no extension download is required. This makes SQLite the fastest database source to use for the first time.
### DuckDB File Configuration
DuckDB file virtual tables allow you to query tables within DuckDB database files. This is useful for accessing analytics results, shared datasets, and data prepared by other DuckDB-based tools.
```sql
CREATE FOREIGN TABLE duckdb_metrics ()
SERVER duckdb_server
OPTIONS (
source 'duckdb',
filename '/data/analytics.duckdb',
table 'metrics'
);
```
The `source` option must be set to `'duckdb'`.
The `filename` option specifies the path to the DuckDB database file.
The `table` option specifies the table name within the DuckDB database. Use `schema.table` format if the table is in a non-default schema.
**Compatibility**: DuckDB database files are version-specific. The DuckDB library embedded in Cognica can read files created by the same or earlier versions of DuckDB, but may not be able to read files created by newer versions. If you encounter compatibility issues, you may need to export the data from the source database and reimport it with the embedded DuckDB version.
### Delta Lake Configuration
Delta Lake virtual tables query Delta format tables stored on local file systems or cloud object storage. Delta Lake provides ACID transactions, time travel, and schema evolution on top of Parquet files.
```sql
CREATE FOREIGN TABLE delta_events ()
SERVER duckdb_server
OPTIONS (
source 'delta',
filename 's3://data-lake/events'
);
```
The `source` option must be set to `'delta'` or `'deltalake'`.
The `filename` option specifies the path to the Delta table root directory. This can be a local file system path or a cloud storage URL:
- Local path: `/data/delta/events`
- S3 path: `s3://bucket-name/path/to/table`
- Azure path: `az://container-name/path/to/table`
- GCS path: `gs://bucket-name/path/to/table`
**Transaction Log Processing**: When you query a Delta Lake virtual table, the `delta` extension reads the transaction log (`_delta_log` directory) to determine which Parquet files comprise the current version of the table. This process respects Delta Lake's ACID semantics — you always see a consistent snapshot of the data, even if concurrent writes are happening.
**Statistics and Pruning**: Delta Lake stores file-level statistics in the transaction log, including min/max values for each column. The extension uses these statistics to skip files that cannot contain matching rows for your filter conditions. This optimization can dramatically reduce I/O for queries with selective filters.
**Partition Pruning**: If the Delta table is partitioned, the extension uses partition information from the transaction log to skip entire partitions that don't match your filter conditions. For example, if the table is partitioned by `date` and your query filters on `date >= '2024-01-01'`, partitions for earlier dates are not read at all.
**Cloud Storage Authentication**: Delta Lake tables on cloud storage require appropriate authentication configuration. See the Cloud Storage Integration section for details on configuring S3, Azure, and GCS credentials.
### Apache Iceberg Configuration
Apache Iceberg virtual tables query Iceberg format tables, providing access to data managed by Iceberg catalogs and written by engines like Spark, Trino, and Flink.
```sql
CREATE FOREIGN TABLE iceberg_sales ()
SERVER duckdb_server
OPTIONS (
source 'iceberg',
filename 's3://warehouse/sales'
);
```
The `source` option must be set to `'iceberg'`.
The `filename` option specifies the path to the Iceberg table's metadata location. This path points to the directory containing the Iceberg metadata files, typically ending in the table name.
**Metadata Structure**: Iceberg stores metadata in a hierarchy of manifest files that describe the table's schema, partitioning, and the data files that comprise the table. The `iceberg` extension reads this metadata to understand the table structure and identify which files need to be read for your query.
**Hidden Partitioning**: One of Iceberg's most powerful features is hidden partitioning. Unlike Hive-style partitioning where partition columns are separate columns in the data, Iceberg applies partition transforms (like extracting the month from a timestamp) automatically. When you filter on a source column, Iceberg recognizes when partition pruning can be applied even though you didn't reference the partition explicitly.
**Schema Evolution**: Iceberg tracks schema changes in its metadata, allowing columns to be added, removed, renamed, or have their types changed without rewriting data files. The extension handles these schema changes transparently, presenting a consistent view that reflects the current schema.
**Snapshot Isolation**: Each Iceberg table query operates against a specific snapshot of the table. By default, this is the current snapshot, but Iceberg's time travel capabilities allow querying historical snapshots (this feature may require additional DuckDB extension configuration).
### Flight SQL Configuration
Flight SQL virtual tables connect to remote databases over the Arrow Flight SQL protocol, which uses gRPC for transport and Apache Arrow for data serialization.
```sql
CREATE FOREIGN TABLE flight_analytics ()
SERVER flight_sql_server
OPTIONS (
source 'cognica',
endpoint 'grpc://analytics-cluster.internal:8815',
table 'public.daily_metrics',
username 'reader',
password 'secret'
);
```
The `source` option specifies the Flight SQL server type. Valid values are `'cognica'`, `'clickhouse'`, `'datafusion'`, `'dremio'`, and `'generic'`. This setting controls query dialect translation — each server type has its own SQL syntax conventions that Cognica handles automatically.
The `endpoint` option specifies the gRPC endpoint URL. Use `grpc://host:port` for unencrypted connections or `grpc+tls://host:port` for TLS-encrypted connections.
The `table` option specifies the remote table name, optionally qualified with catalog and schema.
**Authentication**: Flight SQL virtual tables support two authentication methods:
- **Bearer token**: Set the `auth_token` option to a bearer token string. This is commonly used with OAuth 2.0 or API key authentication.
- **Basic authentication**: Set the `username` and `password` options. The credentials are sent using the Flight SQL handshake mechanism.
```sql
-- Bearer token authentication
CREATE FOREIGN TABLE flight_dremio ()
SERVER flight_sql_server
OPTIONS (
source 'dremio',
endpoint 'grpc+tls://dremio.example.com:32010',
table 'production.sales',
auth_token 'eyJhbGciOiJIUzI1NiIs...'
);
```
**TLS Configuration**: For encrypted connections, the `grpc+tls://` scheme enables TLS automatically. If the server uses a certificate signed by a private CA, provide the CA certificate bundle via the `tls_root_certs` option (PEM format).
**Timeout**: The `timeout_ms` option controls the query timeout in milliseconds (default: 30000).
**Additional Options**:
- `catalog`: Remote catalog name (if the server supports multi-catalog queries)
- `schema`: Remote schema name
### File-Based Source Configuration
For simple file access without the overhead of data lake formats, Cognica provides direct support for common file formats through the Arrow backend.
**Parquet Files**:
```sql
CREATE FOREIGN TABLE parquet_logs ()
SERVER file_server
OPTIONS (
filename '/data/logs/*.parquet',
format 'parquet'
);
```
The `filename` option can include glob patterns (`*`, `**`) to match multiple files. All matching files are treated as a single logical table.
The `format` option must be set to `'parquet'`.
**CSV Files**:
```sql
CREATE FOREIGN TABLE csv_data ()
SERVER file_server
OPTIONS (
filename '/data/exports/data.csv',
format 'csv',
delimiter ',',
header 'true',
quote '"'
);
```
Additional CSV options include:
- `delimiter`: The character separating fields (default: `,`)
- `header`: Whether the first row contains column names (default: `true`)
- `quote`: The character used to quote fields containing special characters (default: `"`)
- `escape`: The character used to escape quotes within quoted fields (default: same as quote)
- `null_value`: The string representing NULL values (default: empty string)
**Partitioned Datasets**:
For datasets organized into directory hierarchies based on column values (Hive-style partitioning), use the `partitioning` option:
```sql
CREATE FOREIGN TABLE partitioned_events ()
SERVER file_server
OPTIONS (
filename 's3://data/events/year=*/month=*/day=*/*.parquet',
format 'parquet',
partitioning 'hive'
);
```
The `partitioning` option can be:
- `'hive'`: Partitions are encoded as `column=value` directory names
- `'directory'`: Partitions are plain directory names without the `column=` prefix
- `'none'`: No partitioning (default)
When partitioning is enabled, filter conditions on partition columns are used to prune directories, avoiding the need to scan files in non-matching partitions.
---
## Cloud Storage Integration
Modern data architectures increasingly rely on cloud object storage as the foundation for data lakes. Cognica supports the three major cloud storage platforms — Amazon S3, Azure Blob Storage, and Google Cloud Storage — allowing you to query data wherever it resides.
### Understanding Cloud Storage Access
Cloud object storage differs fundamentally from traditional file systems. Objects are accessed via HTTP/HTTPS protocols, authentication is handled through provider-specific mechanisms, and performance characteristics are optimized for throughput rather than latency.
Cognica handles these differences transparently, but understanding the underlying mechanisms helps you configure and optimize cloud storage access.
**Authentication Flow**: When you query a virtual table backed by cloud storage, Cognica uses the configured credentials to sign HTTP requests. Each cloud provider has its own signing mechanism:
- S3 uses AWS Signature Version 4, which signs requests using access keys
- Azure uses Shared Access Signatures (SAS) or account keys
- GCS uses OAuth 2.0 tokens or HMAC keys
Credentials are configured at the server level rather than per-table, so all virtual tables accessing the same cloud provider share the same credentials.
**Performance Considerations**: Cloud storage is optimized for high-throughput sequential access. Parquet and other columnar formats are well-suited to this access pattern because they allow reading specific columns without scanning entire files. Row-oriented formats like CSV may perform poorly on cloud storage due to the need to read entire files even when only a few columns are needed.
### Amazon S3 Configuration
Amazon S3 is the most widely used cloud object storage service. Cognica supports S3 through DuckDB's `httpfs` extension, which handles authentication, connection management, and retry logic.
**Server Configuration**: Configure S3 credentials in your Cognica server configuration file:
```yaml
duckdb:
s3:
region: us-west-2
access_key_id: AKIAIOSFODNN7EXAMPLE
secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
The `region` setting specifies the AWS region for API calls. While S3 bucket names are globally unique, API endpoints are regional. Setting the correct region ensures requests are routed efficiently.
The `access_key_id` and `secret_access_key` are IAM credentials that authenticate requests. These credentials must have permissions to read the S3 buckets and objects you want to query. At minimum, the IAM policy must allow:
- `s3:GetObject` on the objects you want to read
- `s3:ListBucket` on the buckets containing those objects
**Temporary Credentials**: For enhanced security, you can use temporary credentials that automatically expire:
```yaml
duckdb:
s3:
region: us-west-2
access_key_id: ASIATEMP...
secret_access_key: temporary-secret...
session_token: FwoGZXIvYXdzEBY...
```
Temporary credentials are obtained from AWS STS (Security Token Service) and include a session token in addition to the access key and secret. They are particularly useful for:
- Applications running on EC2 with instance profiles
- Lambda functions with execution role credentials
- Cross-account access using AssumeRole
**S3-Compatible Storage**: Many organizations use S3-compatible storage systems like MinIO, Ceph, or NetApp StorageGRID. These systems implement the S3 API but run on-premises or in private clouds.
```yaml
duckdb:
s3:
region: us-east-1
endpoint: minio.internal.company.com:9000
access_key_id: minioadmin
secret_access_key: minioadmin
use_ssl: false
url_style_path: true
```
The `endpoint` setting overrides the default S3 endpoint with your storage system's URL.
The `use_ssl` setting controls whether HTTPS is used. For internal networks, you might disable SSL for performance, though this is not recommended for production.
The `url_style_path` setting switches from virtual-hosted-style URLs (`bucket.s3.region.amazonaws.com`) to path-style URLs (`s3.region.amazonaws.com/bucket`). Most S3-compatible systems require path-style URLs.
**Creating S3-Backed Virtual Tables**:
```sql
-- Query Parquet files on S3
CREATE FOREIGN TABLE s3_logs ()
SERVER file_server
OPTIONS (
filename 's3://logs-bucket/application/2024/**/*.parquet',
format 'parquet',
partitioning 'hive'
);
-- Query Delta Lake table on S3
CREATE FOREIGN TABLE delta_events ()
SERVER duckdb_server
OPTIONS (
source 'delta',
filename 's3://data-lake/bronze/events'
);
```
### Azure Blob Storage Configuration
Azure Blob Storage is Microsoft's cloud object storage service. Cognica supports Azure through DuckDB's `azure` extension, which handles Azure-specific authentication and the Azure Blob Storage REST API.
**Server Configuration**: Configure Azure credentials using one of several authentication methods:
**Account Key Authentication** (simplest, but keys don't expire):
```yaml
duckdb:
azure:
account_name: mystorageaccount
account_key: base64encodedaccountkey==
```
The account key provides full access to all containers and blobs in the storage account. While convenient, account keys are long-lived and should be protected carefully.
**Connection String Authentication**:
```yaml
duckdb:
azure:
connection_string: 'DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=base64key==;EndpointSuffix=core.windows.net'
```
Connection strings bundle all authentication parameters into a single string, which can be convenient for configuration management.
**Shared Access Signature (SAS) Authentication**:
```yaml
duckdb:
azure:
account_name: mystorageaccount
sas_token: 'sv=2021-06-08&ss=b&srt=sco&sp=rl&se=2024-12-31T23:59:59Z&st=2024-01-01T00:00:00Z&spr=https&sig=signature...'
```
SAS tokens provide scoped, time-limited access to specific containers or blobs. They are the recommended approach for production deployments because:
- Permissions can be limited to specific operations (read, list, etc.)
- Tokens expire automatically, limiting the impact of credential compromise
- Different tokens can be issued for different purposes with different permissions
**Creating Azure-Backed Virtual Tables**:
```sql
-- Query Parquet files on Azure Blob Storage
CREATE FOREIGN TABLE azure_logs ()
SERVER file_server
OPTIONS (
filename 'az://logs-container/application/**/*.parquet',
format 'parquet'
);
-- Query Delta Lake table on Azure Data Lake Storage Gen2
CREATE FOREIGN TABLE delta_customers ()
SERVER duckdb_server
OPTIONS (
source 'delta',
filename 'abfss://data@mystorageaccount.dfs.core.windows.net/customers'
);
```
### Google Cloud Storage Configuration
Google Cloud Storage (GCS) is Google's cloud object storage service. Cognica supports GCS through DuckDB's `httpfs` extension, which can use GCS-compatible authentication.
**Server Configuration**:
```yaml
duckdb:
gcs:
project_id: my-gcp-project
service_account_json: /path/to/service-account-key.json
```
The `project_id` identifies your GCP project for billing and quota purposes.
The `service_account_json` path points to a JSON key file for a GCP service account. This service account must have appropriate IAM permissions:
- `roles/storage.objectViewer` for read access
- Or specific permissions: `storage.objects.get`, `storage.objects.list`
**Application Default Credentials**: If running on GCP (Compute Engine, GKE, Cloud Run, etc.), you can rely on application default credentials instead of explicit service account keys. The environment automatically provides credentials based on the attached service account.
**Creating GCS-Backed Virtual Tables**:
```sql
-- Query Parquet files on GCS
CREATE FOREIGN TABLE gcs_events ()
SERVER file_server
OPTIONS (
filename 'gs://events-bucket/2024/**/*.parquet',
format 'parquet'
);
-- Query Iceberg table on GCS
CREATE FOREIGN TABLE iceberg_inventory ()
SERVER duckdb_server
OPTIONS (
source 'iceberg',
filename 'gs://warehouse-bucket/inventory'
);
```
---
## Query Optimization
External Virtual Tables can involve significant network I/O and remote computation. Understanding how Cognica optimizes queries against external sources helps you write efficient queries and diagnose performance issues.
### Predicate Pushdown in Depth
Predicate pushdown is the most important optimization for External Virtual Tables. When filter conditions can be evaluated at the source, the amount of data transferred over the network can be reduced by orders of magnitude.
**How Predicate Pushdown Works**: When Cognica's query planner encounters a query against a DuckDB-backed virtual table, it analyzes the WHERE clause to identify conditions that can be expressed in the source's query language. These conditions are then incorporated into the query sent to the external source.
For example, consider this query:
```sql
SELECT customer_id, order_total
FROM pg_orders
WHERE order_date >= '2024-01-01'
AND status = 'completed'
AND region IN ('US', 'EU');
```
The planner recognizes that all three conditions can be expressed in PostgreSQL SQL. It generates:
```sql
SELECT "customer_id", "order_total"
FROM orders_db.public.orders
WHERE order_date >= '2024-01-01'
AND status = 'completed'
AND region IN ('US', 'EU')
```
This query runs on the PostgreSQL server, which can use its indexes on `order_date`, `status`, and `region` to efficiently find matching rows.
**Pushdown-Compatible Conditions**: The following types of conditions can typically be pushed down:
Comparison operators (`=`, `<>`, `<`, `>`, `<=`, `>=`) on columns with compatible types push down directly. The values are translated to the appropriate format for the target system.
`BETWEEN` conditions push down as compound conditions: `x BETWEEN a AND b` becomes `x >= a AND x <= b`.
`IN` lists push down for reasonable list sizes. Very long `IN` lists may be handled differently depending on the source system's capabilities.
`LIKE` patterns push down to sources that support pattern matching. The wildcard characters (`%`, `_`) are translated appropriately.
`IS NULL` and `IS NOT NULL` conditions push down to check for null values.
`AND` and `OR` combinations push down with their structure preserved. The entire boolean expression tree is translated.
**Conditions That May Not Push Down**: Some conditions cannot be pushed down due to differences between Cognica's functions and the source system:
User-defined functions that don't have equivalents in the source system must be evaluated locally.
Complex expressions involving multiple columns in ways the source doesn't support are evaluated locally.
Conditions referencing columns from other tables in a join are typically evaluated locally after the data is fetched.
**Verifying Pushdown**: Use `EXPLAIN` to see which conditions are pushed down:
```sql
EXPLAIN SELECT * FROM pg_orders WHERE order_date >= '2024-01-01';
```
The execution plan shows the query sent to the external source, including any pushed-down predicates.
### Projection Pushdown
Projection pushdown reduces data transfer by requesting only the columns needed for the query. This optimization is particularly important for wide tables with many columns.
**Automatic Column Selection**: Cognica's planner analyzes which columns are referenced in:
- The SELECT list
- The WHERE clause
- JOIN conditions
- GROUP BY and ORDER BY clauses
- Expressions and function arguments
Only these columns are requested from the external source. For example:
```sql
SELECT customer_id, SUM(order_total)
FROM pg_orders
WHERE status = 'completed'
GROUP BY customer_id;
```
Only `customer_id`, `order_total`, and `status` are requested, even if the `pg_orders` table has dozens of additional columns.
**Impact on Performance**: Projection pushdown can dramatically improve performance for tables with:
- Many columns (reduces serialization/deserialization overhead)
- Large text or binary columns (reduces data transfer)
- Computed columns (avoids unnecessary computation at the source)
### Join Optimization Strategies
When queries involve joins between external sources and native collections, or between multiple external sources, the optimizer must decide how to execute the join efficiently.
**Local Join Execution**: In most cases, data from external sources is materialized locally, and joins are executed by Cognica's query engine. This approach:
- Allows joining heterogeneous sources (PostgreSQL with Delta Lake, etc.)
- Applies Cognica's join algorithms (hash join, merge join, etc.)
- Handles type conversions and null semantics consistently
**Optimizing Join Performance**: Several strategies improve join performance with external sources:
**Filter Early**: Apply filters to external tables before joining. This reduces the amount of data transferred and materialized:
```sql
-- Good: Filter on external table before join
SELECT c.name, o.total
FROM pg_customers c
JOIN orders o ON c.id = o.customer_id
WHERE c.region = 'US' -- Pushed to PostgreSQL
AND o.order_date >= '2024-01-01'; -- Applied to native collection
```
**Select Needed Columns**: Only select columns you need. This applies projection pushdown and reduces memory usage during join processing:
```sql
-- Good: Select specific columns
SELECT c.name, c.email, o.order_id, o.total
FROM pg_customers c
JOIN orders o ON c.id = o.customer_id;
-- Avoid: Selecting all columns
SELECT *
FROM pg_customers c
JOIN orders o ON c.id = o.customer_id;
```
**Consider Join Order**: When joining multiple external sources, consider which source is smaller or more selective:
```sql
-- If pg_regions is small, reference it first
SELECT r.region_name, SUM(o.total)
FROM pg_regions r
JOIN pg_orders o ON r.region_code = o.region
WHERE r.is_active = true
GROUP BY r.region_name;
```
### Statistics and Cost Estimation
Cognica's query optimizer uses statistics to estimate the cost of different execution plans. For External Virtual Tables, statistics collection is limited compared to native collections.
**Available Statistics**: For DuckDB-backed sources, limited statistics may be available from the source's metadata:
- Row count estimates from system catalogs
- Column cardinality estimates where available
- Index information for predicate pushdown decisions
For file-based sources, statistics may include:
- Total row count from file metadata
- Min/max values from Parquet/ORC column statistics
- Partition statistics for partitioned datasets
**Improving Estimates**: When automatic statistics are insufficient, you can help the optimizer by:
- Adding filters that reduce the result set size
- Breaking complex queries into simpler stages
- Using CTEs (Common Table Expressions) to materialize intermediate results
---
## Schema Management
External data sources have their own schemas that may change over time. Cognica provides mechanisms for discovering, caching, and refreshing schemas to maintain consistency between virtual table definitions and their underlying sources.
### Automatic Schema Inference
When you create an External Virtual Table without explicitly defining columns, Cognica connects to the source and infers the schema. This process varies by source type:
**Database Sources**: For PostgreSQL, MySQL, and SQLite, Cognica queries the database's system catalogs to retrieve column names, data types, nullability constraints, and other metadata. This information is translated to Cognica's type system and stored in the virtual table definition.
**Data Lake Sources**: For Delta Lake and Iceberg, schema information is read from the format's metadata:
- Delta Lake stores schema in the transaction log (`_delta_log`)
- Iceberg stores schema in manifest files
These formats include rich type information that maps well to Cognica's type system.
**File Sources**: For Parquet, ORC, and Arrow IPC files, schema is embedded in the file metadata. For CSV and JSON files, schema is inferred by sampling the first portion of the file and detecting column types from the values encountered.
### Schema Caching
Inferred schemas are cached locally to avoid the overhead of re-inferring the schema on every query. The cache provides:
- Fast query startup (no metadata queries to external sources)
- Consistent behavior (schema doesn't change mid-session)
- Offline capability (queries work even if metadata endpoints are temporarily unavailable)
The schema cache is stored in Cognica's metadata system (`_sys.virtual_tables`) and persists across server restarts.
### Schema Refresh
When an external source's schema changes, you need to refresh the cached schema in Cognica. There are two approaches:
**Manual Refresh**: Explicitly refresh the schema using ALTER FOREIGN TABLE:
```sql
ALTER FOREIGN TABLE pg_customers REFRESH SCHEMA;
```
This immediately connects to the source, retrieves the current schema, and updates the cache. Use manual refresh when:
- You know the source schema has changed
- You're deploying a coordinated schema change
- You want precise control over when schema changes take effect
**Automatic Refresh**: Configure automatic schema refresh during table creation:
```sql
CREATE FOREIGN TABLE auto_refresh_table ()
SERVER file_server
OPTIONS (
filename '/data/evolving_schema.parquet',
format 'parquet',
schema_refresh 'auto',
schema_refresh_interval '300'
);
```
With automatic refresh enabled, Cognica periodically checks whether the source schema has changed and updates the cache if necessary. The `schema_refresh_interval` specifies the minimum time between checks in seconds.
Automatic refresh is most useful for:
- File-based sources that are updated by external processes
- Development environments where schemas change frequently
- Long-running applications that need to adapt to schema changes
### Handling Schema Changes
When a source schema changes, different scenarios require different handling:
**Column Additions**: New columns in the source are included in the refreshed schema. Existing queries that don't reference the new columns continue to work unchanged.
**Column Removals**: If a column is removed from the source, queries referencing that column will fail after schema refresh. Review and update affected queries before refreshing the schema.
**Type Changes**: If a column's type changes, the new type is reflected in the refreshed schema. Queries may need adjustment if they rely on specific type behavior.
**Column Renames**: Column renames appear as a removal of the old column and addition of the new column. Queries referencing the old column name will fail after refresh.
### Type Mapping Details
Understanding how types are mapped between external sources and Cognica helps you write correct queries and avoid surprises.
**PostgreSQL Type Mapping**:
| PostgreSQL Type | Cognica Type | Notes |
|-----------------|--------------|-------|
| `boolean` | `Bool` | |
| `smallint` | `Int64` | |
| `integer` | `Int64` | |
| `bigint` | `Int64` | |
| `real` | `Double` | Single precision |
| `double precision` | `Double` | |
| `numeric`/`decimal` | `Double` | May lose precision for large values |
| `varchar`/`text` | `String` | |
| `char(n)` | `String` | Padding preserved |
| `timestamp` | `Int64` | Epoch milliseconds |
| `timestamptz` | `Int64` | Converted to UTC |
| `date` | `String` | ISO format (YYYY-MM-DD) |
| `time` | `String` | ISO format (HH:MM:SS) |
| `json`/`jsonb` | `Object` | Parsed JSON |
| `array` | `Array` | Nested type mapped recursively |
| `uuid` | `String` | Standard UUID string format |
| `bytea` | `String` | Base64 encoded |
**Delta Lake / Iceberg Type Mapping**:
| Delta/Iceberg Type | Cognica Type | Notes |
|--------------------|--------------|-------|
| `boolean` | `Bool` | |
| `byte`/`tinyint` | `Int64` | |
| `short`/`smallint` | `Int64` | |
| `integer`/`int` | `Int64` | |
| `long`/`bigint` | `Int64` | |
| `float` | `Double` | |
| `double` | `Double` | |
| `decimal` | `Double` | May lose precision |
| `string` | `String` | |
| `binary` | `String` | Base64 encoded |
| `timestamp` | `Int64` | Epoch microseconds |
| `date` | `String` | ISO format |
| `struct` | `Object` | Fields mapped recursively |
| `array`/`list` | `Array` | Element type mapped |
| `map` | `Object` | Keys must be strings |
---
## Security Considerations
External Virtual Tables extend Cognica's data access to external systems, which introduces additional security considerations. This section covers credential management, network security, and access control.
### Credential Management
External data sources require credentials for authentication. How you manage these credentials significantly impacts your security posture.
**Server-Side Configuration** (Recommended): Store credentials in the Cognica server configuration file rather than in SQL statements. This approach:
- Keeps credentials out of query logs and audit trails
- Centralizes credential management
- Allows different credentials for different environments
```yaml
# /etc/cognica/cognica.yaml - file permissions: 600
duckdb:
s3:
access_key_id: ${AWS_ACCESS_KEY_ID}
secret_access_key: ${AWS_SECRET_ACCESS_KEY}
```
Using environment variables (`${VAR_NAME}`) allows credentials to be injected at runtime without storing them in configuration files.
**Secrets Management Integration**: For production deployments, consider integrating with secrets management services:
- AWS Secrets Manager
- HashiCorp Vault
- Azure Key Vault
- Google Secret Manager
These services provide:
- Automatic credential rotation
- Audit logging of credential access
- Fine-grained access control
- Encryption at rest
**What to Avoid**:
- Embedding credentials in SQL statements visible to users
- Storing credentials in version control
- Using long-lived credentials when short-lived alternatives exist
- Sharing credentials across environments (dev/staging/production)
### Network Security
External Virtual Tables communicate over networks, potentially including the public internet. Secure your network communications:
**Use TLS/SSL**: Enable encryption for all database connections:
```sql
-- PostgreSQL with SSL
connection 'host=db.example.com sslmode=verify-full ...'
```
For cloud storage, HTTPS is typically enforced by default. Do not disable SSL unless absolutely necessary for S3-compatible storage on isolated networks.
**Network Segmentation**: Place Cognica servers in network segments that can reach external sources:
- Use VPC peering for cloud databases
- Configure security groups to allow only necessary traffic
- Use private endpoints where available (AWS PrivateLink, Azure Private Link)
**Firewall Configuration**: External databases should be configured to accept connections only from known Cognica server IP addresses. This limits exposure if credentials are compromised.
### Access Control
External Virtual Tables inherit Cognica's permission system, but you should also consider access control at the source system.
**Source-Side Permissions**: Create dedicated read-only users for Cognica access:
```sql
-- PostgreSQL: Create read-only user
CREATE USER cognica_reader WITH PASSWORD 'secure_password';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO cognica_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO cognica_reader;
```
This limits the impact of credential compromise — the attacker can only read data, not modify or delete it.
**Cognica-Side Permissions**: Use Cognica's GRANT/REVOKE to control who can access external virtual tables:
```sql
-- Grant access to specific role
GRANT SELECT ON pg_customers TO analyst_role;
-- Restrict to specific columns
GRANT SELECT (id, name, email) ON pg_customers TO limited_role;
-- Revoke access
REVOKE SELECT ON pg_customers FROM untrusted_role;
```
**Row-Level Security**: Apply RLS policies to external virtual tables for fine-grained access control:
```sql
-- Enable RLS
ALTER TABLE pg_orders ENABLE ROW LEVEL SECURITY;
-- Create policy restricting access by tenant
CREATE POLICY tenant_isolation ON pg_orders
FOR SELECT
TO app_user
USING (tenant_id = current_setting('app.tenant_id')::int);
```
RLS policies are evaluated locally after data is fetched from the external source. For large tables, consider pushing tenant filters to the source for better performance.
### Audit and Compliance
For compliance requirements (SOC 2, HIPAA, GDPR, etc.), implement appropriate auditing:
**Query Logging**: Enable query logging to track access to external data:
```yaml
logging:
query_log:
enabled: true
include_parameters: false # Avoid logging sensitive filter values
```
**Access Monitoring**: Monitor for unusual access patterns:
- High-volume queries against sensitive external sources
- Access outside normal business hours
- Queries from unexpected users or roles
**Data Classification**: Document which external sources contain sensitive data and apply appropriate controls:
- PII (Personally Identifiable Information)
- PHI (Protected Health Information)
- Financial data
- Trade secrets
---
## Monitoring and Diagnostics
Effective monitoring helps you maintain performance and quickly diagnose issues with External Virtual Tables.
### Query Performance Monitoring
Monitor query execution to identify performance issues:
**Execution Plans**: Use EXPLAIN to understand how queries are executed:
```sql
EXPLAIN SELECT * FROM pg_orders WHERE total > 1000;
```
The execution plan shows:
- Which conditions are pushed to external sources
- Join strategies and their estimated costs
- Data flow through the query pipeline
**Query Timing**: Measure query execution time to establish baselines and detect degradation:
```sql
-- Use client-side timing or application instrumentation
SELECT * FROM pg_orders WHERE order_date >= '2024-01-01';
-- Record execution time
```
### Connection Monitoring
Monitor connection pools and database attachments:
**Pool Status**: The connection pool manages reusable connections to DuckDB. Monitor pool utilization to ensure adequate capacity.
**Attachment Status**: Database attachments (for PostgreSQL, MySQL, etc.) are cached to avoid repeated authentication. Monitor attachment count and health.
### Common Diagnostic Queries
Use these queries to diagnose issues:
```sql
-- List all virtual tables
SELECT table_name, source_type, duckdb_source_type, source_path
FROM _sys.virtual_tables;
-- Test basic connectivity
SELECT 1 FROM external_table LIMIT 1;
-- View schema of external table
SELECT * FROM external_table LIMIT 0;
-- Check for schema mismatches
-- (Returns error if columns don't exist)
SELECT specific_column FROM external_table LIMIT 1;
```
### Health Checks
Implement health checks for external sources:
```sql
-- Simple connectivity check
SELECT COUNT(*) FROM pg_customers LIMIT 1;
-- Check specific table accessibility
SELECT column_name FROM information_schema.columns
WHERE table_name = 'pg_customers' LIMIT 1;
```
Include these checks in your monitoring system to detect external source issues before they impact users.
---
## Best Practices
This section summarizes best practices for using External Virtual Tables effectively in production environments.
### Performance Best Practices
**Maximize Predicate Pushdown**: Write filter conditions that can be pushed to external sources:
```sql
-- Good: Simple, pushable conditions
WHERE order_date >= '2024-01-01' AND status = 'active'
-- Avoid: Complex expressions that can't push down
WHERE EXTRACT(YEAR FROM order_date) = 2024
```
**Minimize Data Transfer**: Select only needed columns and apply filters early:
```sql
-- Good: Specific columns, filtered
SELECT id, name FROM pg_customers WHERE region = 'US';
-- Avoid: All columns, unfiltered
SELECT * FROM pg_customers;
```
**Index Awareness**: Ensure external databases have appropriate indexes for your filter conditions. Query plans that use indexes perform dramatically better than full table scans.
**Batch Large Results**: For queries returning large result sets, consider pagination or batching to avoid memory pressure:
```sql
-- Process in batches
SELECT * FROM large_table WHERE id > :last_id ORDER BY id LIMIT 10000;
```
### Operational Best Practices
**Connection Management**:
- Configure appropriate connection pool sizes based on expected concurrency
- Set reasonable timeouts for external connections
- Implement retry logic in application code for transient failures
**Schema Management**:
- Document when and how schema refreshes occur
- Test schema changes in non-production environments first
- Coordinate schema changes between external sources and Cognica
**Monitoring**:
- Establish baseline performance metrics for external queries
- Alert on query latency degradation
- Monitor connection pool utilization and attachment counts
### Naming Conventions
Use consistent naming to make virtual tables easily identifiable:
```sql
-- Prefix indicates source type
ext_pg_customers -- PostgreSQL
ext_mysql_orders -- MySQL
dl_transactions -- Delta Lake
ice_events -- Iceberg
pq_logs -- Parquet files
```
### Documentation
Maintain documentation for your external virtual tables:
- Source system contact information
- Refresh schedule and procedures
- Data freshness expectations
- Known limitations or quirks
---
## Troubleshooting
This section addresses common issues and their solutions.
### Connection Issues
**Error: Connection refused**
The database server is not accepting connections from the Cognica server.
Solutions:
1. Verify the hostname and port are correct
2. Check that the database server is running
3. Verify firewall rules allow the connection
4. Check if SSL is required but not configured
**Error: Authentication failed**
The provided credentials are invalid or insufficient.
Solutions:
1. Verify username and password are correct
2. Check that the user has appropriate permissions
3. For PostgreSQL, verify `pg_hba.conf` allows the connection method
4. Check if password has special characters that need escaping
**Error: SSL required but not configured**
The database server requires SSL but the connection string doesn't enable it.
Solution:
```sql
-- Add sslmode to connection string
connection 'host=db.example.com sslmode=require ...'
```
### Extension Issues
**Error: Extension "postgres" not found**
The required DuckDB extension is not installed.
Solutions:
1. Verify internet connectivity for automatic extension download
2. For air-gapped environments, pre-install extensions manually
3. Check DuckDB extension repository availability
**Error: Extension version mismatch**
The installed extension is incompatible with the DuckDB version.
Solution: Update the extension by reinstalling it. Extensions are typically updated when DuckDB is updated.
### Cloud Storage Issues
**Error: Access Denied to S3**
AWS credentials lack permission to access the bucket or object.
Solutions:
1. Verify IAM policy includes `s3:GetObject` and `s3:ListBucket`
2. Check bucket policy allows access from the server
3. Verify credentials are configured correctly
4. For cross-account access, check trust relationships
**Error: Invalid endpoint**
The cloud storage endpoint is unreachable or misconfigured.
Solutions:
1. Verify the endpoint URL is correct
2. Check network connectivity to the endpoint
3. For S3-compatible storage, verify `url_style_path` setting
### Schema Issues
**Error: Column not found**
The query references a column that doesn't exist in the cached schema.
Solutions:
1. Verify the column name is spelled correctly
2. Check if the column was recently removed from the source
3. Refresh the schema: `ALTER FOREIGN TABLE table_name REFRESH SCHEMA`
**Error: Type mismatch**
A value cannot be converted to the expected type.
Solutions:
1. Check the type mapping between source and Cognica
2. Use explicit casts where necessary
3. Verify the source column type hasn't changed
### Performance Issues
**Slow queries with no visible cause**
Predicates may not be pushing down, causing full table scans.
Solutions:
1. Use EXPLAIN to verify predicate pushdown
2. Simplify filter conditions to enable pushdown
3. Add indexes to the external source on filtered columns
**Memory errors on large results**
Query results exceed available memory.
Solutions:
1. Add filters to reduce result set size
2. Select fewer columns
3. Use pagination with LIMIT and OFFSET
4. Increase available memory or use streaming cursors
---
## Examples
This section provides complete, realistic examples demonstrating External Virtual Tables in common scenarios.
### Example 1: E-commerce Analytics Platform
An e-commerce company needs to analyze customer behavior by combining transactional data from PostgreSQL with clickstream data from Delta Lake.
**Setup**:
```sql
-- PostgreSQL: Transactional data
CREATE FOREIGN TABLE pg_orders ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=orders-db.internal port=5432 dbname=ecommerce user=analytics password=secret sslmode=require',
table 'public.orders'
);
CREATE FOREIGN TABLE pg_customers ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=crm-db.internal port=5432 dbname=crm user=analytics password=secret sslmode=require',
table 'public.customers'
);
-- Delta Lake: Clickstream data
CREATE FOREIGN TABLE delta_clickstream ()
SERVER duckdb_server
OPTIONS (
source 'delta',
filename 's3://data-lake/clickstream/events'
);
```
**Analysis Query**: Find customers who viewed products multiple times but haven't purchased:
```sql
WITH product_views AS (
SELECT
user_id,
product_id,
COUNT(*) AS view_count,
MAX(event_timestamp) AS last_view
FROM delta_clickstream
WHERE event_type = 'product_view'
AND event_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY user_id, product_id
HAVING COUNT(*) >= 3
),
recent_purchases AS (
SELECT DISTINCT customer_id, product_id
FROM pg_orders
WHERE order_date >= CURRENT_DATE - INTERVAL '30 days'
)
SELECT
c.email,
c.name,
pv.product_id,
pv.view_count,
pv.last_view
FROM product_views pv
JOIN pg_customers c ON pv.user_id = c.id
LEFT JOIN recent_purchases rp
ON pv.user_id = rp.customer_id AND pv.product_id = rp.product_id
WHERE rp.customer_id IS NULL
ORDER BY pv.view_count DESC
LIMIT 1000;
```
This query demonstrates joining three external sources (two PostgreSQL tables and one Delta Lake table) to identify potential customers for retargeting campaigns.
### Example 2: Multi-Region Data Federation
A global company maintains separate databases in each region for compliance. They need unified reporting across all regions.
**Setup**:
```sql
-- US Region Database
CREATE FOREIGN TABLE pg_us_sales ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=us-db.company.com dbname=sales user=report password=secret',
table 'public.sales'
);
-- EU Region Database
CREATE FOREIGN TABLE pg_eu_sales ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=eu-db.company.com dbname=sales user=report password=secret',
table 'public.sales'
);
-- APAC Region Database
CREATE FOREIGN TABLE pg_apac_sales ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=apac-db.company.com dbname=sales user=report password=secret',
table 'public.sales'
);
```
**Global Report Query**:
```sql
SELECT
'US' AS region,
DATE_TRUNC('month', sale_date) AS month,
COUNT(*) AS transactions,
SUM(amount) AS revenue
FROM pg_us_sales
WHERE sale_date >= '2024-01-01'
GROUP BY DATE_TRUNC('month', sale_date)
UNION ALL
SELECT
'EU' AS region,
DATE_TRUNC('month', sale_date) AS month,
COUNT(*) AS transactions,
SUM(amount) AS revenue
FROM pg_eu_sales
WHERE sale_date >= '2024-01-01'
GROUP BY DATE_TRUNC('month', sale_date)
UNION ALL
SELECT
'APAC' AS region,
DATE_TRUNC('month', sale_date) AS month,
COUNT(*) AS transactions,
SUM(amount) AS revenue
FROM pg_apac_sales
WHERE sale_date >= '2024-01-01'
GROUP BY DATE_TRUNC('month', sale_date)
ORDER BY month, region;
```
### Example 3: Real-Time Dashboard with Historical Context
A dashboard application needs to show real-time metrics alongside historical trends from the data lake.
**Setup**:
```sql
-- Real-time data from PostgreSQL
CREATE FOREIGN TABLE pg_transactions ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=txn-db.internal dbname=transactions user=dashboard password=secret',
table 'public.transactions'
);
-- Historical aggregates from Delta Lake
CREATE FOREIGN TABLE delta_daily_metrics ()
SERVER duckdb_server
OPTIONS (
source 'delta',
filename 's3://analytics/daily_metrics'
);
```
**Dashboard Query**: Compare today's metrics to the 30-day average:
```sql
WITH today_metrics AS (
SELECT
COUNT(*) AS transaction_count,
SUM(amount) AS total_amount,
AVG(amount) AS avg_amount
FROM pg_transactions
WHERE created_at >= CURRENT_DATE
AND created_at < CURRENT_DATE + INTERVAL '1 day'
),
historical_avg AS (
SELECT
AVG(transaction_count) AS avg_daily_transactions,
AVG(total_amount) AS avg_daily_amount,
AVG(avg_transaction_amount) AS avg_transaction_amount
FROM delta_daily_metrics
WHERE metric_date >= CURRENT_DATE - INTERVAL '30 days'
AND metric_date < CURRENT_DATE
)
SELECT
t.transaction_count AS today_transactions,
h.avg_daily_transactions AS avg_30day_transactions,
ROUND(100.0 * t.transaction_count / h.avg_daily_transactions - 100, 1) AS transactions_vs_avg_pct,
t.total_amount AS today_revenue,
h.avg_daily_amount AS avg_30day_revenue,
ROUND(100.0 * t.total_amount / h.avg_daily_amount - 100, 1) AS revenue_vs_avg_pct,
t.avg_amount AS today_avg_transaction,
h.avg_transaction_amount AS historical_avg_transaction
FROM today_metrics t, historical_avg h;
```
### Example 4: Data Lake Query with Partition Pruning
A data lake contains years of log data partitioned by date. Efficient queries must leverage partition pruning.
**Setup**:
```sql
CREATE FOREIGN TABLE s3_application_logs ()
SERVER file_server
OPTIONS (
filename 's3://logs-bucket/application/year=*/month=*/day=*/*.parquet',
format 'parquet',
partitioning 'hive'
);
```
**Efficient Query with Partition Pruning**:
```sql
-- This query only reads partitions for the specified date range
SELECT
timestamp,
level,
service,
message,
trace_id
FROM s3_application_logs
WHERE year = '2024'
AND month = '06'
AND day BETWEEN '01' AND '15'
AND level = 'ERROR'
AND service = 'payment-gateway'
ORDER BY timestamp DESC
LIMIT 1000;
```
The partition filters (`year`, `month`, `day`) are applied during file listing, so only Parquet files in matching directories are read. The `level` and `service` filters are pushed down to the Parquet reader, which uses column statistics to skip row groups that don't match.
### Example 5: Hybrid OLTP/OLAP Architecture
An application stores operational data in Cognica but needs to join with reference data from PostgreSQL and analytics results from Iceberg.
**Setup**:
```sql
-- PostgreSQL: Product catalog (reference data)
CREATE FOREIGN TABLE pg_products ()
SERVER duckdb_server
OPTIONS (
source 'postgresql',
connection 'host=catalog-db.internal dbname=catalog user=app password=secret',
table 'public.products'
);
-- Iceberg: ML model scores (analytics output)
CREATE FOREIGN TABLE iceberg_product_scores ()
SERVER duckdb_server
OPTIONS (
source 'iceberg',
filename 's3://ml-output/product_recommendations'
);
-- Native Cognica collection: User activity
-- (This is a regular Cognica collection, not a virtual table)
```
**Personalized Recommendations Query**:
```sql
-- Get product recommendations for a user based on their activity
WITH user_interests AS (
SELECT
product_category,
COUNT(*) AS interest_score
FROM user_activity
WHERE user_id = :user_id
AND activity_type IN ('view', 'wishlist', 'cart')
AND activity_date >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY product_category
),
scored_products AS (
SELECT
p.product_id,
p.name,
p.category,
p.price,
ps.recommendation_score,
COALESCE(ui.interest_score, 0) AS user_interest
FROM pg_products p
JOIN iceberg_product_scores ps ON p.product_id = ps.product_id
LEFT JOIN user_interests ui ON p.category = ui.product_category
WHERE p.is_available = true
AND ps.score_date = (SELECT MAX(score_date) FROM iceberg_product_scores)
)
SELECT
product_id,
name,
category,
price,
recommendation_score,
user_interest,
(recommendation_score * 0.7 + user_interest * 0.3) AS combined_score
FROM scored_products
ORDER BY combined_score DESC
LIMIT 20;
```
This query demonstrates the power of External Virtual Tables for building sophisticated applications that combine data from multiple sources in real-time.
---
## Appendix: Quick Reference
### Option Reference by Source Type
**DuckDB-Backed Sources (duckdb_server)**:
| Source | Required Options | Optional Options |
|--------|------------------|------------------|
| PostgreSQL | `source`, `connection`, `table` | `read_only` |
| MySQL | `source`, `connection`, `table` | `read_only` |
| SQLite | `source`, `connection`, `table` | `read_only` |
| DuckDB File | `source`, `filename`, `table` | `read_only` |
| Delta Lake | `source`, `filename` | `read_only` |
| Iceberg | `source`, `filename` | `read_only` |
**Flight SQL-Backed Sources (flight_sql_server)**:
| Source | Required Options | Optional Options |
|--------|------------------|------------------|
| Cognica | `source`, `endpoint`, `table` | `username`, `password`, `auth_token`, `catalog`, `schema`, `tls_root_certs`, `timeout_ms` |
| ClickHouse | `source`, `endpoint`, `table` | `username`, `password`, `auth_token`, `catalog`, `schema`, `tls_root_certs`, `timeout_ms` |
| DataFusion | `source`, `endpoint`, `table` | `username`, `password`, `auth_token`, `catalog`, `schema`, `tls_root_certs`, `timeout_ms` |
| Dremio | `source`, `endpoint`, `table` | `username`, `password`, `auth_token`, `catalog`, `schema`, `tls_root_certs`, `timeout_ms` |
| Generic | `source`, `endpoint`, `table` | `username`, `password`, `auth_token`, `catalog`, `schema`, `tls_root_certs`, `timeout_ms` |
**File-Backed Sources (file_server)**:
| Format | Required Options | Optional Options |
|--------|------------------|------------------|
| Parquet | `filename`, `format` | `partitioning`, `schema_refresh`, `schema_refresh_interval` |
| CSV | `filename`, `format` | `delimiter`, `header`, `quote`, `escape`, `null_value` |
| ORC | `filename`, `format` | `partitioning` |
| JSON | `filename`, `format` | |
| Arrow IPC | `filename`, `format` | |
### Cloud Storage URL Formats
| Provider | URL Format | Example |
|----------|------------|---------|
| Amazon S3 | `s3://bucket/path` | `s3://my-bucket/data/table` |
| Azure Blob | `az://container/path` | `az://mycontainer/data/table` |
| Azure ADLS Gen2 | `abfss://container@account.dfs.core.windows.net/path` | `abfss://data@mystorage.dfs.core.windows.net/table` |
| Google Cloud Storage | `gs://bucket/path` | `gs://my-bucket/data/table` |
### Type Mapping Summary
| Source Type | Cognica Type |
|-------------|--------------|
| Integer types | `Int64` |
| Floating-point types | `Double` |
| Decimal/Numeric | `Double` |
| String/Text types | `String` |
| Boolean | `Bool` |
| Timestamp | `Int64` (epoch ms) |
| Date | `String` (ISO format) |
| JSON/Struct | `Object` |
| Array/List | `Array` |
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.1 | 2026-03 | Added Arrow Flight SQL backend (Cognica, ClickHouse, DataFusion, Dremio, Generic) |
| 1.0 | 2025-12 | Initial release with PostgreSQL, MySQL, SQLite, Delta Lake, Iceberg, and file format support |
---
*For additional support, please refer to the Cognica documentation or contact the development team.*
---
# User & Permission Management
## 1. Overview
Cognica Database implements a comprehensive role-based access control (RBAC) system compatible with PostgreSQL. Key concepts:
- **Roles** are the foundation of the permission system. A role can be a user (can log in) or a group (collection of privileges).
- **Privileges** control what operations a role can perform on database objects.
- **Role Membership** allows roles to inherit privileges from other roles.
- **Row-Level Security (RLS)** provides fine-grained access control at the row level.
**Terminology:**
- In PostgreSQL (and Cognica), "user" and "role" are nearly synonymous. The difference is that a "user" is a role with the `LOGIN` attribute.
- `CREATE USER` is equivalent to `CREATE ROLE ... LOGIN`.
---
## 2. Role Management
### 2.1. CREATE ROLE
Creates a new database role without login capability (a group role).
**Syntax:**
```sql
CREATE ROLE role_name
[ [ WITH ] option [ ... ] ]
where option can be:
SUPERUSER | NOSUPERUSER
| INHERIT | NOINHERIT
| CREATEROLE | NOCREATEROLE
| CREATEDB | NOCREATEDB
| LOGIN | NOLOGIN
| REPLICATION | NOREPLICATION
| BYPASSRLS | NOBYPASSRLS
| CONNECTION LIMIT connlimit
| PASSWORD 'password' | PASSWORD NULL
| VALID UNTIL 'timestamp'
| IN ROLE role_name [, ...]
| ROLE role_name [, ...]
| ADMIN role_name [, ...]
```
**Examples:**
```sql
-- Create a basic group role
CREATE ROLE analysts;
-- Create a role with specific attributes
CREATE ROLE senior_developer
INHERIT
CREATEROLE
CREATEDB;
-- Create a role that is member of other roles
CREATE ROLE junior_developer
IN ROLE developers, readers;
-- Create a role and immediately add members
CREATE ROLE team_leads
ROLE alice, bob, charlie;
-- Create a role with admin members (they can grant this role to others)
CREATE ROLE dba_team
ADMIN alice, bob;
```
### 2.2. CREATE USER
Creates a new database role with login capability.
**Syntax:**
```sql
CREATE USER user_name
[ [ WITH ] option [ ... ] ]
```
`CREATE USER` is equivalent to `CREATE ROLE ... LOGIN`.
**Examples:**
```sql
-- Create a basic user with password
CREATE USER alice WITH PASSWORD 'secure_password_123';
-- Create a user with password expiration
CREATE USER contractor
PASSWORD 'temp_pass'
VALID UNTIL '2025-12-31 23:59:59';
-- Create a user with connection limit
CREATE USER api_service
PASSWORD 'service_password'
CONNECTION LIMIT 10;
-- Create a superuser (use with caution)
CREATE USER admin_user
SUPERUSER
PASSWORD 'admin_password';
-- Create a user that bypasses row-level security
CREATE USER etl_user
BYPASSRLS
PASSWORD 'etl_password';
-- Create a user that can create databases
CREATE USER developer
CREATEDB
PASSWORD 'dev_password';
```
### 2.3. ALTER ROLE
Modifies the attributes of an existing role.
**Syntax:**
```sql
-- Modify role attributes
ALTER ROLE role_name
[ [ WITH ] option [ ... ] ]
-- Rename a role
ALTER ROLE role_name RENAME TO new_role_name
-- Set configuration parameters for a role
ALTER ROLE role_name SET parameter TO value
ALTER ROLE role_name SET parameter = value
ALTER ROLE role_name RESET parameter
ALTER ROLE role_name RESET ALL
```
**Examples:**
```sql
-- Change password
ALTER ROLE alice PASSWORD 'new_secure_password';
-- Remove password (allow trust authentication only)
ALTER ROLE alice PASSWORD NULL;
-- Set password expiration
ALTER ROLE alice VALID UNTIL '2025-06-30';
-- Remove password expiration
ALTER ROLE alice VALID UNTIL 'infinity';
-- Grant superuser privilege
ALTER ROLE alice SUPERUSER;
-- Revoke superuser privilege
ALTER ROLE alice NOSUPERUSER;
-- Allow role to create databases
ALTER ROLE developer CREATEDB;
-- Prevent role from creating databases
ALTER ROLE developer NOCREATEDB;
-- Enable RLS bypass
ALTER ROLE etl_user BYPASSRLS;
-- Change connection limit
ALTER ROLE api_service CONNECTION LIMIT 50;
-- Remove connection limit
ALTER ROLE api_service CONNECTION LIMIT -1;
-- Rename a role
ALTER ROLE old_name RENAME TO new_name;
-- Set default search_path for a role
ALTER ROLE developer SET search_path TO myschema, public;
-- Reset a configuration parameter
ALTER ROLE developer RESET search_path;
```
### 2.4. DROP ROLE
Removes a database role.
**Syntax:**
```sql
DROP ROLE [ IF EXISTS ] role_name [, ...]
DROP USER [ IF EXISTS ] user_name [, ...]
```
**Important:** Before dropping a role, you must:
1. Remove the role from all role memberships
2. Reassign or drop all objects owned by the role
3. Revoke all privileges granted to the role
**Examples:**
```sql
-- Drop a single role
DROP ROLE analysts;
-- Drop multiple roles
DROP ROLE junior_developer, intern;
-- Drop role only if it exists (no error if not found)
DROP ROLE IF EXISTS temporary_role;
-- Drop user
DROP USER alice;
DROP USER IF EXISTS bob, charlie;
```
### 2.5. Role Attributes Reference
| Attribute | Default | Description |
|-----------|---------|-------------|
| `SUPERUSER` | `NOSUPERUSER` | Bypasses all permission checks except login |
| `INHERIT` | `INHERIT` | Automatically inherits privileges from member roles |
| `CREATEROLE` | `NOCREATEROLE` | Can create, alter, and drop other roles |
| `CREATEDB` | `NOCREATEDB` | Can create new databases |
| `LOGIN` | `NOLOGIN` | Can establish database sessions |
| `REPLICATION` | `NOREPLICATION` | Can initiate streaming replication |
| `BYPASSRLS` | `NOBYPASSRLS` | Bypasses row-level security policies |
| `CONNECTION LIMIT` | `-1` (unlimited) | Maximum concurrent connections |
| `PASSWORD` | `NULL` | Authentication password (SCRAM-SHA-256) |
| `VALID UNTIL` | `NULL` (never expires) | Password expiration timestamp |
---
## 3. Role Membership
Role membership allows organizing roles into hierarchies where child roles can inherit privileges from parent roles.
### 3.1. GRANT Role Membership
Grants membership in a role to another role.
**Syntax:**
```sql
GRANT role_name [, ...] TO role_name [, ...]
[ WITH ADMIN OPTION ]
[ WITH INHERIT OPTION ]
[ WITH SET OPTION ]
[ GRANTED BY grantor_role ]
```
**Examples:**
```sql
-- Basic membership grant
GRANT developers TO alice;
-- Grant multiple roles to multiple members
GRANT readers, writers TO alice, bob, charlie;
-- Grant with admin option (member can grant this role to others)
GRANT team_lead TO alice WITH ADMIN OPTION;
-- Grant with all options (PostgreSQL 16+ style)
GRANT senior_developer TO alice
WITH ADMIN OPTION
WITH INHERIT OPTION
WITH SET OPTION;
-- Grant without inheritance (member must SET ROLE to use privileges)
GRANT admin_role TO alice WITH INHERIT FALSE;
-- Grant without SET option (member cannot SET ROLE to this role)
GRANT restricted_role TO alice WITH SET FALSE;
```
### 3.2. REVOKE Role Membership
Removes membership in a role from another role.
**Syntax:**
```sql
REVOKE [ ADMIN OPTION FOR ] role_name [, ...] FROM role_name [, ...]
[ GRANTED BY grantor_role ]
[ CASCADE | RESTRICT ]
```
**Examples:**
```sql
-- Basic membership revoke
REVOKE developers FROM alice;
-- Revoke multiple roles from multiple members
REVOKE readers, writers FROM alice, bob;
-- Revoke only the admin option (keep membership)
REVOKE ADMIN OPTION FOR team_lead FROM alice;
-- Revoke with cascade (also revokes from roles that got it via alice)
REVOKE developers FROM alice CASCADE;
```
### 3.3. Membership Options
| Option | Default | Description |
|--------|---------|-------------|
| `ADMIN OPTION` | `FALSE` | Member can grant this role to other roles |
| `INHERIT OPTION` | `TRUE` | Member automatically inherits privileges |
| `SET OPTION` | `TRUE` | Member can use `SET ROLE` to become this role |
**Example: Options in Action**
```sql
-- Create role hierarchy
CREATE ROLE base_privileges;
CREATE ROLE extended_privileges;
CREATE USER alice LOGIN PASSWORD 'password';
-- Grant base privileges with inheritance (automatic)
GRANT base_privileges TO alice; -- alice inherits immediately
-- Grant extended privileges without inheritance
GRANT extended_privileges TO alice WITH INHERIT FALSE;
-- alice must do: SET ROLE extended_privileges; to use these
-- Grant without SET option
CREATE ROLE audit_role;
GRANT audit_role TO alice WITH SET FALSE;
-- alice inherits but CANNOT do: SET ROLE audit_role;
```
### 3.4. Inheritance Behavior
When `INHERIT` is enabled (default), a role automatically receives all privileges of its member roles without needing to use `SET ROLE`.
```sql
-- Setup
CREATE ROLE read_access;
CREATE ROLE write_access;
CREATE USER alice INHERIT PASSWORD 'pass'; -- INHERIT is default
GRANT SELECT ON ALL TABLES IN SCHEMA public TO read_access;
GRANT INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO write_access;
-- alice is member of both roles
GRANT read_access TO alice;
GRANT write_access TO alice;
-- alice can now SELECT, INSERT, UPDATE without SET ROLE
-- because INHERIT is enabled
SELECT * FROM some_table; -- Works
INSERT INTO some_table VALUES (...); -- Works
```
**Without Inheritance:**
```sql
CREATE USER bob NOINHERIT PASSWORD 'pass';
GRANT read_access TO bob;
-- bob must explicitly switch roles
SET ROLE read_access; -- Now bob can SELECT
SELECT * FROM some_table; -- Works
RESET ROLE; -- Back to bob's own privileges
```
---
## 4. Table Privileges
### 4.1. GRANT on Tables
Grants privileges on tables to roles.
**Syntax:**
```sql
GRANT { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER }
[, ...] | ALL [ PRIVILEGES ] }
ON { [ TABLE ] table_name [, ...]
| ALL TABLES IN SCHEMA schema_name [, ...] }
TO role_specification [, ...]
[ WITH GRANT OPTION ]
where role_specification can be:
role_name
| PUBLIC
| CURRENT_ROLE
| CURRENT_USER
| SESSION_USER
```
**Examples:**
```sql
-- Grant SELECT on a single table
GRANT SELECT ON users TO analysts;
-- Grant multiple privileges on a single table
GRANT SELECT, INSERT, UPDATE ON orders TO sales_team;
-- Grant all privileges on a table
GRANT ALL PRIVILEGES ON products TO product_manager;
-- or simply:
GRANT ALL ON products TO product_manager;
-- Grant to PUBLIC (all roles)
GRANT SELECT ON public_reports TO PUBLIC;
-- Grant on multiple tables
GRANT SELECT ON users, orders, products TO reporting_role;
-- Grant on all tables in a schema
GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly_role;
-- Grant with ability to re-grant
GRANT SELECT, INSERT ON orders TO team_lead WITH GRANT OPTION;
```
### 4.2. REVOKE on Tables
Revokes previously granted privileges.
**Syntax:**
```sql
REVOKE [ GRANT OPTION FOR ]
{ { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER }
[, ...] | ALL [ PRIVILEGES ] }
ON { [ TABLE ] table_name [, ...]
| ALL TABLES IN SCHEMA schema_name [, ...] }
FROM role_specification [, ...]
[ CASCADE | RESTRICT ]
```
**Examples:**
```sql
-- Revoke specific privilege
REVOKE INSERT ON users FROM intern;
-- Revoke multiple privileges
REVOKE INSERT, UPDATE, DELETE ON users FROM readonly_role;
-- Revoke all privileges
REVOKE ALL PRIVILEGES ON sensitive_data FROM public_role;
-- Revoke only the grant option (keep the privilege)
REVOKE GRANT OPTION FOR SELECT ON reports FROM team_lead;
-- Revoke from PUBLIC
REVOKE SELECT ON internal_logs FROM PUBLIC;
-- Revoke with cascade (also revokes from roles that got it via re-grant)
REVOKE SELECT ON orders FROM alice CASCADE;
-- Revoke on all tables in schema
REVOKE ALL ON ALL TABLES IN SCHEMA hr FROM contractors;
```
### 4.3. Privilege Types
| Privilege | Applicable To | Allows |
|-----------|---------------|--------|
| `SELECT` | Tables, Views, Sequences | Read data from the object |
| `INSERT` | Tables | Insert new rows |
| `UPDATE` | Tables, Sequences | Modify existing rows / advance sequence |
| `DELETE` | Tables | Delete rows |
| `TRUNCATE` | Tables | Empty the table (faster than DELETE) |
| `REFERENCES` | Tables | Create foreign key references |
| `TRIGGER` | Tables | Create triggers on the table |
**Privilege Combinations for Common Operations:**
| Operation | Required Privileges |
|-----------|---------------------|
| `SELECT * FROM t` | SELECT on t |
| `SELECT a, b FROM t` | SELECT on t (or SELECT on columns a, b) |
| `INSERT INTO t VALUES (...)` | INSERT on t |
| `INSERT INTO t (a, b) VALUES (...)` | INSERT on t (or INSERT on columns a, b) |
| `UPDATE t SET a = 1` | UPDATE on t (or UPDATE on column a), SELECT if WHERE clause |
| `UPDATE t SET a = 1 WHERE b = 2` | UPDATE on column a, SELECT on column b |
| `DELETE FROM t` | DELETE on t |
| `DELETE FROM t WHERE a = 1` | DELETE on t, SELECT on column a |
| `TRUNCATE t` | TRUNCATE on t |
### 4.4. WITH GRANT OPTION
When `WITH GRANT OPTION` is specified, the grantee can grant the same privilege to others.
```sql
-- alice can now grant SELECT to other roles
GRANT SELECT ON reports TO alice WITH GRANT OPTION;
-- alice grants to bob
SET ROLE alice;
GRANT SELECT ON reports TO bob; -- This works
-- bob tries to grant to charlie (fails - bob doesn't have grant option)
SET ROLE bob;
GRANT SELECT ON reports TO charlie; -- ERROR: permission denied
```
**Revoking Grant Option:**
```sql
-- Remove only the grant option, alice keeps SELECT
REVOKE GRANT OPTION FOR SELECT ON reports FROM alice;
-- Remove privilege entirely with cascade
-- (also removes from roles alice granted to)
REVOKE SELECT ON reports FROM alice CASCADE;
```
---
## 5. Column-Level Privileges
Column-level privileges provide finer control than table-level privileges.
### 5.1. GRANT on Columns
**Syntax:**
```sql
GRANT { { SELECT | INSERT | UPDATE | REFERENCES } ( column_name [, ...] )
[, ...] | ALL [ PRIVILEGES ] ( column_name [, ...] ) }
ON [ TABLE ] table_name [, ...]
TO role_specification [, ...]
[ WITH GRANT OPTION ]
```
**Examples:**
```sql
-- Grant SELECT on specific columns
GRANT SELECT (id, name, email) ON users TO public_api;
-- Grant UPDATE on specific columns
GRANT UPDATE (status, updated_at) ON orders TO customer_service;
-- Grant multiple privileges on different columns
GRANT SELECT (id, name), UPDATE (status) ON tasks TO workers;
-- Combination of table and column privileges
GRANT SELECT ON orders TO analysts; -- Full table access
GRANT SELECT (order_id, amount) ON orders TO limited_view; -- Partial access
-- Grant INSERT on specific columns (for tables with defaults)
GRANT INSERT (name, email) ON users TO registration_service;
```
### 5.2. REVOKE on Columns
**Syntax:**
```sql
REVOKE [ GRANT OPTION FOR ]
{ { SELECT | INSERT | UPDATE | REFERENCES } ( column_name [, ...] )
[, ...] | ALL [ PRIVILEGES ] ( column_name [, ...] ) }
ON [ TABLE ] table_name [, ...]
FROM role_specification [, ...]
[ CASCADE | RESTRICT ]
```
**Examples:**
```sql
-- Revoke SELECT on specific columns
REVOKE SELECT (ssn, salary) ON employees FROM general_access;
-- Revoke all column privileges
REVOKE ALL (sensitive_data) ON users FROM public_role;
```
### 5.3. Column Privilege Behavior
Column privileges interact with table privileges:
1. **Table privilege grants access to all columns:**
```sql
GRANT SELECT ON users TO alice;
-- alice can: SELECT * FROM users;
-- alice can: SELECT id, name, email FROM users;
```
2. **Column privileges grant access to specific columns only:**
```sql
GRANT SELECT (id, name) ON users TO bob;
-- bob can: SELECT id, name FROM users;
-- bob CANNOT: SELECT * FROM users; (lacks email, etc.)
-- bob CANNOT: SELECT email FROM users;
```
3. **Column privileges are additive:**
```sql
GRANT SELECT (id) ON users TO charlie;
GRANT SELECT (name) ON users TO charlie;
-- charlie can: SELECT id, name FROM users;
```
4. **UPDATE requires SELECT for WHERE clause:**
```sql
GRANT UPDATE (status) ON orders TO worker;
-- worker can: UPDATE orders SET status = 'done';
-- worker CANNOT: UPDATE orders SET status = 'done' WHERE id = 1;
-- (lacks SELECT on id)
GRANT UPDATE (status), SELECT (id) ON orders TO worker;
-- worker can: UPDATE orders SET status = 'done' WHERE id = 1;
```
---
## 6. Default Privileges
Default privileges automatically apply to objects created in the future.
### 6.1. ALTER DEFAULT PRIVILEGES
**Syntax:**
```sql
ALTER DEFAULT PRIVILEGES
[ FOR ROLE role_name [, ...] ]
[ IN SCHEMA schema_name [, ...] ]
abbreviated_grant_or_revoke
where abbreviated_grant_or_revoke is:
GRANT { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER }
[, ...] | ALL [ PRIVILEGES ] }
ON TABLES
TO role_specification [, ...] [ WITH GRANT OPTION ]
REVOKE [ GRANT OPTION FOR ]
{ { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER }
[, ...] | ALL [ PRIVILEGES ] }
ON TABLES
FROM role_specification [, ...] [ CASCADE | RESTRICT ]
```
**Examples:**
```sql
-- All future tables created by current user get SELECT for analysts
ALTER DEFAULT PRIVILEGES
GRANT SELECT ON TABLES TO analysts;
-- All future tables in schema 'reports' get SELECT for reporting_role
ALTER DEFAULT PRIVILEGES IN SCHEMA reports
GRANT SELECT ON TABLES TO reporting_role;
-- All future tables created by alice in public schema
ALTER DEFAULT PRIVILEGES FOR ROLE alice IN SCHEMA public
GRANT SELECT, INSERT ON TABLES TO developers;
-- Remove default privilege
ALTER DEFAULT PRIVILEGES IN SCHEMA public
REVOKE SELECT ON TABLES FROM PUBLIC;
-- Setup for application: app_owner creates tables, app_user accesses them
ALTER DEFAULT PRIVILEGES FOR ROLE app_owner
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user;
```
### 6.2. Schema-Specific Defaults
Default privileges can be set globally or per-schema:
```sql
-- Global default (all schemas)
ALTER DEFAULT PRIVILEGES
GRANT SELECT ON TABLES TO global_reader;
-- Schema-specific (only 'analytics' schema)
ALTER DEFAULT PRIVILEGES IN SCHEMA analytics
GRANT SELECT ON TABLES TO analytics_team;
-- The analytics_team gets SELECT on:
-- - Tables created in 'analytics' schema (from schema-specific default)
-- They DO NOT automatically get SELECT on tables in other schemas
```
---
## 7. Row-Level Security
Row-Level Security (RLS) provides fine-grained access control at the row level. When enabled, queries automatically filter rows based on policies.
### 7.1. Enabling RLS
**Syntax:**
```sql
-- Enable RLS on a table
ALTER TABLE table_name ENABLE ROW LEVEL SECURITY;
-- Disable RLS on a table
ALTER TABLE table_name DISABLE ROW LEVEL SECURITY;
-- Force RLS even for table owner
ALTER TABLE table_name FORCE ROW LEVEL SECURITY;
-- Don't force RLS for table owner
ALTER TABLE table_name NO FORCE ROW LEVEL SECURITY;
```
**Examples:**
```sql
-- Enable RLS
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
-- Force owner to also obey RLS
ALTER TABLE sensitive_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE sensitive_data FORCE ROW LEVEL SECURITY;
```
**Important:** After enabling RLS, if no policies are defined, no rows are visible to non-superusers (except the table owner).
### 7.2. CREATE POLICY
Creates a row-level security policy.
**Syntax:**
```sql
CREATE POLICY name ON table_name
[ AS { PERMISSIVE | RESTRICTIVE } ]
[ FOR { ALL | SELECT | INSERT | UPDATE | DELETE } ]
[ TO { role_name | PUBLIC | CURRENT_ROLE | CURRENT_USER | SESSION_USER } [, ...] ]
[ USING ( using_expression ) ]
[ WITH CHECK ( check_expression ) ]
```
**Examples:**
```sql
-- Users can only see their own data
CREATE POLICY user_isolation ON orders
FOR ALL
TO PUBLIC
USING (user_id = current_user_id());
-- Users can only see active records
CREATE POLICY active_only ON products
FOR SELECT
TO PUBLIC
USING (status = 'active');
-- Managers can see all orders in their department
CREATE POLICY manager_access ON orders
FOR SELECT
TO managers
USING (department_id IN (
SELECT department_id FROM manager_departments
WHERE manager_id = current_user_id()
));
-- Users can only insert records for themselves
CREATE POLICY insert_own ON user_data
FOR INSERT
TO PUBLIC
WITH CHECK (user_id = current_user_id());
-- Users can update their own non-locked records
CREATE POLICY update_own ON documents
FOR UPDATE
TO PUBLIC
USING (owner_id = current_user_id() AND NOT is_locked)
WITH CHECK (owner_id = current_user_id());
-- Restrictive policy: only during business hours
CREATE POLICY business_hours ON sensitive_ops
AS RESTRICTIVE
FOR ALL
TO PUBLIC
USING (
EXTRACT(HOUR FROM CURRENT_TIMESTAMP) BETWEEN 9 AND 17
AND EXTRACT(DOW FROM CURRENT_TIMESTAMP) BETWEEN 1 AND 5
);
```
### 7.3. ALTER POLICY
Modifies an existing policy.
**Syntax:**
```sql
ALTER POLICY name ON table_name
[ TO { role_name | PUBLIC | CURRENT_ROLE | CURRENT_USER | SESSION_USER } [, ...] ]
[ USING ( using_expression ) ]
[ WITH CHECK ( check_expression ) ]
ALTER POLICY name ON table_name RENAME TO new_name
```
**Examples:**
```sql
-- Change the USING expression
ALTER POLICY user_isolation ON orders
USING (user_id = current_user_id() OR is_public = true);
-- Change target roles
ALTER POLICY manager_access ON orders
TO managers, executives;
-- Rename policy
ALTER POLICY old_policy_name ON orders RENAME TO new_policy_name;
```
### 7.4. DROP POLICY
Removes a policy.
**Syntax:**
```sql
DROP POLICY [ IF EXISTS ] name ON table_name
```
**Examples:**
```sql
-- Drop a policy
DROP POLICY user_isolation ON orders;
-- Drop if exists
DROP POLICY IF EXISTS deprecated_policy ON old_table;
```
### 7.5. Policy Types
**PERMISSIVE (default):**
- Multiple permissive policies are combined with OR
- A row is visible if ANY permissive policy allows it
**RESTRICTIVE:**
- Multiple restrictive policies are combined with AND
- A row is visible only if ALL restrictive policies allow it
**Combined behavior:**
```
(permissive_1 OR permissive_2 OR ...) AND restrictive_1 AND restrictive_2 AND ...
```
**Example:**
```sql
-- Table: documents
-- Permissive: users can see own documents OR public documents
CREATE POLICY own_or_public ON documents
AS PERMISSIVE
FOR SELECT
USING (owner_id = current_user_id() OR is_public = true);
-- Permissive: admins can see everything
CREATE POLICY admin_access ON documents
AS PERMISSIVE
FOR SELECT
TO admins
USING (true);
-- Restrictive: only non-deleted documents
CREATE POLICY not_deleted ON documents
AS RESTRICTIVE
FOR SELECT
USING (deleted_at IS NULL);
-- Result for regular users:
-- (owner_id = current_user_id() OR is_public = true) AND deleted_at IS NULL
-- Result for admins:
-- (owner_id = current_user_id() OR is_public = true OR true) AND deleted_at IS NULL
-- Simplifies to: deleted_at IS NULL (admins see all non-deleted)
```
### 7.6. Policy Commands
| Command | Applies To | Description |
|---------|------------|-------------|
| `ALL` | SELECT, INSERT, UPDATE, DELETE | Policy applies to all commands |
| `SELECT` | SELECT, UPDATE (for WHERE), DELETE (for WHERE) | Controls which rows can be read |
| `INSERT` | INSERT | Controls which rows can be inserted |
| `UPDATE` | UPDATE | Controls which rows can be updated |
| `DELETE` | DELETE | Controls which rows can be deleted |
**Per-Command Policies:**
```sql
-- Different policies for different operations
CREATE POLICY read_all ON orders
FOR SELECT
USING (true); -- Everyone can read
CREATE POLICY insert_own ON orders
FOR INSERT
WITH CHECK (user_id = current_user_id()); -- Can only insert own
CREATE POLICY update_own ON orders
FOR UPDATE
USING (user_id = current_user_id()) -- Can only see own for update
WITH CHECK (user_id = current_user_id()); -- Can only update to own
CREATE POLICY delete_own ON orders
FOR DELETE
USING (user_id = current_user_id()); -- Can only delete own
```
### 7.7. USING vs WITH CHECK
| Clause | Purpose | Used For |
|--------|---------|----------|
| `USING` | Filter existing rows | SELECT, UPDATE (which rows), DELETE |
| `WITH CHECK` | Validate new/modified rows | INSERT, UPDATE (new values) |
**USING Expression:**
- Applied to existing rows before they are returned or modified
- For SELECT: filters which rows are visible
- For UPDATE/DELETE: filters which rows can be affected
**WITH CHECK Expression:**
- Applied to new row values
- For INSERT: validates the new row
- For UPDATE: validates the updated row values
- If omitted, defaults to the USING expression
**Example:**
```sql
-- Users can see and modify their own records
-- But cannot change ownership
CREATE POLICY ownership ON records
FOR ALL
USING (owner_id = current_user_id()) -- Can only see own
WITH CHECK (owner_id = current_user_id()); -- Cannot change owner
-- This prevents:
-- UPDATE records SET owner_id = other_user WHERE ...
-- Because WITH CHECK would fail
```
### 7.8. RLS Bypass Rules
Certain roles can bypass RLS:
| Role Type | Bypass RLS? | Notes |
|-----------|-------------|-------|
| Superuser | Always | Cannot be restricted |
| `BYPASSRLS` role | Always | Explicitly granted bypass |
| Table owner | By default | Unless `FORCE ROW LEVEL SECURITY` is set |
| Regular users | Never | Must pass all policies |
**Examples:**
```sql
-- Create user that bypasses RLS (for ETL processes)
CREATE USER etl_processor BYPASSRLS PASSWORD 'etl_pass';
-- Force owner to also obey RLS
ALTER TABLE sensitive_data FORCE ROW LEVEL SECURITY;
-- Now even the owner must pass policies
-- (except superusers, who always bypass)
```
---
## 8. Session Management
### 8.1. SET ROLE
Changes the current role for the session.
**Syntax:**
```sql
SET ROLE role_name
SET ROLE NONE
```
**Examples:**
```sql
-- Switch to another role (must be member of that role)
SET ROLE admin_role;
-- Check current role
SELECT current_role;
-- Switch back to session role
SET ROLE NONE;
-- or
RESET ROLE;
```
**Requirements:**
- You must be a member of the target role
- The membership must have `SET OPTION` (default is TRUE)
```sql
-- alice is member of developers
GRANT developers TO alice;
-- alice can switch
SET ROLE developers; -- Works
-- alice is member of restricted without SET option
GRANT restricted TO alice WITH SET FALSE;
-- alice cannot switch
SET ROLE restricted; -- ERROR: permission denied
```
### 8.2. RESET ROLE
Resets the current role to the original session role.
**Syntax:**
```sql
RESET ROLE
```
Equivalent to `SET ROLE NONE`.
**Example:**
```sql
-- Login as alice
-- session_user = alice, current_role = alice
SET ROLE admin_role;
-- session_user = alice, current_role = admin_role
-- Do admin work...
RESET ROLE;
-- session_user = alice, current_role = alice
```
### 8.3. Session vs Current Role
| Function | Returns | Description |
|----------|---------|-------------|
| `session_user` | Original authenticated role | Never changes during session |
| `current_user` | Currently active role | Changes with SET ROLE |
| `current_role` | Same as current_user | Alias for current_user |
**Example:**
```sql
-- Login as alice
SELECT session_user, current_user, current_role;
-- alice, alice, alice
SET ROLE developers;
SELECT session_user, current_user, current_role;
-- alice, developers, developers
RESET ROLE;
SELECT session_user, current_user, current_role;
-- alice, alice, alice
```
**Use Cases:**
```sql
-- RLS policy using session_user (tracks who actually logged in)
CREATE POLICY audit_trail ON changes
FOR INSERT
WITH CHECK (created_by = session_user);
-- RLS policy using current_user (respects SET ROLE)
CREATE POLICY role_based ON documents
FOR SELECT
USING (allowed_role = current_role);
```
---
## 9. System Functions
### 9.1. Role Information Functions
| Function | Return Type | Description |
|----------|-------------|-------------|
| `current_user` | name | Current active role name |
| `current_role` | name | Same as current_user |
| `session_user` | name | Original authenticated role |
| `pg_get_userbyid(oid)` | name | Role name for given OID |
**Examples:**
```sql
-- Get current user information
SELECT current_user, session_user;
-- Get role name from OID
SELECT pg_get_userbyid(10); -- Returns 'admin' (bootstrap superuser)
-- Use in queries
SELECT * FROM orders WHERE created_by = current_user;
```
### 9.2. Privilege Checking Functions
| Function | Return Type | Description |
|----------|-------------|-------------|
| `has_table_privilege(table, privilege)` | boolean | Check current user's privilege |
| `has_table_privilege(user, table, privilege)` | boolean | Check specific user's privilege |
| `has_column_privilege(table, column, privilege)` | boolean | Check column privilege |
| `has_column_privilege(user, table, column, privilege)` | boolean | Check specific user's column privilege |
**Examples:**
```sql
-- Check if current user can SELECT from orders
SELECT has_table_privilege('orders', 'SELECT');
-- Check if alice can INSERT into users
SELECT has_table_privilege('alice', 'users', 'INSERT');
-- Check multiple privileges
SELECT has_table_privilege('orders', 'SELECT, UPDATE');
-- Check column privilege
SELECT has_column_privilege('users', 'email', 'SELECT');
-- Use in conditional logic
DO $$
BEGIN
IF has_table_privilege('sensitive_data', 'SELECT') THEN
RAISE NOTICE 'You have access to sensitive data';
ELSE
RAISE NOTICE 'Access denied to sensitive data';
END IF;
END $$;
```
---
## 10. System Catalog Views
### 10.1. pg_catalog Views
**pg_roles — All database roles:**
```sql
SELECT rolname, rolsuper, rolinherit, rolcreaterole, rolcreatedb,
rolcanlogin, rolreplication, rolbypassrls, rolconnlimit
FROM pg_roles;
```
| Column | Type | Description |
|--------|------|-------------|
| `rolname` | name | Role name |
| `rolsuper` | boolean | Is superuser |
| `rolinherit` | boolean | Inherits privileges |
| `rolcreaterole` | boolean | Can create roles |
| `rolcreatedb` | boolean | Can create databases |
| `rolcanlogin` | boolean | Can log in |
| `rolreplication` | boolean | Can do replication |
| `rolbypassrls` | boolean | Bypasses RLS |
| `rolconnlimit` | integer | Connection limit |
| `rolvaliduntil` | timestamptz | Password expiration |
**pg_user — Login roles only:**
```sql
SELECT usename, usesysid, usecreatedb, usesuper
FROM pg_user;
```
**pg_auth_members — Role memberships:**
```sql
SELECT r.rolname AS role, m.rolname AS member,
am.admin_option, am.inherit_option, am.set_option
FROM pg_auth_members am
JOIN pg_roles r ON r.oid = am.roleid
JOIN pg_roles m ON m.oid = am.member;
```
### 10.2. information_schema Views
**applicable_roles — Roles applicable to current user:**
```sql
SELECT * FROM information_schema.applicable_roles;
```
**enabled_roles — Currently enabled roles:**
```sql
SELECT * FROM information_schema.enabled_roles;
```
**role_table_grants — Table privileges by role:**
```sql
SELECT grantor, grantee, table_name, privilege_type, is_grantable
FROM information_schema.role_table_grants
WHERE grantee = 'alice';
```
**table_privileges — All table privileges:**
```sql
SELECT grantor, grantee, table_schema, table_name, privilege_type
FROM information_schema.table_privileges
WHERE table_schema = 'public';
```
**column_privileges — Column privileges:**
```sql
SELECT grantor, grantee, table_name, column_name, privilege_type
FROM information_schema.column_privileges
WHERE grantee = current_user;
```
---
## 11. Authentication
### 11.1. Password Management
**Setting a password:**
```sql
-- On user creation
CREATE USER alice PASSWORD 'secure_password';
-- Later modification
ALTER USER alice PASSWORD 'new_secure_password';
-- Remove password (trust authentication only)
ALTER USER alice PASSWORD NULL;
```
**Password requirements:**
- Passwords are stored using SCRAM-SHA-256 hashing
- Minimum password length depends on server configuration
- Passwords are case-sensitive
### 11.2. SCRAM-SHA-256
Cognica uses SCRAM-SHA-256 (Salted Challenge Response Authentication Mechanism) for password authentication, which is the PostgreSQL 14+ default.
**Benefits:**
- Password is never sent in plaintext
- Server stores only a hash, not the actual password
- Resistant to replay attacks
- Channel binding support for man-in-the-middle protection
**Stored password format:**
```
SCRAM-SHA-256$iterations:salt$StoredKey:ServerKey
```
### 11.3. Password Expiration
**Setting expiration:**
```sql
-- Expire at specific time
ALTER USER contractor VALID UNTIL '2025-12-31 23:59:59';
-- Expire immediately (forces password change on next login)
ALTER USER alice VALID UNTIL 'now';
-- Remove expiration
ALTER USER alice VALID UNTIL 'infinity';
```
**Checking expiration:**
```sql
SELECT rolname, rolvaliduntil
FROM pg_roles
WHERE rolvaliduntil IS NOT NULL
AND rolvaliduntil < CURRENT_TIMESTAMP + INTERVAL '30 days';
```
---
## 12. Security Best Practices
### Principle of Least Privilege
```sql
-- BAD: Granting all privileges
GRANT ALL ON ALL TABLES IN SCHEMA public TO app_user;
-- GOOD: Grant only necessary privileges
GRANT SELECT ON products TO web_app;
GRANT SELECT, INSERT ON orders TO web_app;
GRANT UPDATE (status) ON orders TO web_app;
```
### Use Role Hierarchies
```sql
-- Create role hierarchy
CREATE ROLE readonly;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
CREATE ROLE readwrite;
GRANT readonly TO readwrite; -- Inherit SELECT
GRANT INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO readwrite;
CREATE ROLE admin;
GRANT readwrite TO admin; -- Inherit all of readwrite
GRANT TRUNCATE ON ALL TABLES IN SCHEMA public TO admin;
-- Assign users to appropriate roles
GRANT readonly TO analyst_user;
GRANT readwrite TO developer_user;
GRANT admin TO dba_user;
```
### Protect Sensitive Data with RLS
```sql
-- Enable RLS on sensitive tables
ALTER TABLE customer_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE customer_data FORCE ROW LEVEL SECURITY;
-- Create restrictive policies
CREATE POLICY customer_isolation ON customer_data
FOR ALL
USING (customer_id = get_current_customer_id());
```
### Use Service Accounts with Limited Privileges
```sql
-- Create service account with minimal privileges
CREATE USER api_service
PASSWORD 'strong_random_password'
CONNECTION LIMIT 50;
-- Grant only necessary privileges
GRANT SELECT ON public_data TO api_service;
GRANT INSERT ON api_logs TO api_service;
```
### Regular Privilege Audits
```sql
-- Find roles with superuser privilege
SELECT rolname FROM pg_roles WHERE rolsuper;
-- Find roles that can bypass RLS
SELECT rolname FROM pg_roles WHERE rolbypassrls OR rolsuper;
-- Find roles with CREATEROLE (can create other roles)
SELECT rolname FROM pg_roles WHERE rolcreaterole;
-- Find all privileges on a specific table
SELECT grantee, privilege_type, is_grantable
FROM information_schema.table_privileges
WHERE table_name = 'sensitive_table';
-- Find excessive privileges (ALL on tables)
SELECT grantee, table_name
FROM information_schema.table_privileges
WHERE privilege_type = 'ALL';
```
### Password Policy Enforcement
```sql
-- Set password expiration for all users
DO $$
DECLARE
r RECORD;
BEGIN
FOR r IN SELECT rolname FROM pg_roles WHERE rolcanlogin AND rolname != 'admin'
LOOP
EXECUTE format('ALTER ROLE %I VALID UNTIL %L',
r.rolname,
(CURRENT_TIMESTAMP + INTERVAL '90 days')::text);
END LOOP;
END $$;
```
---
## 13. Common Patterns and Examples
### Multi-Tenant Application
```sql
-- Setup
CREATE TABLE tenants (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL
);
CREATE TABLE tenant_users (
user_id INTEGER REFERENCES pg_roles(oid),
tenant_id INTEGER REFERENCES tenants(id)
);
CREATE TABLE tenant_data (
id SERIAL PRIMARY KEY,
tenant_id INTEGER REFERENCES tenants(id),
data JSONB
);
-- Enable RLS
ALTER TABLE tenant_data ENABLE ROW LEVEL SECURITY;
-- Function to get current tenant
CREATE FUNCTION current_tenant_id() RETURNS INTEGER AS $$
SELECT tenant_id FROM tenant_users
WHERE user_id = (SELECT oid FROM pg_roles WHERE rolname = current_user)
$$ LANGUAGE SQL SECURITY DEFINER;
-- Policy: users can only see their tenant's data
CREATE POLICY tenant_isolation ON tenant_data
FOR ALL
USING (tenant_id = current_tenant_id())
WITH CHECK (tenant_id = current_tenant_id());
```
### Audit Trail with RLS
```sql
-- Audit table
CREATE TABLE audit_log (
id SERIAL PRIMARY KEY,
table_name TEXT,
action TEXT,
old_data JSONB,
new_data JSONB,
user_name TEXT DEFAULT session_user,
timestamp TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);
-- RLS: users can only see their own audit entries
ALTER TABLE audit_log ENABLE ROW LEVEL SECURITY;
CREATE POLICY own_audit ON audit_log
FOR SELECT
USING (user_name = session_user);
-- Admins can see all
CREATE POLICY admin_audit ON audit_log
FOR SELECT
TO admins
USING (true);
```
### Department-Based Access Control
```sql
-- Setup
CREATE TABLE employees (
id SERIAL PRIMARY KEY,
name TEXT,
department_id INTEGER,
manager_id INTEGER,
salary NUMERIC
);
-- Enable RLS
ALTER TABLE employees ENABLE ROW LEVEL SECURITY;
-- Policy: see own record
CREATE POLICY own_record ON employees
FOR SELECT
USING (name = current_user);
-- Policy: managers see their department
CREATE POLICY manager_view ON employees
FOR SELECT
TO managers
USING (
department_id IN (
SELECT department_id FROM employees WHERE name = current_user
)
);
-- Column privilege: salary only for HR
REVOKE SELECT ON employees FROM PUBLIC;
GRANT SELECT (id, name, department_id) ON employees TO PUBLIC;
GRANT SELECT (salary) ON employees TO hr_department;
```
### API Key Management
```sql
-- Table for API keys
CREATE TABLE api_keys (
id SERIAL PRIMARY KEY,
key_hash TEXT NOT NULL, -- Hashed API key
owner_role TEXT NOT NULL,
permissions TEXT[],
expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
);
-- Enable RLS
ALTER TABLE api_keys ENABLE ROW LEVEL SECURITY;
-- Users can only see their own API keys
CREATE POLICY own_keys ON api_keys
FOR ALL
USING (owner_role = current_user)
WITH CHECK (owner_role = current_user);
-- Admins can manage all keys
CREATE POLICY admin_keys ON api_keys
FOR ALL
TO admins
USING (true)
WITH CHECK (true);
```
### Time-Based Access Control
```sql
-- Policy: only allow access during business hours
CREATE POLICY business_hours_only ON sensitive_operations
AS RESTRICTIVE
FOR ALL
USING (
EXTRACT(HOUR FROM CURRENT_TIMESTAMP AT TIME ZONE 'America/New_York')
BETWEEN 9 AND 17
AND EXTRACT(DOW FROM CURRENT_TIMESTAMP) BETWEEN 1 AND 5
);
-- Policy: allow maintenance window access
CREATE POLICY maintenance_window ON sensitive_operations
AS PERMISSIVE
FOR ALL
TO maintenance_role
USING (
CURRENT_TIMESTAMP BETWEEN
(CURRENT_DATE + TIME '02:00') AND
(CURRENT_DATE + TIME '04:00')
);
```
---
## 14. Troubleshooting
### Common Errors and Solutions
**Error: permission denied for table X**
```sql
-- Check your current role
SELECT current_user, session_user;
-- Check what privileges you have
SELECT privilege_type
FROM information_schema.table_privileges
WHERE table_name = 'X' AND grantee = current_user;
-- Solution: Grant necessary privilege
GRANT SELECT ON X TO your_role;
```
**Error: permission denied to set role**
```sql
-- Check membership
SELECT r.rolname
FROM pg_auth_members am
JOIN pg_roles r ON r.oid = am.roleid
JOIN pg_roles m ON m.oid = am.member
WHERE m.rolname = current_user;
-- Check if SET option is granted
SELECT am.set_option
FROM pg_auth_members am
JOIN pg_roles r ON r.oid = am.roleid
WHERE r.rolname = 'target_role'
AND am.member = (SELECT oid FROM pg_roles WHERE rolname = current_user);
-- Solution: Grant with SET option
GRANT target_role TO your_role WITH SET TRUE;
```
**Error: new row violates row-level security policy**
```sql
-- This happens when WITH CHECK fails
-- Check what policy is blocking
SELECT polname, polcmd, polqual, polwithcheck
FROM pg_policy
WHERE polrelid = 'your_table'::regclass;
-- Ensure your INSERT/UPDATE satisfies WITH CHECK expression
```
**Error: no policy allows access to table**
```sql
-- RLS is enabled but no permissive policy matches
-- Check RLS status
SELECT relrowsecurity, relforcerowsecurity
FROM pg_class
WHERE relname = 'your_table';
-- Check existing policies
SELECT polname, polroles, polcmd, polpermissive, polqual
FROM pg_policy
WHERE polrelid = 'your_table'::regclass;
-- Solution: Create appropriate policy or grant bypass
```
### Debugging Permission Issues
```sql
-- 1. Check current identity
SELECT current_user, session_user;
-- 2. Check role attributes
SELECT * FROM pg_roles WHERE rolname = current_user;
-- 3. Check role memberships
WITH RECURSIVE membership AS (
SELECT roleid, member, 1 as level
FROM pg_auth_members
WHERE member = (SELECT oid FROM pg_roles WHERE rolname = current_user)
UNION ALL
SELECT am.roleid, am.member, m.level + 1
FROM pg_auth_members am
JOIN membership m ON am.member = m.roleid
)
SELECT r.rolname, m.level
FROM membership m
JOIN pg_roles r ON r.oid = m.roleid;
-- 4. Check table privileges
SELECT grantor, grantee, privilege_type, is_grantable
FROM information_schema.table_privileges
WHERE table_name = 'problem_table';
-- 5. Check RLS policies
SELECT polname, polcmd,
pg_get_expr(polqual, polrelid) as using_expr,
pg_get_expr(polwithcheck, polrelid) as check_expr
FROM pg_policy
WHERE polrelid = 'problem_table'::regclass;
-- 6. Test permission function
SELECT has_table_privilege(current_user, 'problem_table', 'SELECT');
```
### Performance Considerations
**RLS Policy Performance:**
```sql
-- BAD: Subquery executed for every row
CREATE POLICY slow_policy ON large_table
USING (user_id IN (SELECT user_id FROM user_groups WHERE group_id = get_group()));
-- BETTER: Use session variable
CREATE POLICY fast_policy ON large_table
USING (user_id = current_setting('app.current_user_id')::integer);
-- Set at session start
SET app.current_user_id = '123';
```
**Permission Cache:**
- Permission check results are cached in memory
- Cache is invalidated when roles or privileges change
- No manual cache management needed
---
## Quick Reference Card
### Role Management
```sql
CREATE USER name PASSWORD 'pass';
CREATE ROLE name;
ALTER ROLE name PASSWORD 'new_pass';
ALTER ROLE name SUPERUSER | NOSUPERUSER;
DROP ROLE name;
```
### Role Membership
```sql
GRANT role TO member;
GRANT role TO member WITH ADMIN OPTION;
REVOKE role FROM member;
```
### Table Privileges
```sql
GRANT SELECT ON table TO role;
GRANT ALL ON table TO role;
REVOKE SELECT ON table FROM role;
```
### Column Privileges
```sql
GRANT SELECT (col1, col2) ON table TO role;
REVOKE UPDATE (col1) ON table FROM role;
```
### Row-Level Security
```sql
ALTER TABLE t ENABLE ROW LEVEL SECURITY;
CREATE POLICY p ON t USING (condition);
CREATE POLICY p ON t FOR SELECT USING (cond) WITH CHECK (cond);
DROP POLICY p ON t;
```
### Session Management
```sql
SET ROLE role_name;
RESET ROLE;
SELECT current_user, session_user;
```
### Permission Checking
```sql
SELECT has_table_privilege('table', 'SELECT');
SELECT has_column_privilege('table', 'col', 'UPDATE');
```
---
# libcognica-database C API
## 1. Getting Started
### 1.1 Minimal Example
```c
#include
#include
int main(void) {
cognica_database db = NULL;
cognica_connection conn = NULL;
cognica_result result = NULL;
cognica_open("/tmp/mydb", &db);
cognica_connect(db, &conn);
cognica_execute(conn, "CREATE TABLE t (id INT64 PRIMARY KEY, name VARCHAR)", &result);
cognica_result_destroy(result);
cognica_execute(conn, "INSERT INTO t VALUES (1, 'Alice')", &result);
cognica_result_destroy(result);
cognica_execute(conn, "SELECT id, name FROM t", &result);
for (int64_t r = 0; r < cognica_result_row_count(result); ++r) {
int64_t len = 0;
printf("id=%lld name=%.*s\n",
(long long)cognica_result_get_int64(result, r, 0),
(int)len, cognica_result_get_varchar(result, r, 1, &len));
}
cognica_result_destroy(result);
cognica_disconnect(conn);
cognica_close(db);
return 0;
}
```
Build: `cc -o example example.c -lcognica-database`
### 1.2 Handle Ownership Model
All resources follow a strict parent-child hierarchy. A parent handle refuses to close while any child handle is still alive, returning `COGNICA_ERROR_MISUSE`. Resources must be destroyed in reverse order of creation.
```
cognica_database
+-- cognica_connection
+-- cognica_result
+-- cognica_prepared
+-- cognica_cursor
+-- cognica_appender
```
Multiple `cognica_database` handles may coexist in a single process. Each owns an independent engine instance with no shared global state.
### 1.3 Error Handling Convention
Every function that can fail returns `cognica_status`. On failure, a human-readable error message is available from the nearest handle:
- Database-level errors: `cognica_database_errmsg(db)`
- Connection-level errors: `cognica_connection_errmsg(conn)`
The returned string pointer is valid until the next operation on the same handle. Callers should copy the string if it needs to outlive that window.
### 1.4 Thread Safety
A `cognica_database` handle is safe to share across threads. A `cognica_connection` handle is not: the caller must serialize all access to a given connection. The recommended pattern is one connection per thread.
---
## 2. Types and Constants
### 2.1 Opaque Handle Types
| Type | Description |
|------|-------------|
| `cognica_database` | An open database instance. |
| `cognica_connection` | A session on a database. |
| `cognica_prepared` | A compiled SQL statement with bindable parameters. |
| `cognica_result` | A materialized result set from a query. |
| `cognica_cursor` | A streaming cursor for batch-wise result fetching. |
| `cognica_document` | A JSON-like document with typed fields. |
| `cognica_appender` | A bulk-insert handle for a specific table. |
| `cognica_config` | A configuration handle for database initialization. |
All handle types are pointers to opaque structs. The internal layout is not part of the public API.
### 2.2 Status Codes
```c
typedef enum {
COGNICA_OK = 0,
COGNICA_ERROR = 1,
COGNICA_ERROR_INVALID_ARG = 2,
COGNICA_ERROR_NOT_FOUND = 3,
COGNICA_ERROR_EXISTS = 4,
COGNICA_ERROR_PERMISSION = 5,
COGNICA_ERROR_ABORTED = 6,
COGNICA_ERROR_CANCELLED = 7,
COGNICA_ERROR_OOM = 8,
COGNICA_ERROR_IO = 9,
COGNICA_ERROR_CORRUPT = 10,
COGNICA_ERROR_BUSY = 11,
COGNICA_ERROR_MISUSE = 12,
COGNICA_ERROR_SCHEMA = 13,
COGNICA_ERROR_CONSTRAINT = 14,
COGNICA_ERROR_TYPE = 15,
COGNICA_ERROR_OVERFLOW = 16,
COGNICA_ERROR_SYNTAX = 17,
COGNICA_ERROR_LIMIT = 18
} cognica_status;
```
| Code | Name | Description |
|------|------|-------------|
| 0 | `COGNICA_OK` | Success. |
| 1 | `COGNICA_ERROR` | Generic error. Check the error message for details. |
| 2 | `COGNICA_ERROR_INVALID_ARG` | A NULL pointer or out-of-range value was passed. |
| 3 | `COGNICA_ERROR_NOT_FOUND` | A table, column, or named object does not exist. |
| 4 | `COGNICA_ERROR_EXISTS` | The object already exists (e.g., `CREATE TABLE` without `IF NOT EXISTS`). |
| 5 | `COGNICA_ERROR_PERMISSION` | Permission denied. |
| 6 | `COGNICA_ERROR_ABORTED` | Transaction aborted due to a conflict. |
| 7 | `COGNICA_ERROR_CANCELLED` | Query was cancelled via `cognica_cancel`. |
| 8 | `COGNICA_ERROR_OOM` | Out of memory. |
| 9 | `COGNICA_ERROR_IO` | Disk I/O error. |
| 10 | `COGNICA_ERROR_CORRUPT` | Data corruption detected. |
| 11 | `COGNICA_ERROR_BUSY` | Resource is busy due to lock contention. |
| 12 | `COGNICA_ERROR_MISUSE` | API misuse, such as closing a parent while children are alive. |
| 13 | `COGNICA_ERROR_SCHEMA` | Schema violation. |
| 14 | `COGNICA_ERROR_CONSTRAINT` | Constraint violation (e.g., `UNIQUE`, `NOT NULL`). |
| 15 | `COGNICA_ERROR_TYPE` | Type mismatch. |
| 16 | `COGNICA_ERROR_OVERFLOW` | Numeric overflow. |
| 17 | `COGNICA_ERROR_SYNTAX` | SQL syntax error. |
| 18 | `COGNICA_ERROR_LIMIT` | A resource limit was exceeded. |
### 2.3 Value Type Codes
```c
typedef enum {
COGNICA_TYPE_NULL = 0x00,
COGNICA_TYPE_BOOLEAN = 0x01,
COGNICA_TYPE_INT64 = 0x02,
COGNICA_TYPE_DOUBLE = 0x03,
COGNICA_TYPE_VARCHAR = 0x04,
COGNICA_TYPE_ARRAY = 0x05,
COGNICA_TYPE_DOCUMENT = 0x06,
COGNICA_TYPE_BLOB = 0x07,
COGNICA_TYPE_TIMESTAMP = 0x08,
COGNICA_TYPE_TIMESTAMPTZ = 0x09,
COGNICA_TYPE_DATE = 0x0A,
COGNICA_TYPE_TIME = 0x0B,
COGNICA_TYPE_INTERVAL = 0x0C,
COGNICA_TYPE_DECIMAL = 0x0D,
COGNICA_TYPE_UUID = 0x0E
} cognica_type;
```
These values are returned by `cognica_result_column_type` and `cognica_document_field_type`. They match the internal CVM type system 1:1.
### 2.4 Temporal Value Types
| Type | C Type | Representation |
|------|--------|----------------|
| `cognica_timestamp` | `int64_t` | Microseconds since Unix epoch (1970-01-01T00:00:00Z). |
| `cognica_date` | `int32_t` | Days since Unix epoch (1970-01-01). |
| `cognica_time` | `int64_t` | Microseconds since midnight. |
| `cognica_interval` | `struct` | Composite of `int32_t months`, `int32_t days`, and `int64_t microseconds`. Follows the PostgreSQL interval convention. |
### 2.5 Transaction Isolation Levels
```c
typedef enum {
COGNICA_ISOLATION_READ_UNCOMMITTED = 0,
COGNICA_ISOLATION_READ_COMMITTED = 1,
COGNICA_ISOLATION_REPEATABLE_READ = 2,
COGNICA_ISOLATION_SERIALIZABLE = 3
} cognica_isolation_level;
```
### 2.6 Version Constants
```c
#define COGNICA_VERSION_MAJOR 1
#define COGNICA_VERSION_MINOR 0
#define COGNICA_VERSION_PATCH 0
```
---
## 3. Version
#### `cognica_version`
```c
const char* cognica_version(void);
```
Returns the version string in `"major.minor.patch"` format. The pointer is valid for the lifetime of the process and must not be freed.
#### `cognica_version_major`
```c
int32_t cognica_version_major(void);
```
Returns the major version number.
#### `cognica_version_minor`
```c
int32_t cognica_version_minor(void);
```
Returns the minor version number.
#### `cognica_version_patch`
```c
int32_t cognica_version_patch(void);
```
Returns the patch version number.
---
## 4. Configuration
Configuration handles control how a database is opened. They are optional: `cognica_open` uses library defaults. For fine-grained control, create a config, set keys, and pass it to `cognica_open_with_config`.
#### `cognica_config_create`
```c
cognica_status cognica_config_create(cognica_config* out_config);
```
Creates a configuration initialized to library defaults.
| Parameter | Description |
|-----------|-------------|
| `out_config` | Receives the new config handle. Must not be NULL. |
**Returns:** `COGNICA_OK` on success.
#### `cognica_config_create_from_file`
```c
cognica_status cognica_config_create_from_file(const char* path, cognica_config* out_config);
```
Creates a configuration by parsing a YAML file. All keys in the file are validated; unknown keys produce `COGNICA_ERROR_INVALID_ARG`.
| Parameter | Description |
|-----------|-------------|
| `path` | Path to a YAML configuration file. Must not be NULL. |
| `out_config` | Receives the new config handle. |
**Returns:** `COGNICA_OK` on success; `COGNICA_ERROR_IO` if the file cannot be read.
#### `cognica_config_set`
```c
cognica_status cognica_config_set(cognica_config config, const char* key, const char* value);
```
Sets a configuration value by dot-separated key.
| Parameter | Description |
|-----------|-------------|
| `config` | A config handle. |
| `key` | Dot-separated key, e.g., `"db.storage.db_path"`, `"db.storage.write_buffer_size"`. |
| `value` | String representation of the value. |
**Returns:** `COGNICA_OK` on success; `COGNICA_ERROR_INVALID_ARG` if the key is unknown.
#### `cognica_config_destroy`
```c
void cognica_config_destroy(cognica_config config);
```
Destroys a configuration handle and frees its memory. Safe to call with NULL. Do not call this on a config that was consumed by a successful `cognica_open_with_config` call.
---
## 5. Database Lifecycle
#### `cognica_open`
```c
cognica_status cognica_open(const char* path, cognica_database* out_db);
```
Opens a database at the given directory path using library defaults. Creates the directory if it does not exist. If the directory already contains a Cognica database, the existing data is opened.
| Parameter | Description |
|-----------|-------------|
| `path` | Filesystem path for the database directory. Must not be NULL. |
| `out_db` | Receives the database handle. Must not be NULL. |
**Returns:** `COGNICA_OK` on success; `COGNICA_ERROR_INVALID_ARG` if either argument is NULL; `COGNICA_ERROR_IO` on filesystem errors.
#### `cognica_open_with_config`
```c
cognica_status cognica_open_with_config(cognica_config config, cognica_database* out_db);
```
Opens a database using a custom configuration. The database path is read from the `"db.storage.db_path"` key in the configuration.
On success, the config handle is **consumed** and must not be used or destroyed afterwards. On failure, the config remains valid and the caller is still responsible for destroying it.
| Parameter | Description |
|-----------|-------------|
| `config` | A configuration handle. Consumed on success. |
| `out_db` | Receives the database handle. |
**Returns:** `COGNICA_OK` on success.
#### `cognica_close`
```c
cognica_status cognica_close(cognica_database db);
```
Closes a database and releases all associated resources.
| Scenario | Behavior |
|----------|----------|
| `db` is NULL | Returns `COGNICA_OK` with no effect (idempotent). |
| All child connections are disconnected | Returns `COGNICA_OK`; the handle is destroyed. |
| Child connections are still live | Returns `COGNICA_ERROR_MISUSE`; the handle remains intact. The caller must disconnect children first, then call `cognica_close` again. |
Callers must check the return value. Discarding the pointer without confirming `COGNICA_OK` is a potential memory leak.
#### `cognica_database_errmsg`
```c
const char* cognica_database_errmsg(cognica_database db);
```
Returns the error message from the most recent failed operation on this database handle. Returns an empty string (`""`) when there is no error. Returns `""` when `db` is NULL. The pointer is valid until the next operation on the same handle.
#### `cognica_database_path`
```c
const char* cognica_database_path(cognica_database db);
```
Returns the filesystem path of the database. Returns NULL when `db` is NULL. The pointer is valid for the lifetime of the database handle.
---
## 6. Connection Management
#### `cognica_connect`
```c
cognica_status cognica_connect(cognica_database db, cognica_connection* out_conn);
```
Creates a new connection (session) on the given database. Multiple connections may exist simultaneously on the same database. Each connection has independent transaction state.
| Parameter | Description |
|-----------|-------------|
| `db` | An open database handle. |
| `out_conn` | Receives the connection handle. |
**Returns:** `COGNICA_OK` on success.
#### `cognica_disconnect`
```c
cognica_status cognica_disconnect(cognica_connection conn);
```
Destroys a connection. Any active transaction is rolled back as part of destruction.
| Scenario | Behavior |
|----------|----------|
| `conn` is NULL | Returns `COGNICA_OK` with no effect (idempotent). |
| No child handles (results, prepared statements, cursors, appenders) | Returns `COGNICA_OK`; the connection is destroyed. |
| Child handles are still live | Returns `COGNICA_ERROR_MISUSE`; the connection remains intact. |
#### `cognica_connection_errmsg`
```c
const char* cognica_connection_errmsg(cognica_connection conn);
```
Returns the error message from the most recent failed operation on this connection. Returns `""` when there is no error or when `conn` is NULL.
---
## 7. SQL Execution
#### `cognica_execute`
```c
cognica_status cognica_execute(cognica_connection conn, const char* sql, cognica_result* out_result);
```
Executes a SQL statement and returns a materialized result set.
| Statement Type | Result Behavior |
|----------------|-----------------|
| SELECT | `cognica_result_row_count` returns the number of rows. |
| INSERT, UPDATE, DELETE | `cognica_result_rows_affected` returns the number of modified rows. `cognica_result_row_count` returns 0. |
| DDL (CREATE, DROP, ALTER) | The result is empty. Both row count and affected rows are 0. |
The caller must destroy the result with `cognica_result_destroy` regardless of statement type. On failure, `*out_result` is set to NULL and no result needs to be destroyed.
| Parameter | Description |
|-----------|-------------|
| `conn` | A connection handle. Must not be NULL. |
| `sql` | A null-terminated SQL string. Must not be NULL. |
| `out_result` | Receives the result handle. Must not be NULL. |
**Returns:** `COGNICA_OK` on success; `COGNICA_ERROR_SYNTAX` for parse errors; other codes for execution errors.
---
## 8. Result Set Access
### 8.1 Metadata
#### `cognica_result_row_count`
```c
int64_t cognica_result_row_count(cognica_result result);
```
Returns the number of rows in the result set. Returns 0 for DML and DDL results.
#### `cognica_result_column_count`
```c
int32_t cognica_result_column_count(cognica_result result);
```
Returns the number of columns in the result set.
#### `cognica_result_column_name`
```c
const char* cognica_result_column_name(cognica_result result, int32_t col);
```
Returns the name of column `col` (0-based). The pointer is valid for the lifetime of the result.
#### `cognica_result_column_index`
```c
int32_t cognica_result_column_index(cognica_result result, const char* name);
```
Returns the 0-based index of the column with the given name. Returns -1 if no such column exists.
#### `cognica_result_column_type`
```c
cognica_type cognica_result_column_type(cognica_result result, int64_t row, int32_t col);
```
Returns the `cognica_type` of the value at position (`row`, `col`). The type is per-cell, not per-column, because columns in a semi-structured result set may contain values of different types across rows.
#### `cognica_result_rows_scanned`
```c
int64_t cognica_result_rows_scanned(cognica_result result);
```
Returns the total number of rows scanned during query execution. This may be larger than the returned row count due to filtering.
#### `cognica_result_rows_affected`
```c
int64_t cognica_result_rows_affected(cognica_result result);
```
Returns the number of rows affected by a DML statement (INSERT, UPDATE, DELETE). Returns 0 for SELECT and DDL.
#### `cognica_result_destroy`
```c
void cognica_result_destroy(cognica_result result);
```
Destroys the result and frees its memory. Safe to call with NULL. All pointers previously obtained from this result (column names, varchar values, blob data) become invalid.
### 8.2 Value Accessors
All value accessors take a 0-based `row` index and a 0-based `col` index.
#### `cognica_result_is_null`
```c
int32_t cognica_result_is_null(cognica_result result, int64_t row, int32_t col);
```
Returns 1 if the value at (`row`, `col`) is SQL NULL, 0 otherwise. Always check for NULL before calling type-specific getters.
#### `cognica_result_get_bool`
```c
int32_t cognica_result_get_bool(cognica_result result, int64_t row, int32_t col);
```
Returns 1 for true, 0 for false.
#### `cognica_result_get_int64`
```c
int64_t cognica_result_get_int64(cognica_result result, int64_t row, int32_t col);
```
Returns the INT64 value at (`row`, `col`).
#### `cognica_result_get_double`
```c
double cognica_result_get_double(cognica_result result, int64_t row, int32_t col);
```
Returns the DOUBLE value at (`row`, `col`).
#### `cognica_result_get_varchar`
```c
const char* cognica_result_get_varchar(cognica_result result, int64_t row, int32_t col, int64_t* out_length);
```
Returns a pointer to the VARCHAR data and writes its byte length to `*out_length`. The returned pointer is valid for the lifetime of the result. The data is not guaranteed to be null-terminated; always use the length.
#### `cognica_result_get_blob`
```c
const void* cognica_result_get_blob(cognica_result result, int64_t row, int32_t col, int64_t* out_length);
```
Returns a pointer to the BLOB data and writes its byte length to `*out_length`. The pointer is valid for the lifetime of the result.
#### `cognica_result_get_timestamp`
```c
cognica_timestamp cognica_result_get_timestamp(cognica_result result, int64_t row, int32_t col);
```
Returns the TIMESTAMP value as microseconds since the Unix epoch.
#### `cognica_result_get_date`
```c
cognica_date cognica_result_get_date(cognica_result result, int64_t row, int32_t col);
```
Returns the DATE value as days since the Unix epoch.
#### `cognica_result_get_time`
```c
cognica_time cognica_result_get_time(cognica_result result, int64_t row, int32_t col);
```
Returns the TIME value as microseconds since midnight.
#### `cognica_result_get_interval`
```c
cognica_interval cognica_result_get_interval(cognica_result result, int64_t row, int32_t col);
```
Returns the INTERVAL value as a `cognica_interval` struct with `months`, `days`, and `microseconds` fields.
#### `cognica_result_get_uuid`
```c
const char* cognica_result_get_uuid(cognica_result result, int64_t row, int32_t col);
```
Returns the UUID value as a null-terminated string in standard `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` format. The pointer is valid for the lifetime of the result.
#### `cognica_result_get_decimal`
```c
const char* cognica_result_get_decimal(cognica_result result, int64_t row, int32_t col);
```
Returns the DECIMAL value as a null-terminated string. The pointer is valid for the lifetime of the result.
#### `cognica_result_get_document`
```c
cognica_status cognica_result_get_document(cognica_result result, int64_t row, cognica_document* out_doc);
```
Returns a read-only document view of the row at index `row`. The view is backed by the result's row data and becomes invalid when the result is destroyed. The handle itself is heap-allocated: the caller must call `cognica_document_destroy` to free the wrapper. To obtain a copy that outlives the result, call `cognica_document_clone` on the returned view. Mutations on a view-mode document are rejected with `COGNICA_ERROR_MISUSE`.
---
## 9. Prepared Statements
Prepared statements allow a SQL query to be parsed and compiled once, then executed multiple times with different parameter values. Parameters are referenced in SQL as `$1`, `$2`, etc. Binding indices are 1-based to match.
### 9.1 Lifecycle
#### `cognica_prepared_create`
```c
cognica_status cognica_prepared_create(cognica_connection conn, const char* sql, cognica_prepared* out_stmt);
```
Parses and compiles a SQL statement. The statement may contain parameter placeholders `$1`, `$2`, etc.
| Parameter | Description |
|-----------|-------------|
| `conn` | A connection handle. |
| `sql` | A null-terminated SQL string with optional `$N` placeholders. |
| `out_stmt` | Receives the prepared statement handle. |
**Returns:** `COGNICA_OK` on success; `COGNICA_ERROR_SYNTAX` for parse errors.
#### `cognica_prepared_destroy`
```c
void cognica_prepared_destroy(cognica_prepared stmt);
```
Destroys a prepared statement. Safe to call with NULL.
#### `cognica_prepared_param_count`
```c
int32_t cognica_prepared_param_count(cognica_prepared stmt);
```
Returns the number of parameters (`$N` placeholders) in the prepared statement.
#### `cognica_prepared_column_count`
```c
int32_t cognica_prepared_column_count(cognica_prepared stmt);
```
Returns the number of result columns. Returns 0 for DML and DDL statements.
#### `cognica_prepared_column_name`
```c
const char* cognica_prepared_column_name(cognica_prepared stmt, int32_t col);
```
Returns the name of result column `col` (0-based).
#### `cognica_prepared_column_index`
```c
int32_t cognica_prepared_column_index(cognica_prepared stmt, const char* name);
```
Returns the 0-based index of the result column with the given name. Returns -1 if not found.
### 9.2 Parameter Binding
All binding functions take a 1-based `idx` matching the `$1`, `$2`, ... placeholders in the SQL string. All parameters default to SQL NULL until explicitly bound.
#### `cognica_bind_null`
```c
cognica_status cognica_bind_null(cognica_prepared stmt, int32_t idx);
```
Binds parameter `idx` to SQL NULL.
#### `cognica_bind_bool`
```c
cognica_status cognica_bind_bool(cognica_prepared stmt, int32_t idx, int32_t value);
```
Binds parameter `idx` to a boolean value. 0 is false; any non-zero value is true.
#### `cognica_bind_int64`
```c
cognica_status cognica_bind_int64(cognica_prepared stmt, int32_t idx, int64_t value);
```
Binds parameter `idx` to a 64-bit integer.
#### `cognica_bind_double`
```c
cognica_status cognica_bind_double(cognica_prepared stmt, int32_t idx, double value);
```
Binds parameter `idx` to a double-precision floating-point number.
#### `cognica_bind_varchar`
```c
cognica_status cognica_bind_varchar(cognica_prepared stmt, int32_t idx, const char* value, int64_t length);
```
Binds parameter `idx` to a string. The library copies the data; the caller may free the buffer after this call returns. Pass -1 for `length` to use `strlen(value)`.
#### `cognica_bind_blob`
```c
cognica_status cognica_bind_blob(cognica_prepared stmt, int32_t idx, const void* data, int64_t length);
```
Binds parameter `idx` to a binary blob. The library copies the data.
#### `cognica_bind_timestamp`
```c
cognica_status cognica_bind_timestamp(cognica_prepared stmt, int32_t idx, cognica_timestamp value);
```
Binds parameter `idx` to a timestamp (microseconds since Unix epoch).
#### `cognica_bind_date`
```c
cognica_status cognica_bind_date(cognica_prepared stmt, int32_t idx, cognica_date value);
```
Binds parameter `idx` to a date (days since Unix epoch).
#### `cognica_bind_time`
```c
cognica_status cognica_bind_time(cognica_prepared stmt, int32_t idx, cognica_time value);
```
Binds parameter `idx` to a time (microseconds since midnight).
#### `cognica_bind_interval`
```c
cognica_status cognica_bind_interval(cognica_prepared stmt, int32_t idx, cognica_interval value);
```
Binds parameter `idx` to an interval.
#### `cognica_clear_bindings`
```c
cognica_status cognica_clear_bindings(cognica_prepared stmt);
```
Resets all bound parameters to SQL NULL. Call this between successive executions of the same prepared statement when rebinding all parameters.
### 9.3 Execution
#### `cognica_prepared_execute`
```c
cognica_status cognica_prepared_execute(cognica_prepared stmt, cognica_connection conn, cognica_result* out_result);
```
Executes a prepared statement with its currently bound parameters. The result set follows the same conventions as `cognica_execute`. Bound parameter values are preserved after execution; the same statement can be executed again without rebinding.
---
## 10. Streaming Cursor
For result sets that may be large, the streaming cursor API fetches rows in batches. This avoids loading the entire result set into memory.
#### `cognica_cursor_open`
```c
cognica_status cognica_cursor_open(cognica_connection conn, const char* sql, cognica_cursor* out_cursor);
```
Opens a streaming cursor for the given SQL query. The query begins execution but rows are not materialized until `cognica_cursor_fetch` is called.
#### `cognica_cursor_fetch`
```c
cognica_status cognica_cursor_fetch(cognica_cursor cursor, int64_t batch_size, cognica_result* out_result);
```
Fetches the next batch of up to `batch_size` rows. The returned result must be destroyed with `cognica_result_destroy` before fetching the next batch. When no more rows remain, `*out_result` is set to NULL and the function returns `COGNICA_OK`.
#### `cognica_cursor_has_more`
```c
int32_t cognica_cursor_has_more(cognica_cursor cursor);
```
Returns 1 if additional rows may be available, 0 if the cursor is exhausted.
#### `cognica_cursor_rows_returned`
```c
int64_t cognica_cursor_rows_returned(cognica_cursor cursor);
```
Returns the cumulative number of rows returned across all fetch calls on this cursor.
#### `cognica_cursor_rows_scanned`
```c
int64_t cognica_cursor_rows_scanned(cognica_cursor cursor);
```
Returns the cumulative number of rows scanned during query execution.
#### `cognica_cursor_close`
```c
void cognica_cursor_close(cognica_cursor cursor);
```
Closes the cursor and releases its resources. Safe to call with NULL. Always close cursors when done, even if not all rows were fetched.
---
## 11. Transaction Control
Without explicit transaction control, each SQL statement executes in auto-commit mode.
### 11.1 Basic Transactions
#### `cognica_transaction_begin`
```c
cognica_status cognica_transaction_begin(cognica_connection conn);
```
Begins a transaction with the connection's current default isolation level.
#### `cognica_transaction_begin_with_isolation`
```c
cognica_status cognica_transaction_begin_with_isolation(cognica_connection conn, cognica_isolation_level level);
```
Begins a transaction with the specified isolation level.
#### `cognica_transaction_commit`
```c
cognica_status cognica_transaction_commit(cognica_connection conn);
```
Commits the active transaction. Returns `COGNICA_ERROR_ABORTED` if a conflict is detected during commit.
#### `cognica_transaction_rollback`
```c
cognica_status cognica_transaction_rollback(cognica_connection conn);
```
Rolls back the active transaction, undoing all changes since the last `BEGIN`.
#### `cognica_transaction_is_active`
```c
int32_t cognica_transaction_is_active(cognica_connection conn);
```
Returns 1 if a transaction is active on this connection, 0 otherwise.
#### `cognica_transaction_set_isolation`
```c
cognica_status cognica_transaction_set_isolation(cognica_connection conn, cognica_isolation_level level);
```
Sets the default isolation level for subsequent transactions on this connection.
#### `cognica_transaction_get_isolation`
```c
cognica_isolation_level cognica_transaction_get_isolation(cognica_connection conn);
```
Returns the current isolation level of the connection.
### 11.2 Savepoints
Savepoints enable partial rollback within a transaction.
#### `cognica_savepoint_create`
```c
cognica_status cognica_savepoint_create(cognica_connection conn, const char* name);
```
Creates a named savepoint within the active transaction.
#### `cognica_savepoint_rollback`
```c
cognica_status cognica_savepoint_rollback(cognica_connection conn, const char* name);
```
Rolls back all changes made after the named savepoint. The savepoint remains valid and can be rolled back to again.
#### `cognica_savepoint_release`
```c
cognica_status cognica_savepoint_release(cognica_connection conn, const char* name);
```
Releases (destroys) a savepoint, merging its changes into the enclosing transaction. After release, the savepoint name can no longer be used for rollback.
---
## 12. Session Parameters and Multi-Database
#### `cognica_set_param`
```c
cognica_status cognica_set_param(cognica_connection conn, const char* key, const char* value);
```
Sets a session parameter. Equivalent to `SET key = value` in SQL.
#### `cognica_get_param`
```c
const char* cognica_get_param(cognica_connection conn, const char* key);
```
Returns the current value of a session parameter. Returns NULL if the parameter is not set.
#### `cognica_use_database`
```c
cognica_status cognica_use_database(cognica_connection conn, const char* database_name);
```
Switches the active database on this connection. Equivalent to `USE database_name` in SQL.
#### `cognica_current_database`
```c
const char* cognica_current_database(cognica_connection conn);
```
Returns the name of the currently active database on this connection.
#### `cognica_cancel`
```c
cognica_status cognica_cancel(cognica_connection conn);
```
Cooperatively cancels the currently executing query on this connection. This function is safe to call from a thread other than the one executing the query. The executing query will return `COGNICA_ERROR_CANCELLED` at its next cancellation check point.
---
## 13. Document API
The document API provides a JSON-like object model for building and inspecting structured data. Documents are used with the appender API and can be extracted from result rows via `cognica_result_get_document`. Field names support dot notation for nested access (e.g., `"address.city"`).
### 13.1 Lifecycle
#### `cognica_document_create`
```c
cognica_status cognica_document_create(cognica_document* out_doc);
```
Creates a new empty document. The caller owns the handle and must destroy it with `cognica_document_destroy`.
#### `cognica_document_from_json`
```c
cognica_status cognica_document_from_json(const char* json, int64_t length, cognica_document* out_doc);
```
Parses a JSON string into a document. Pass -1 for `length` to use `strlen(json)`.
#### `cognica_document_to_json`
```c
const char* cognica_document_to_json(cognica_document doc, int64_t* out_length);
```
Serializes the document to a JSON string. Writes the byte length to `*out_length` (pass NULL to ignore). The returned pointer is valid for the lifetime of the document or until the next mutation.
#### `cognica_document_clone`
```c
cognica_status cognica_document_clone(cognica_document doc, cognica_document* out_doc);
```
Creates a deep copy of a document. The clone is fully independent: it can outlive the original and supports mutation even if the original was a read-only view.
#### `cognica_document_destroy`
```c
void cognica_document_destroy(cognica_document doc);
```
Destroys a document handle. For owned documents, this frees the backing data. For view-mode documents (obtained from `cognica_result_get_document` or `cognica_document_get_document`), this frees only the wrapper; the backing data remains owned by the parent. Safe to call with NULL.
### 13.2 Field Inspection
#### `cognica_document_has`
```c
int32_t cognica_document_has(cognica_document doc, const char* field);
```
Returns 1 if the document contains the named field, 0 otherwise. Supports dot notation.
#### `cognica_document_remove`
```c
cognica_status cognica_document_remove(cognica_document doc, const char* field);
```
Removes a field from the document.
#### `cognica_document_field_count`
```c
int32_t cognica_document_field_count(cognica_document doc);
```
Returns the number of top-level fields in the document.
#### `cognica_document_field_type`
```c
cognica_type cognica_document_field_type(cognica_document doc, const char* field);
```
Returns the `cognica_type` of the named field. Returns `COGNICA_TYPE_NULL` if the field does not exist.
### 13.3 Field Getters
All getters support dot notation for nested field access. On success, the value is written to the out-parameter and `COGNICA_OK` is returned. On failure (e.g., field not found, type mismatch), the out-parameter is unchanged and an error status is returned.
#### `cognica_document_get_bool`
```c
cognica_status cognica_document_get_bool(cognica_document doc, const char* field, int32_t* out_value);
```
#### `cognica_document_get_int64`
```c
cognica_status cognica_document_get_int64(cognica_document doc, const char* field, int64_t* out_value);
```
#### `cognica_document_get_double`
```c
cognica_status cognica_document_get_double(cognica_document doc, const char* field, double* out_value);
```
#### `cognica_document_get_varchar`
```c
cognica_status cognica_document_get_varchar(cognica_document doc, const char* field, const char** out_value, int64_t* out_length);
```
Returns a pointer to the string data and its byte length. The pointer is valid for the lifetime of the document or until the next mutation on the same field.
#### `cognica_document_get_blob`
```c
cognica_status cognica_document_get_blob(cognica_document doc, const char* field, const void** out_value, int64_t* out_length);
```
Returns a pointer to the binary data and its byte length.
#### `cognica_document_get_document`
```c
cognica_status cognica_document_get_document(cognica_document doc, const char* field, cognica_document* out_child);
```
Returns a read-only view into the nested object at `field`. The caller must call `cognica_document_destroy` on the returned handle to free the wrapper. The view becomes invalid when the parent document is destroyed. Use `cognica_document_clone` to obtain an owned copy.
### 13.4 Field Setters
#### `cognica_document_set_null`
```c
cognica_status cognica_document_set_null(cognica_document doc, const char* field);
```
Sets the named field to SQL NULL.
#### `cognica_document_set_bool`
```c
cognica_status cognica_document_set_bool(cognica_document doc, const char* field, int32_t value);
```
#### `cognica_document_set_int64`
```c
cognica_status cognica_document_set_int64(cognica_document doc, const char* field, int64_t value);
```
#### `cognica_document_set_double`
```c
cognica_status cognica_document_set_double(cognica_document doc, const char* field, double value);
```
#### `cognica_document_set_varchar`
```c
cognica_status cognica_document_set_varchar(cognica_document doc, const char* field, const char* value, int64_t length);
```
Sets a string field. The library copies the data. Pass -1 for `length` to use `strlen(value)`.
#### `cognica_document_set_blob`
```c
cognica_status cognica_document_set_blob(cognica_document doc, const char* field, const void* data, int64_t length);
```
Sets a binary field. The library copies the data.
#### `cognica_document_set_document`
```c
cognica_status cognica_document_set_document(cognica_document doc, const char* field, cognica_document value);
```
Sets a nested document field. The library clones the value; the caller retains ownership of the original.
### 13.5 Array Operations
#### `cognica_document_set_array`
```c
cognica_status cognica_document_set_array(cognica_document doc, const char* field);
```
Creates or replaces the named field with an empty array.
#### `cognica_document_array_length`
```c
int32_t cognica_document_array_length(cognica_document doc, const char* field);
```
Returns the number of elements in the array at `field`. Returns 0 if the field does not exist or is not an array.
#### `cognica_document_array_append_int64`
```c
cognica_status cognica_document_array_append_int64(cognica_document doc, const char* field, int64_t value);
```
Appends an integer element to the array at `field`.
#### `cognica_document_array_append_varchar`
```c
cognica_status cognica_document_array_append_varchar(cognica_document doc, const char* field, const char* value, int64_t length);
```
Appends a string element. Pass -1 for `length` to use `strlen(value)`.
#### `cognica_document_array_append_document`
```c
cognica_status cognica_document_array_append_document(cognica_document doc, const char* field, cognica_document value);
```
Appends a document element. The library clones the value.
#### `cognica_document_array_get_int64`
```c
cognica_status cognica_document_array_get_int64(cognica_document doc, const char* field, int32_t index, int64_t* out_value);
```
Reads the integer element at 0-based `index` in the array at `field`.
#### `cognica_document_array_get_varchar`
```c
cognica_status cognica_document_array_get_varchar(cognica_document doc, const char* field, int32_t index, const char** out_value, int64_t* out_length);
```
Reads the string element at 0-based `index`.
#### `cognica_document_array_get_document`
```c
cognica_status cognica_document_array_get_document(cognica_document doc, const char* field, int32_t index, cognica_document* out_child);
```
Returns a read-only view of the document element at 0-based `index`. Same ownership rules as `cognica_document_get_document`.
---
## 14. Appender (Bulk Insert)
The appender API provides high-throughput bulk insertion by bypassing the SQL parser and feeding rows directly into the storage engine. Values are appended in column order as defined by the table schema.
#### `cognica_appender_create`
```c
cognica_status cognica_appender_create(cognica_connection conn, const char* table, cognica_appender* out_appender);
```
Creates an appender for the named table. The table must already exist.
#### `cognica_appender_begin_row`
```c
cognica_status cognica_appender_begin_row(cognica_appender appender);
```
Begins a new row. After this call, append values for each column in order using the `cognica_appender_append_*` functions.
#### `cognica_appender_append_null`
```c
cognica_status cognica_appender_append_null(cognica_appender appender);
```
Appends a NULL value for the next column.
#### `cognica_appender_append_bool`
```c
cognica_status cognica_appender_append_bool(cognica_appender appender, int32_t value);
```
#### `cognica_appender_append_int64`
```c
cognica_status cognica_appender_append_int64(cognica_appender appender, int64_t value);
```
#### `cognica_appender_append_double`
```c
cognica_status cognica_appender_append_double(cognica_appender appender, double value);
```
#### `cognica_appender_append_varchar`
```c
cognica_status cognica_appender_append_varchar(cognica_appender appender, const char* value, int64_t length);
```
Pass -1 for `length` to use `strlen(value)`.
#### `cognica_appender_append_blob`
```c
cognica_status cognica_appender_append_blob(cognica_appender appender, const void* data, int64_t length);
```
#### `cognica_appender_append_timestamp`
```c
cognica_status cognica_appender_append_timestamp(cognica_appender appender, cognica_timestamp value);
```
#### `cognica_appender_append_date`
```c
cognica_status cognica_appender_append_date(cognica_appender appender, cognica_date value);
```
#### `cognica_appender_append_time`
```c
cognica_status cognica_appender_append_time(cognica_appender appender, cognica_time value);
```
#### `cognica_appender_append_interval`
```c
cognica_status cognica_appender_append_interval(cognica_appender appender, cognica_interval value);
```
#### `cognica_appender_append_document`
```c
cognica_status cognica_appender_append_document(cognica_appender appender, cognica_document value);
```
Appends a document value for the next column.
#### `cognica_appender_end_row`
```c
cognica_status cognica_appender_end_row(cognica_appender appender);
```
Completes the current row. Returns an error if the number of appended values does not match the table's column count.
#### `cognica_appender_flush`
```c
cognica_status cognica_appender_flush(cognica_appender appender);
```
Flushes all buffered rows to storage. Rows are automatically flushed when the internal buffer is full, but explicit flushing provides a durability guarantee.
#### `cognica_appender_close`
```c
cognica_status cognica_appender_close(cognica_appender appender);
```
Flushes remaining rows and closes the appender. After this call, the appender must still be destroyed with `cognica_appender_destroy`.
#### `cognica_appender_destroy`
```c
void cognica_appender_destroy(cognica_appender appender);
```
Destroys the appender handle. If `cognica_appender_close` was not called, any unflushed rows are discarded. Safe to call with NULL.
---
## 15. Arrow C Data Interface
Header: `` (optional, include in addition to ``)
This header provides functions to export result sets and cursor batches as Apache Arrow record batches using the Arrow C Data Interface. The exported `ArrowSchema` and `ArrowArray` structs are ABI-stable and can be imported directly by any Arrow-compatible consumer (PyArrow, DuckDB, Polars, Arrow C++, DataFusion) without linking against an Arrow library.
#### `cognica_result_to_arrow`
```c
cognica_status cognica_result_to_arrow(cognica_result result, struct ArrowSchema* out_schema, struct ArrowArray* out_array);
```
Exports the entire result set as an Arrow record batch.
On success, both `out_schema` and `out_array` are populated. The caller is responsible for calling their respective `release()` callbacks when done. The result set must remain alive for the entire lifetime of the exported array; releasing the result before the Arrow consumer has finished reading is undefined behavior.
Supported types for zero-copy export: BOOLEAN, INT64, DOUBLE, VARCHAR. Other types are exported as VARCHAR (their string serialization).
#### `cognica_result_column_to_arrow`
```c
cognica_status cognica_result_column_to_arrow(cognica_result result, int32_t col, struct ArrowSchema* out_schema, struct ArrowArray* out_array);
```
Exports a single column as a flat Arrow array (not a record batch). The array length equals the result row count.
#### `cognica_cursor_fetch_arrow`
```c
cognica_status cognica_cursor_fetch_arrow(cognica_cursor cursor, int64_t batch_size, struct ArrowSchema* out_schema, struct ArrowArray* out_array);
```
Fetches the next batch from a streaming cursor and exports it directly as an Arrow record batch. This is a convenience function that avoids creating and destroying an intermediate `cognica_result`. When no more rows remain, `out_array->release` is set to NULL.
---
*Copyright 2023-2026 Cognica, Inc. All rights reserved.*
---
# Cognica Database Internals
## A Unified Data Processing System
**A comprehensive academic textbook on database engine architecture, query processing, and distributed systems.**
This book provides an in-depth exploration of the Cognica database engine, covering theoretical foundations, implementation details, and practical engineering considerations. From the unified query algebra that bridges relational, document, and full-text search paradigms to the bytecode virtual machine that executes queries, each chapter combines rigorous theory with real-world implementation insights.
---
## Author
**Jaepil Jeong** (jaepil@cognica.io)
---
## Part I: Foundations
*Establishing the theoretical framework for unified data processing*
| Chapter | Title | Topics |
|---------|-------|--------|
| [1](cognica-internals/1) | [Introduction to Unified Data Processing](cognica-internals/1) | Data paradigm fragmentation, unified algebra motivation, system architecture overview |
| [2](cognica-internals/2) | [Mathematical Foundations of Query Algebras](cognica-internals/2) | Set theory, Boolean algebra, posting lists, algebraic properties |
| [3](cognica-internals/3) | [Extending the Algebra to Graph Structures](cognica-internals/3) | Graph type system, path expressions, pattern matching, traversal algebra |
| [4](cognica-internals/4) | [Query Optimization Theory](cognica-internals/4) | Cost models, plan enumeration, join ordering, cardinality estimation |
---
## Part II: Storage Engine
*Building the persistent foundation for data management*
| Chapter | Title | Topics |
|---------|-------|--------|
| [5](cognica-internals/5) | [LSM-Tree Storage Architecture](cognica-internals/5) | Memtables, compaction strategies, write amplification, RocksDB integration |
| [6](cognica-internals/6) | [Document Storage and Schema Management](cognica-internals/6) | Document model, schema inference, flexible typing, collection management |
| [7](cognica-internals/7) | [Inverted Index Architecture](cognica-internals/7) | Posting lists, skip lists, term dictionaries, clustered term index |
---
## Part III: Query Processing
*Transforming SQL into executable plans*
| Chapter | Title | Topics |
|---------|-------|--------|
| [8](cognica-internals/8) | [SQL Parser and Semantic Analysis](cognica-internals/8) | libpg_query, AST construction, name resolution, type checking |
| [9](cognica-internals/9) | [Logical Planning and Optimization](cognica-internals/9) | Logical operators, equivalence rules, predicate pushdown, join reordering |
| [10](cognica-internals/10) | [Physical Planning and Execution Strategies](cognica-internals/10) | Physical operators, access path selection, parallel execution, plan caching |
---
## Part IV: Graph Query Processing
*From property graphs to Cypher-over-SQL*
| Chapter | Title | Topics |
|---------|-------|--------|
| [11](cognica-internals/11) | [Graph Storage and Operations](cognica-internals/11) | Property graph model, dual-collection storage, traversal algorithms, adjacency cache |
| [12](cognica-internals/12) | [Cypher Query Language](cognica-internals/12) | Cypher lexer/parser, AST, Cypher-to-SQL rewriting, recursive CTE path translation |
---
## Part V: Execution Engine
*The heart of query evaluation*
| Chapter | Title | Topics |
|---------|-------|--------|
| [13](cognica-internals/13) | [CVM Architecture](cognica-internals/13) | Bytecode VM design, register allocation, opcode dispatch, stack management |
| [14](cognica-internals/14) | [CVM Compilation Pipeline](cognica-internals/14) | IR generation, lowering passes, optimization phases, code generation |
| [15](cognica-internals/15) | [Vectorized Execution](cognica-internals/15) | SIMD operations, batch processing, columnar evaluation, filter pushdown |
| [16](cognica-internals/16) | [Copy-and-Patch JIT Compilation](cognica-internals/16) | Stencil-based JIT, runtime code generation, hot path optimization |
| [17](cognica-internals/17) | [Zero-Copy JOIN Implementation](cognica-internals/17) | Composite rows, reference semantics, memory efficiency, JOIN algorithms |
---
## Part VI: Similarity Search and Ranking
*Bridging structured and unstructured data retrieval*
| Chapter | Title | Topics |
|---------|-------|--------|
| [18](cognica-internals/18) | [Text Analysis Pipeline](cognica-internals/18) | Tokenization, normalization, stemming, stopwords, ICU integration |
| [19](cognica-internals/19) | [BM25 Scoring](cognica-internals/19) | TF-IDF, BM25 formula, IDF computation, term frequency saturation, numerical stability |
| [20](cognica-internals/20) | [Bayesian BM25 and Probabilistic Calibration](cognica-internals/20) | Sigmoid likelihood, composite prior, base rate prior, three-term decomposition, WAND compatibility |
| [21](cognica-internals/21) | [Vector Search and HNSW Index](cognica-internals/21) | Dense embeddings, ANN search, HNSW algorithm, distance metrics, quantization |
| [22](cognica-internals/22) | [Vector Score Calibration](cognica-internals/22) | Likelihood ratio calibration, cross-modal estimation, index-aware statistics, unified fusion |
| [23](cognica-internals/23) | [From Bayesian Inference to Neural Computation](cognica-internals/23) | Conjunction shrinkage, log-odds framework, neural emergence, activation functions, exact pruning |
| [24](cognica-internals/24) | [Hybrid Search Architecture](cognica-internals/24) | Score fusion, log-odds conjunction, multi-stage retrieval, query composition |
| [25](cognica-internals/25) | [Query Evaluation Strategies (WAND/BMW)](cognica-internals/25) | Top-K algorithms, early termination, block-max optimization, neural pruning |
---
## Part VII: Distributed Systems
*Scaling beyond a single node*
| Chapter | Title | Topics |
|---------|-------|--------|
| [26](cognica-internals/26) | [Raft Consensus Protocol](cognica-internals/26) | Leader election, log replication, safety guarantees, membership changes |
| [27](cognica-internals/27) | [Transaction Processing](cognica-internals/27) | ACID properties, isolation levels, MVCC, SSI, deadlock detection |
---
## Part VIII: System Integration
*Connecting to the outside world*
| Chapter | Title | Topics |
|---------|-------|--------|
| [28](cognica-internals/28) | [PostgreSQL Wire Protocol](cognica-internals/28) | Message formats, authentication, extended query protocol, COPY |
| [29](cognica-internals/29) | [External Table Integration](cognica-internals/29) | Foreign data wrappers, Arrow Flight SQL, predicate pushdown, federation |
| [30](cognica-internals/30) | [Multi-Protocol Service Layer](cognica-internals/30) | HTTP REST, Flight SQL, protocol multiplexing |
---
## Part IX: Advanced Topics
*Engineering for production excellence*
| Chapter | Title | Topics |
|---------|-------|--------|
| [31](cognica-internals/31) | [Memory Management](cognica-internals/31) | Arena allocation, memory pools, cache hierarchies, OOM handling |
| [32](cognica-internals/32) | [Observability and Debugging](cognica-internals/32) | Metrics, logging, tracing, profiling, EXPLAIN analysis |
| [33](cognica-internals/33) | [Performance Engineering](cognica-internals/33) | Benchmarking, bottleneck analysis, tuning strategies, capacity planning |
| [34](cognica-internals/34) | [Context-Isolated Architecture](cognica-internals/34) | ServerContext, singleton removal, bridge patterns, multi-tenancy foundations |
---
## Appendices
*Reference materials for implementation and deployment*
| Appendix | Title | Contents |
|----------|-------|----------|
| [A](cognica-internals/35) | [CVM Opcode Reference](cognica-internals/35) | Complete opcode listing, instruction formats, type system, built-in functions |
| [B](cognica-internals/36) | [SQL Compatibility Reference](cognica-internals/36) | Supported statements, data types, operators, functions, PostgreSQL compatibility |
| [C](cognica-internals/37) | [Configuration Reference](cognica-internals/37) | All configuration options with types, defaults, and tuning guidelines |
| [D](cognica-internals/38) | [API Reference](cognica-internals/38) | PostgreSQL protocol, Flight SQL, HTTP REST endpoints |
---
## Key Innovations
This book documents several key innovations in the Cognica database engine:
### Unified Query Algebra
A mathematical framework that treats posting lists as the universal abstraction, enabling seamless queries across relational, document, full-text search, and graph paradigms.
### Cypher-over-SQL Graph Processing
A parse-time Cypher-to-SQL rewriting engine that translates graph pattern matching into standard SQL subquery pipelines, enabling graph queries to benefit from the full SQL optimizer without a separate graph execution engine.
### Clustered Term Index
A novel inverted index organization that reduces key count by 62,500x compared to traditional term-per-key approaches, dramatically improving write performance.
### Cognica Virtual Machine (CVM)
A 256+ opcode bytecode interpreter with computed-goto dispatch, achieving near-native performance for interpreted query execution.
### Copy-and-Patch JIT
A stencil-based JIT compilation technique that provides 2-5x speedup for hot query paths with minimal compilation overhead.
### Zero-Copy JOINs
Composite row representation that eliminates data copying during JOIN operations, reducing memory pressure and improving cache efficiency.
### Bayesian BM25 and Probabilistic Calibration
A three-term posterior decomposition that transforms unbounded BM25 scores into calibrated probabilities, enabling principled multi-signal fusion with 68-77% calibration error reduction in unsupervised settings.
### Vector Score Calibration
Likelihood ratio calibration that transforms vector similarity scores into relevance probabilities using distributional statistics extracted from ANN indexes at zero additional cost.
### Neural Emergence from Bayesian Inference
The discovery that combining calibrated probability signals through log-odds conjunction analytically produces the structure of a feedforward neural network — reversing the conventional direction of explanation in neural network theory.
### WAND/BMW Query Evaluation
Block-max weighted AND algorithms achieving 50-88% document skip rates for top-K retrieval, with proven exact neural pruning guarantees.
---
## Reading Paths
### For Database Developers
Start with Part I for theoretical foundations, then proceed through Parts II-V for core implementation details. Appendix A is essential for CVM development.
### For Application Developers
Focus on Chapters 8 (SQL Parser), 12 (Cypher), 28 (PostgreSQL Protocol), and 30 (Multi-Protocol Layer). Appendices B and D provide API references.
### For Graph Database Developers
Chapter 3 provides the theoretical graph algebra, Chapter 11 covers storage and traversal algorithms, and Chapter 12 details the Cypher-to-SQL rewriting pipeline.
### For Operations Engineers
Chapters 31-33 cover production concerns. Appendix C provides comprehensive configuration documentation.
### For Researchers
Part I establishes the theoretical framework. Chapters 19-25 cover state-of-the-art information retrieval techniques, including the probabilistic calibration trilogy (Chapters 20, 22, 23) that connects information retrieval to neural computation.
## Version Information
- **Cognica Version**: 1.0
- **Last Updated**: March 2026
- **Total Chapters**: 34
- **Total Appendices**: 4
---
*Copyright 2023-2026 Cognica, Inc. All rights reserved.*
# Chapter 1: Introduction to Unified Data Processing
## 1.1 The Data Paradigm Fragmentation Problem
Modern applications face an increasingly complex data landscape. A typical enterprise system might need to:
- Store structured business records in a **relational database** for ACID transactions
- Index product descriptions in a **full-text search engine** for keyword queries
- Compute similarity between items using **vector embeddings** in a specialized vector database
- Navigate relationships between entities through a **graph database**
- Cache frequently accessed data in a **key-value store**
This approach, known as **polyglot persistence**, emerged from the recognition that different data models excel at different workloads. A relational database optimized for transactions differs fundamentally from a search engine optimized for text retrieval. Each system brings its own query language, consistency model, operational requirements, and failure modes.
### 1.1.1 The Operational Burden
Consider a product search feature that must:
1. Find products matching text keywords (full-text search)
2. Filter by category and price range (relational predicates)
3. Rank by semantic similarity to user preferences (vector search)
4. Traverse supplier relationships (graph queries)
In a polyglot architecture, this single user-facing feature requires orchestrating four separate database systems:
```mermaid
graph TB
subgraph "Polyglot Architecture"
APP[Application Server]
APP -->|Text Query| ES[Elasticsearch]
APP -->|SQL Query| PG[PostgreSQL]
APP -->|Vector Query| PIN[Pinecone]
APP -->|Graph Query| NEO[Neo4j]
ES -->|Doc IDs| APP
PG -->|Filtered IDs| APP
PIN -->|Similar IDs| APP
NEO -->|Related IDs| APP
APP -->|Merge & Rank| RESULT[Final Results]
end
```
The application must:
- Maintain connections to four different systems
- Translate between four query languages
- Merge results with potentially inconsistent identifiers
- Handle partial failures when any system becomes unavailable
- Manage data synchronization across all systems
Each additional system multiplies operational complexity. DevOps teams must master four different backup strategies, monitoring dashboards, scaling procedures, and failure recovery protocols.
### 1.1.2 The Consistency Challenge
Beyond operational complexity lies a more fundamental problem: **consistency**. When a product's price changes in PostgreSQL, how quickly does Elasticsearch reflect the update? What happens if the vector database's embedding becomes stale?
Polyglot architectures typically offer only **eventual consistency** across systems, with no transactional guarantees spanning multiple databases. Applications must implement complex reconciliation logic, handle temporary inconsistencies gracefully, and design around the possibility that different systems hold conflicting views of the same data.
### 1.1.3 The Impedance Mismatch
Each data paradigm brings its own conceptual model:
| Paradigm | Data Model | Query Model | Result Model |
|----------|------------|-------------|--------------|
| Relational | Tables, Rows | SQL, Joins | Result Sets |
| Document | JSON/BSON | Query DSL | Documents |
| Full-Text | Terms, Postings | Boolean/BM25 | Scored Hits |
| Vector | Embeddings | k-NN | Distances |
| Graph | Nodes, Edges | Traversal | Paths |
Translating between these models introduces **impedance mismatch** - the conceptual friction of mapping one paradigm's abstractions onto another's. A document database's nested structure doesn't naturally decompose into relational joins. A graph traversal doesn't directly map to vector similarity. Each translation loses information or introduces complexity.
## 1.2 The Case for Unification
What if a single system could natively support all these paradigms? Not through adapters or plugins, but through a unified foundation that treats relational predicates, text relevance, vector similarity, and graph traversal as variations of the same underlying algebra?
This is the vision of **unified data processing**: a single database engine where:
- One storage layer manages all data
- One query language expresses all operations
- One optimizer plans across paradigms
- One transaction model ensures consistency
- One operational model simplifies deployment
### 1.2.1 The Posting List Insight
The key insight enabling unification comes from information retrieval theory. Consider how a full-text search engine finds documents containing a term:
$$
\tau_{\text{database}} = \{d_1, d_5, d_7, d_{12}, ...\}
$$
This **posting list** - the set of document IDs containing term "database" - is simply a set. Set operations form a Boolean algebra with well-understood properties:
- **Intersection** ($\cap$): Documents containing BOTH terms
- **Union** ($\cup$): Documents containing EITHER term
- **Complement** ($\neg$): Documents NOT containing a term
Now consider a relational filter: "products where price < 100". This also produces a set of qualifying document IDs:
$$
\sigma_{\text{price}<100} = \{d_2, d_5, d_7, d_9, ...\}
$$
A vector similarity search for the top-k nearest neighbors? Another set of document IDs:
$$
\text{kNN}(\vec{q}, k) = \{d_3, d_5, d_8, ...\}
$$
A graph traversal finding all nodes within 2 hops? Yet another set:
$$
\text{traverse}(v_0, 2) = \{d_5, d_6, d_7, ...\}
$$
**The profound realization**: All these seemingly different operations produce the same thing - sets of document identifiers. They can all be represented as posting lists, combined with the same Boolean operations, and optimized with the same algebraic transformations.
### 1.2.2 Unified Architecture
This insight leads to a unified architecture where posting lists serve as the universal abstraction:
```mermaid
graph TB
subgraph "Unified Architecture"
APP[Application]
APP --> COGNICA[Cognica]
subgraph "Cognica Engine"
SQL[SQL Parser] --> PLAN[Query Planner]
PLAN --> OPT[Cross-Paradigm Optimizer]
OPT --> EXEC[Unified Executor]
EXEC --> REL[Relational Scan]
EXEC --> FTS[FTS Posting Lists]
EXEC --> VEC[Vector Index]
EXEC --> GRF[Graph Traversal]
REL --> MERGE[Posting List Merge]
FTS --> MERGE
VEC --> MERGE
GRF --> MERGE
MERGE --> RESULT[Unified Results]
end
end
```
A single query can seamlessly combine:
```sql
SELECT p.name, p.price
FROM products p
WHERE p.category = 'electronics' -- Relational
AND MATCH(p.description) AGAINST ('wireless') -- Full-text
AND vector_similarity(p.embedding, ?) > 0.8 -- Vector
AND EXISTS ( -- Graph
SELECT 1 FROM suppliers s
WHERE s.id = p.supplier_id
AND s.rating > 4.0
)
ORDER BY bm25_score(p.description) DESC
LIMIT 10;
```
The query planner recognizes each predicate's paradigm, retrieves the corresponding posting lists, and merges them using optimized set operations - all within a single transaction, with consistent results, through one query interface.
## 1.3 Cognica Architecture Overview
Cognica implements this unified vision through carefully designed components that work together to process queries across paradigms.
### 1.3.1 Design Principles
**Principle 1: Posting Lists as Universal Currency**
Every index, filter, and search operation ultimately produces posting lists. The system maintains a common representation that flows through all processing stages, enabling uniform optimization and execution.
**Principle 2: Algebraic Optimization**
Because posting lists form a Boolean algebra, the query optimizer can apply algebraic transformations regardless of the original paradigm:
- Predicate pushdown works for relational filters and text queries
- Join reordering applies to relational joins and graph traversals
- Cost-based selection chooses between index scan and sequential scan
**Principle 3: Vectorized Execution**
Modern CPUs achieve highest throughput when processing data in batches. Cognica's execution engine processes posting lists in columnar batches, exploiting SIMD instructions and cache locality.
**Principle 4: Tiered Compilation**
Frequently executed query patterns compile from bytecode interpretation through JIT compilation to native code, achieving performance competitive with hand-written C++ while maintaining the flexibility of a general-purpose query engine.
### 1.3.2 System Components
```mermaid
graph TD
subgraph proto["Protocol Layer"]
direction LR
PG["PostgreSQL Wire"]
HTTP["HTTP/REST"]
FLIGHT["Flight SQL"]
end
SERVICE["Service Layer"]
subgraph qp["Query Processing"]
PARSER["SQL Parser"] --> ANALYZER["Semantic Analyzer"]
ANALYZER --> PLANNER["Query Planner"]
PLANNER --> OPTIMIZER["Cost-Based Optimizer"]
end
subgraph exec["Execution Engine"]
direction LR
INTERP["Bytecode Interpreter"]
JIT["JIT Compiler"]
end
subgraph store["Storage Layer"]
direction LR
ROCKS["RocksDB LSM-Tree"]
FTS_IDX["Inverted Index"]
HNSW["HNSW Vector Index"]
end
RAFT["Raft Consensus"]
proto --> SERVICE --> PARSER
OPTIMIZER --> exec --> store --> RAFT
```
**Protocol Layer**: Cognica speaks multiple protocols natively. The PostgreSQL wire protocol enables compatibility with existing tools like `psql`, JDBC drivers, and BI platforms. Arrow Flight SQL enables high-throughput analytical queries.
**Query Processing Pipeline**: SQL queries parse through libpg_query (PostgreSQL's actual parser), ensuring compatibility with PostgreSQL syntax. The semantic analyzer resolves names, checks types, and expands views. The query planner generates logical plans, and the cost-based optimizer selects physical implementations.
**Execution Engine**: The Cognica Virtual Machine (CVM) executes queries through a register-based bytecode interpreter. Hot paths automatically compile to native code via copy-and-patch JIT compilation. Vectorized operators process data in columnar batches for maximum throughput.
**Storage Layer**: RocksDB provides the foundational LSM-tree storage with ACID transactions and MVCC. Specialized indexes layer on top: inverted indexes for full-text search, HNSW graphs for vector similarity, secondary indexes for relational queries.
**Distribution Layer**: The Raft consensus protocol provides distributed consistency for multi-node deployments. All storage operations replicate through the consensus layer, ensuring durability and enabling horizontal scaling.
### 1.3.3 Query Execution Flow
To illustrate how these components work together, consider a hybrid query that combines text search with relational filtering:
```sql
SELECT title, author, bm25_score(content) as score
FROM articles
WHERE MATCH(content) AGAINST ('database internals')
AND published_date > '2024-01-01'
ORDER BY score DESC
LIMIT 10;
```
**Step 1: Parsing**
The SQL parser produces an Abstract Syntax Tree (AST) representing the query structure. The `MATCH...AGAINST` clause parses as a full-text search predicate; the date comparison as a relational predicate.
**Step 2: Semantic Analysis**
The analyzer resolves `articles` to a collection, validates that `content` has a full-text index, confirms `published_date` is a timestamp type, and verifies `bm25_score()` is a valid scoring function.
**Step 3: Logical Planning**
The planner produces a logical plan:
```
Limit(10)
Sort(score DESC)
Project(title, author, bm25_score(content) as score)
Filter(published_date > '2024-01-01')
FTSSearch(content, 'database internals')
Scan(articles)
```
**Step 4: Optimization**
The optimizer recognizes that the FTS search and date filter can execute independently, then intersect their posting lists:
```
Limit(10)
Sort(score DESC)
Project(title, author, score)
PostingListIntersect
FTSSearch(content, 'database internals') -> posting list + scores
IndexScan(published_date > '2024-01-01') -> posting list
```
**Step 5: Physical Planning**
The physical planner selects concrete implementations:
- FTS search uses WAND algorithm for efficient top-k retrieval
- Date filter uses secondary index range scan
- Intersection uses sorted merge with score propagation
- Sort uses in-memory heap for small result sets
**Step 6: Code Generation**
The CVM compiler generates bytecode implementing the physical plan. Register allocation assigns document IDs, scores, and intermediate results to virtual registers.
**Step 7: Execution**
The bytecode interpreter executes the plan, fetching posting lists from indexes, computing intersections, scoring documents with BM25, and returning the top 10 results.
**Step 8: Result Delivery**
Results serialize through the PostgreSQL wire protocol back to the client, appearing exactly as they would from a PostgreSQL database.
## 1.4 Historical Context
Cognica's unified approach builds on decades of database and information retrieval research.
### 1.4.1 Evolution of Database Systems
**1970s - Relational Model**: Edgar Codd's relational model established the mathematical foundation for database systems. Relational algebra provided a formal framework for query optimization, proving that different query expressions could produce identical results.
**1980s - Query Optimization**: System R and INGRES pioneered cost-based query optimization, demonstrating that declarative queries could compile to efficient execution plans through algebraic transformation.
**1990s - Object-Relational**: As applications grew complex, object-relational databases attempted to bridge the gap between relational storage and object-oriented programming. This era introduced extensible type systems and user-defined functions.
**2000s - NoSQL Movement**: Web-scale applications drove the NoSQL revolution. Document stores (MongoDB), key-value stores (Redis), and graph databases (Neo4j) optimized for specific access patterns at the cost of query flexibility.
**2010s - NewSQL and Convergence**: Systems like CockroachDB and TiDB proved that distributed ACID transactions were achievable. Meanwhile, traditional databases began adding JSON support, full-text search, and other features.
**2020s - Unified Systems**: The current generation aims to eliminate the polyglot complexity entirely. Rather than adding features piecemeal, systems like Cognica rethink the foundational abstractions to enable native multi-paradigm support.
### 1.4.2 Information Retrieval Foundations
Full-text search engines developed independently from databases, with their own theoretical foundations:
**Boolean Retrieval**: The earliest IR systems matched Boolean combinations of terms. Documents either matched a query or didn't - no ranking, just set operations on posting lists.
**Vector Space Model**: Salton's vector space model represented documents and queries as vectors in term space, enabling similarity-based ranking through cosine similarity.
**Probabilistic Models**: Robertson's probability ranking principle established that documents should rank by their probability of relevance. This led to BM25, still the dominant text ranking function.
**Neural Retrieval**: Modern neural models encode documents and queries as dense vectors, enabling semantic similarity beyond lexical matching. This drives the current interest in vector databases.
### 1.4.3 Prior Unification Attempts
Previous attempts at unification typically followed one of two paths:
**Extension Approach**: Traditional databases added features incrementally. PostgreSQL added `tsvector` for full-text search, `jsonb` for documents, and `pgvector` for embeddings. While functional, these extensions often feel bolted-on, with limited cross-feature optimization.
**Federation Approach**: Systems like Presto and Trino federate queries across multiple backends. While providing a unified interface, they cannot optimize across data sources or provide cross-source transactions.
Cognica takes a different path: **native unification**. Rather than extending a relational database or federating separate systems, it builds from a foundation where posting lists are first-class citizens, enabling deep optimization across paradigms.
## 1.5 What This Book Covers
This book provides a comprehensive treatment of Cognica's design and implementation, suitable for:
- **Graduate students** studying database systems, information retrieval, or distributed systems
- **Database researchers** exploring unified query processing
- **Senior engineers** building or operating data-intensive applications
- **Contributors** seeking to understand and extend Cognica
### 1.5.1 Part I: Foundations (Chapters 1-4)
We establish the mathematical framework for unified query processing:
- **Chapter 2** formalizes posting lists as a Boolean algebra and defines the type system spanning documents, vectors, terms, and graphs
- **Chapter 3** extends the algebra to incorporate graph structures while preserving algebraic properties
- **Chapter 4** develops query optimization theory, including cost models, selectivity estimation, and transformation rules
### 1.5.2 Part II: Storage Engine (Chapters 5-7)
We examine how data persists and indexes organize:
- **Chapter 5** details the LSM-tree storage architecture based on RocksDB
- **Chapter 6** explains document storage, schema management, and secondary indexes
- **Chapter 7** deep-dives into inverted index architecture, including the innovative clustered term index
### 1.5.3 Part III: Query Processing (Chapters 8-10)
We trace queries from SQL text to executable plans:
- **Chapter 8** covers SQL parsing and semantic analysis
- **Chapter 9** explains logical planning and optimization
- **Chapter 10** details physical planning and execution strategy selection
### 1.5.4 Part IV: Execution Engine (Chapters 11-15)
We explore the Cognica Virtual Machine in depth:
- **Chapter 11** presents CVM architecture: instruction formats, registers, and dispatch
- **Chapter 12** details the compilation pipeline from SQL to bytecode
- **Chapter 13** explains vectorized execution for batch processing
- **Chapter 14** covers copy-and-patch JIT compilation
- **Chapter 15** describes zero-copy JOIN implementation
### 1.5.5 Part V: Similarity Search and Ranking (Chapters 16-20)
We examine text and vector search capabilities:
- **Chapter 16** details the text analysis pipeline
- **Chapter 17** explains BM25 scoring and its Bayesian extension for calibrated relevance
- **Chapter 18** covers vector search with HNSW indexes
- **Chapter 19** describes hybrid search architecture combining text and vectors
- **Chapter 20** presents query evaluation strategies including WAND and Block-Max WAND
### 1.5.6 Part VI: Distributed Systems (Chapters 21-22)
We cover distributed operation:
- **Chapter 21** explains the Raft consensus protocol implementation
- **Chapter 22** details transaction processing and MVCC
### 1.5.7 Part VII: System Integration (Chapters 23-25)
We examine external interfaces:
- **Chapter 23** covers PostgreSQL wire protocol compatibility
- **Chapter 24** details external table integration with Parquet, Arrow, and cloud storage
- **Chapter 25** describes the multi-protocol service layer
### 1.5.8 Part VIII: Advanced Topics (Chapters 26-28)
We conclude with advanced subjects:
- **Chapter 26** details memory management strategies
- **Chapter 27** covers observability and debugging
- **Chapter 28** discusses performance engineering
### 1.5.9 Appendices
Reference materials include:
- **Appendix A**: Complete CVM opcode reference
- **Appendix B**: SQL compatibility matrix
- **Appendix C**: Configuration reference
- **Appendix D**: API specifications
## 1.6 Summary
This chapter introduced the challenge of data paradigm fragmentation and the vision of unified data processing. Key points:
1. **Polyglot persistence** creates operational complexity, consistency challenges, and impedance mismatch between data paradigms
2. **Posting lists** provide a universal abstraction - all query predicates ultimately produce sets of document identifiers that combine through Boolean operations
3. **Cognica** implements unified processing through carefully designed components: a multi-protocol service layer, PostgreSQL-compatible SQL processing, a bytecode virtual machine with JIT compilation, and specialized indexes for text and vector search
4. **Historical context** shows that Cognica builds on decades of database and information retrieval research, taking a different path than extension or federation approaches
The following chapter formalizes these intuitions mathematically, establishing the algebraic foundations that enable cross-paradigm optimization.
# Chapter 2: Mathematical Foundations of Query Algebras
This chapter establishes the rigorous mathematical framework underlying unified query processing. We formalize posting lists as a Boolean algebra, define the type system spanning multiple data paradigms, and develop the operator calculus that enables cross-paradigm optimization.
## 2.1 Set Theory and Boolean Algebra Review
Before developing the unified algebra, we review the mathematical structures upon which it rests.
### 2.1.1 Sets and Relations
A **set** is an unordered collection of distinct elements. We write $x \in S$ to indicate element $x$ belongs to set $S$, and $|S|$ to denote the cardinality (size) of $S$.
The **power set** $2^S$ of a set $S$ is the set of all subsets of $S$:
$$
2^S = \{T \mid T \subseteq S\}
$$
For a finite set with $n$ elements, $|2^S| = 2^n$.
A **relation** $R \subseteq A \times B$ is a set of ordered pairs from the Cartesian product of sets $A$ and $B$. A **function** $f: A \rightarrow B$ is a relation where each element of $A$ maps to exactly one element of $B$.
### 2.1.2 Boolean Algebra Axioms
A **Boolean algebra** is a set $B$ with two binary operations $\land$ (meet/AND) and $\lor$ (join/OR), a unary operation $\neg$ (complement/NOT), and two distinguished elements $0$ (bottom) and $1$ (top), satisfying:
**Commutativity**:
$$
a \land b = b \land a
$$
$$
a \lor b = b \lor a
$$
**Associativity**:
$$
(a \land b) \land c = a \land (b \land c)
$$
$$
(a \lor b) \lor c = a \lor (b \lor c)
$$
**Absorption**:
$$
a \land (a \lor b) = a
$$
$$
a \lor (a \land b) = a
$$
**Identity**:
$$
a \land 1 = a
$$
$$
a \lor 0 = a
$$
**Distributivity**:
$$
a \land (b \lor c) = (a \land b) \lor (a \land c)
$$
$$
a \lor (b \land c) = (a \lor b) \land (a \lor c)
$$
**Complementation**:
$$
a \land \neg a = 0
$$
$$
a \lor \neg a = 1
$$
The power set $2^S$ with intersection ($\cap$), union ($\cup$), complement, empty set ($\emptyset$), and universal set ($S$) forms a Boolean algebra. This is the algebra of posting lists.
### 2.1.3 Lattice Theory Fundamentals
A **partially ordered set** (poset) $(L, \leq)$ is a set $L$ with a reflexive, antisymmetric, transitive relation $\leq$.
A **lattice** is a poset where every pair of elements has a least upper bound (join, $\lor$) and greatest lower bound (meet, $\land$).
A **complete lattice** has joins and meets for all subsets, not just pairs. The power set $2^S$ ordered by inclusion forms a complete lattice:
- Meet of a family: $\bigcap_{i \in I} S_i$
- Join of a family: $\bigcup_{i \in I} S_i$
- Bottom element: $\emptyset$
- Top element: $S$
This structure is fundamental to query optimization, where query plans form a lattice ordered by cost, and transformations navigate toward the optimal plan.
## 2.2 Posting Lists as Universal Abstraction
We now formalize how posting lists serve as the universal abstraction connecting disparate query paradigms.
### 2.2.1 Document Universe
Let $\mathcal{D}$ denote the **document universe** - the set of all documents in the database. Each document $d \in \mathcal{D}$ has a unique identifier $\text{id}(d) \in \mathbb{N}$.
In practice, $\mathcal{D}$ might contain millions or billions of documents. We assume documents are immutable for query purposes; updates create new document versions.
### 2.2.2 Posting List Definition
A **posting list** is a function mapping a term (or more generally, a predicate) to the set of documents satisfying that predicate:
$$
P: \mathcal{T} \rightarrow 2^{\mathcal{D}}
$$
where $\mathcal{T}$ is the term space. For a term $t$, the posting list $P(t)$ contains exactly those documents containing term $t$:
$$
P(t) = \{d \in \mathcal{D} \mid t \in \text{terms}(d)\}
$$
**Example**: For documents:
- $d_1$: "database systems"
- $d_2$: "distributed database"
- $d_3$: "operating systems"
The posting lists are:
- $P(\text{database}) = \{d_1, d_2\}$
- $P(\text{systems}) = \{d_1, d_3\}$
- $P(\text{distributed}) = \{d_2\}$
- $P(\text{operating}) = \{d_3\}$
### 2.2.3 Boolean Algebra of Posting Lists
Posting lists form a Boolean algebra under set operations:
**Conjunction (AND)**: Documents matching both predicates
$$
P(t_1) \cap P(t_2) = \{d \in \mathcal{D} \mid t_1 \in \text{terms}(d) \land t_2 \in \text{terms}(d)\}
$$
**Disjunction (OR)**: Documents matching either predicate
$$
P(t_1) \cup P(t_2) = \{d \in \mathcal{D} \mid t_1 \in \text{terms}(d) \lor t_2 \in \text{terms}(d)\}
$$
**Negation (NOT)**: Documents not matching the predicate
$$
\overline{P(t)} = \mathcal{D} \setminus P(t) = \{d \in \mathcal{D} \mid t \notin \text{terms}(d)\}
$$
These operations satisfy the Boolean algebra axioms:
**Commutativity**:
$$
P_1 \cap P_2 = P_2 \cap P_1
$$
**Associativity**:
$$
(P_1 \cap P_2) \cap P_3 = P_1 \cap (P_2 \cap P_3)
$$
**Distributivity**:
$$
P_1 \cap (P_2 \cup P_3) = (P_1 \cap P_2) \cup (P_1 \cap P_3)
$$
**De Morgan's Laws**:
$$
\overline{P_1 \cap P_2} = \overline{P_1} \cup \overline{P_2}
$$
$$
\overline{P_1 \cup P_2} = \overline{P_1} \cap \overline{P_2}
$$
### 2.2.4 Scored Posting Lists
For ranking queries, we extend posting lists to carry scores:
$$
P_s: \mathcal{T} \rightarrow 2^{\mathcal{D} \times \mathbb{R}}
$$
A scored posting list maps terms to sets of (document, score) pairs:
$$
P_s(t) = \{(d, s) \mid d \in P(t), s = \text{score}(d, t)\}
$$
Score combination during Boolean operations requires specification:
**AND with scores**: Various strategies exist
- Minimum: $\min(s_1, s_2)$
- Product: $s_1 \cdot s_2$ (probabilistic interpretation)
- Sum: $s_1 + s_2$ (additive combination)
**OR with scores**:
- Maximum: $\max(s_1, s_2)$
- Probabilistic OR: $1 - (1-s_1)(1-s_2)$
Chapter 19 develops these score combination strategies in depth.
## 2.3 Type System Formalization
A unified query system must bridge multiple data paradigms. We formalize the type system that enables this bridging.
### 2.3.1 Document Space
The **document space** $\mathcal{D}$ is the primary universe. Each document $d \in \mathcal{D}$ is a semi-structured record with:
- A unique identifier: $\text{id}: \mathcal{D} \rightarrow \mathbb{N}$
- A schema: $\text{schema}: \mathcal{D} \rightarrow \Sigma$
- Field values: $\text{field}: \mathcal{D} \times \mathcal{F} \rightarrow \mathcal{V}$
where $\mathcal{F}$ is the field name space and $\mathcal{V}$ is the value space.
The value space is a discriminated union:
$$
\mathcal{V} = \text{Null} \mid \text{Bool} \mid \text{Int64} \mid \text{Float64} \mid \text{String} \mid \text{Binary} \mid \text{Array}(\mathcal{V}) \mid \text{Object}(\mathcal{F} \rightarrow \mathcal{V})
$$
### 2.3.2 Term Space
The **term space** $\mathcal{T}$ contains atomic units of text:
$$
\mathcal{T} = \Sigma^* / \sim
$$
where $\Sigma^*$ is the set of all strings and $\sim$ is an equivalence relation defined by text normalization (lowercasing, stemming, etc.).
The **terms function** extracts terms from text:
$$
\text{terms}: \text{String} \rightarrow 2^{\mathcal{T}}
$$
This function encapsulates the text analysis pipeline (Chapter 16).
### 2.3.3 Vector Space
The **vector space** $\mathcal{V}_n$ is $n$-dimensional real space:
$$
\mathcal{V}_n = \mathbb{R}^n
$$
Documents may have vector representations through embedding functions:
$$
\text{embed}: \mathcal{D} \times \mathcal{F} \rightarrow \mathcal{V}_n
$$
Common metrics on vector space include:
**Euclidean (L2) distance**:
$$
d_{L2}(\vec{u}, \vec{v}) = \sqrt{\sum_{i=1}^{n} (u_i - v_i)^2}
$$
**Cosine distance**:
$$
d_{\cos}(\vec{u}, \vec{v}) = 1 - \frac{\vec{u} \cdot \vec{v}}{|\vec{u}| \cdot |\vec{v}|}
$$
**Inner product** (for normalized vectors):
$$
\text{ip}(\vec{u}, \vec{v}) = \vec{u} \cdot \vec{v} = \sum_{i=1}^{n} u_i \cdot v_i
$$
### 2.3.4 Field Space
The **field space** $\mathcal{F}$ names document attributes:
$$
\mathcal{F} = \text{String}
$$
Fields support hierarchical naming for nested documents:
$$
\text{address.city} \in \mathcal{F}
$$
Field paths form a tree structure rooted at the document.
### 2.3.5 Bijective Mappings
The power of the unified type system comes from explicit mappings between spaces:
**Document to Posting List**: Every document belongs to posting lists for its terms
$$
\text{PostingFor}: \mathcal{D} \rightarrow 2^{2^{\mathcal{D}}}
$$
$$
\text{PostingFor}(d) = \{P(t) \mid t \in \text{terms}(d)\}
$$
**Term to Posting List**: Direct mapping
$$
P: \mathcal{T} \rightarrow 2^{\mathcal{D}}
$$
**Vector to Posting List**: k-NN search produces a posting list
$$
\text{kNN}: \mathcal{V}_n \times \mathbb{N} \rightarrow 2^{\mathcal{D}}
$$
$$
\text{kNN}(\vec{q}, k) = \text{argmin}_{S \subseteq \mathcal{D}, |S|=k} \sum_{d \in S} d(\vec{q}, \text{embed}(d))
$$
**Relational Predicate to Posting List**: Filter produces a posting list
$$
\sigma_\phi: 2^{\mathcal{D}} \rightarrow 2^{\mathcal{D}}
$$
$$
\sigma_\phi(\mathcal{D}) = \{d \in \mathcal{D} \mid \phi(d) = \text{true}\}
$$
These mappings enable the query planner to treat operations from different paradigms uniformly.
## 2.4 Operator Calculus
With the type system established, we define the operators that transform posting lists.
### 2.4.1 Primitive Operators
**Term Retrieval** ($\tau$): Retrieve the posting list for a term
$$
\tau: \mathcal{T} \rightarrow 2^{\mathcal{D}}
$$
$$
\tau_t = \{d \in \mathcal{D} \mid t \in \text{terms}(d)\}
$$
**Filter** ($\sigma$): Apply a predicate to filter a posting list
$$
\sigma: (D \rightarrow \text{Bool}) \times 2^{\mathcal{D}} \rightarrow 2^{\mathcal{D}}
$$
$$
\sigma_\phi(P) = \{d \in P \mid \phi(d)\}
$$
**Projection** ($\pi$): Select specific fields from documents
$$
\pi: 2^{\mathcal{F}} \times 2^{\mathcal{D}} \rightarrow 2^{\mathcal{D}'}
$$
where $\mathcal{D}'$ is the space of projected documents.
**Vector Search** ($\nu$): Find nearest neighbors
$$
\nu: \mathcal{V}_n \times \mathbb{N} \times \mathbb{R} \rightarrow 2^{\mathcal{D}}
$$
$$
\nu_{\vec{q},k,\epsilon} = \{d \in \mathcal{D} \mid d(\vec{q}, \text{embed}(d)) < \epsilon\} \cap \text{top-}k
$$
**Score** ($\rho$): Attach scores to documents
$$
\rho: (\mathcal{D} \rightarrow \mathbb{R}) \times 2^{\mathcal{D}} \rightarrow 2^{\mathcal{D} \times \mathbb{R}}
$$
$$
\rho_f(P) = \{(d, f(d)) \mid d \in P\}
$$
### 2.4.2 Composition as Monoid
Operators compose to form pipelines. The set of operators forms a **monoid** under composition:
$$
(\text{Op}, \circ, \text{id})
$$
where:
- $\text{Op}$ is the set of operators on posting lists
- $\circ$ is function composition
- $\text{id}$ is the identity operator
**Associativity**: $(f \circ g) \circ h = f \circ (g \circ h)$
**Identity**: $f \circ \text{id} = \text{id} \circ f = f$
This monoid structure enables pipeline optimization through algebraic manipulation.
### 2.4.3 Operator Properties
**Idempotence**: Applying the same filter twice is equivalent to applying it once
$$
\sigma_\phi \circ \sigma_\phi = \sigma_\phi
$$
**Filter Commutativity**: Independent filters commute
$$
\sigma_{\phi_1} \circ \sigma_{\phi_2} = \sigma_{\phi_2} \circ \sigma_{\phi_1}
$$
**Filter Conjunction**: Sequential filters equivalent to conjunctive filter
$$
\sigma_{\phi_1} \circ \sigma_{\phi_2} = \sigma_{\phi_1 \land \phi_2}
$$
**Projection Pushdown**: Projection through filter (when fields independent)
$$
\pi_A \circ \sigma_\phi = \sigma_\phi \circ \pi_A \quad \text{if } \text{fields}(\phi) \subseteq A
$$
**Selection Pushdown**: Filter through intersection
$$
\sigma_\phi(P_1 \cap P_2) = \sigma_\phi(P_1) \cap \sigma_\phi(P_2)
$$
### 2.4.4 Equivalence Classes
Two operator expressions are **equivalent** if they produce identical results for all inputs:
$$
e_1 \equiv e_2 \iff \forall P \in 2^{\mathcal{D}}: e_1(P) = e_2(P)
$$
Equivalence induces equivalence classes on query plans. The optimizer explores these classes to find minimum-cost representatives.
**Theorem (Completeness)**: The transformation rules in Section 2.4.3 are complete for the class of select-project-join queries over posting lists.
This theorem ensures the optimizer can reach any equivalent plan through rule application.
## 2.5 Extending to Scored Operations
Relevance ranking requires extending the algebra to handle scores.
### 2.5.1 Scored Posting Lists
A **scored posting list** is a set of (document, score) pairs:
$$
P_s \subseteq \mathcal{D} \times \mathbb{R}^+
$$
We define projection functions:
$$
\text{docs}(P_s) = \{d \mid (d, s) \in P_s\}
$$
$$
\text{score}(P_s, d) = s \text{ where } (d, s) \in P_s
$$
### 2.5.2 Score Combination Semantics
When combining scored posting lists, we must specify how scores combine:
**Conjunction (AND)**: For $(d, s_1) \in P_1$ and $(d, s_2) \in P_2$
*Probabilistic*:
$$
\text{score}_{\land}(d) = s_1 \cdot s_2
$$
*Additive*:
$$
\text{score}_{\land}(d) = s_1 + s_2
$$
*Minimum*:
$$
\text{score}_{\land}(d) = \min(s_1, s_2)
$$
**Disjunction (OR)**: For documents in $P_1 \cup P_2$
*Probabilistic* (inclusion-exclusion):
$$
\text{score}_{\lor}(d) = 1 - (1 - s_1)(1 - s_2) = s_1 + s_2 - s_1 \cdot s_2
$$
*Maximum*:
$$
\text{score}_{\lor}(d) = \max(s_1, s_2)
$$
*Sum*:
$$
\text{score}_{\lor}(d) = s_1 + s_2
$$
### 2.5.3 Score Normalization
For probabilistic combination, scores must be in $[0, 1]$. Raw relevance scores (e.g., BM25) are unbounded. Chapter 17 develops **Bayesian BM25** which provides calibrated probability scores suitable for probabilistic combination.
The normalization function:
$$
\text{normalize}: \mathbb{R}^+ \rightarrow [0, 1]
$$
must preserve relative ordering:
$$
s_1 > s_2 \implies \text{normalize}(s_1) > \text{normalize}(s_2)
$$
## 2.6 Categorical Perspective
For readers familiar with category theory, we briefly sketch the categorical interpretation of the unified algebra.
### 2.6.1 Category of Posting Lists
Define category $\mathbf{Post}$:
- **Objects**: Posting lists $P \in 2^{\mathcal{D}}$
- **Morphisms**: Set functions preserving document identity
This category is equivalent to the category of finite sets and functions.
### 2.6.2 Functors Between Paradigms
Each data paradigm defines a functor to $\mathbf{Post}$:
**Text Functor** $F_T: \mathbf{Term} \rightarrow \mathbf{Post}$
$$
F_T(t) = \tau_t
$$
**Vector Functor** $F_V: \mathbf{Vec} \rightarrow \mathbf{Post}$
$$
F_V(\vec{q}, k) = \text{kNN}(\vec{q}, k)
$$
**Relational Functor** $F_R: \mathbf{Rel} \rightarrow \mathbf{Post}$
$$
F_R(\phi) = \sigma_\phi(\mathcal{D})
$$
### 2.6.3 Natural Transformations
Optimization rules are natural transformations between functors:
$$
\eta: F \Rightarrow G
$$
For example, the transformation from nested-loop join to hash join is a natural transformation in the category of query plans.
### 2.6.4 Adjunctions
The free/forgetful adjunction captures the relationship between structured queries and their posting list representations:
$$
\text{Free} \dashv \text{Forget}: \mathbf{Query} \rightleftharpoons \mathbf{Post}
$$
This adjunction formalizes how queries generate posting lists and how posting list operations induce query transformations.
## 2.7 Implementation Considerations
The mathematical framework guides efficient implementation.
### 2.7.1 Posting List Representation
Posting lists are stored as sorted arrays of document IDs:
```
PostingList = [doc_id_1, doc_id_2, ..., doc_id_n] // sorted
```
Sorted order enables:
- Binary search for membership: $O(\log n)$
- Merge intersection: $O(n + m)$
- Merge union: $O(n + m)$
### 2.7.2 Skip Pointers
For large posting lists, skip pointers accelerate intersection:
```
PostingList with Skips:
[doc_id_1, ..., doc_id_k] -> skip to doc_id_{k*s}
[doc_id_{k*s+1}, ..., doc_id_{2*k*s}] -> skip to doc_id_{2*k*s}
...
```
Skip distance $s = \sqrt{n}$ minimizes worst-case intersection time.
### 2.7.3 Compression
Document IDs compress through delta encoding:
```
Original: [100, 105, 110, 200, 250]
Deltas: [100, 5, 5, 90, 50]
```
Variable-byte encoding further reduces space:
- Small deltas: 1 byte
- Large deltas: 2-4 bytes
### 2.7.4 Block-Based Processing
Modern CPUs process data fastest in cache-line-sized blocks. Posting lists partition into blocks:
```
Block 1: [doc_1, ..., doc_128] max_score=0.95
Block 2: [doc_129, ..., doc_256] max_score=0.87
...
```
Block-level metadata enables pruning in top-k queries (Chapter 20).
## 2.8 Summary
This chapter established the mathematical foundations for unified query processing:
1. **Boolean algebra** of sets provides the framework for posting list operations, with well-defined intersection, union, and complement satisfying algebraic axioms
2. **Type system** spans documents, terms, vectors, and fields with explicit mappings between spaces, enabling unified representation of multi-paradigm data
3. **Operator calculus** defines primitive operators (term retrieval, filter, projection, vector search, scoring) that compose into query pipelines with algebraic properties enabling optimization
4. **Scored extension** handles relevance ranking through scored posting lists with various combination semantics (probabilistic, additive, min/max)
5. **Categorical perspective** reveals the deep structure: functors map paradigms to posting lists, natural transformations capture optimization rules, and adjunctions formalize the query-posting list relationship
The following chapter extends this framework to incorporate graph structures, proving that graph posting lists preserve the algebraic properties while enabling traversal and pattern matching operations.
# Chapter 3: Extending the Algebra to Graph Structures
This chapter extends the unified mathematical framework to incorporate graph operations while preserving the algebraic properties established in Chapter 2. We demonstrate that graph posting lists form isomorphic structures to document posting lists, enabling graph traversal, pattern matching, and cross-paradigm queries within the same algebraic framework.
## 3.1 Motivation for Graph Integration
Graph databases have emerged as essential tools for modeling relationships: social networks, knowledge graphs, fraud detection networks, recommendation systems, and supply chain dependencies. Yet traditional graph databases operate as isolated systems, requiring separate data synchronization and offering no transactional consistency with relational or search workloads.
### 3.1.1 The Relationship Modeling Challenge
Consider a product recommendation system that must:
1. Find products matching user search terms (full-text search)
2. Filter by inventory and pricing constraints (relational predicates)
3. Rank by embedding similarity to user preferences (vector search)
4. Traverse purchase history and social connections (graph traversal)
5. Return results with explanation paths (graph + relational join)
In a polyglot architecture, the graph component requires:
- Duplicating entity data from the relational store
- Maintaining referential integrity across systems
- Orchestrating cross-system queries with no shared transaction
- Merging results with incompatible data models
### 3.1.2 Graph-Posting List Unification
The key insight enabling graph integration is that graph operations also produce sets of identifiers:
**Traversal**: Starting from vertex $v$, find all vertices reachable in $k$ hops
$$
\text{traverse}(v, k) = \{u \in V \mid \text{dist}(v, u) \leq k\}
$$
**Pattern Match**: Find all vertices matching a structural pattern
$$
\text{match}(G, P) = \{v \in V \mid v \text{ participates in pattern } P\}
$$
**Path Query**: Find vertices connected by paths matching a regular expression
$$
\text{RPQ}(v, r) = \{u \in V \mid \exists \text{ path } v \xrightarrow{r} u\}
$$
Each operation returns a set - a posting list of vertex identifiers. These posting lists combine with document posting lists through the same Boolean operations, enabling unified optimization across paradigms.
## 3.2 Graph Type System
We formalize the graph type system as an extension of the document type system from Chapter 2.
### 3.2.1 Property Graph Model
A **property graph** is a tuple $G = (V, E, \rho, \lambda, \sigma)$ where:
- $V$ is the set of **vertices** (nodes)
- $E \subseteq V \times V$ is the set of directed **edges**
- $\rho: V \cup E \rightarrow 2^{\mathcal{F} \times \mathcal{V}}$ assigns **properties** (key-value pairs)
- $\lambda: V \cup E \rightarrow 2^L$ assigns **labels** from label set $L$
- $\sigma: E \rightarrow \mathcal{T}$ assigns a **type** to each edge
Vertices and edges both carry properties and labels, making them semi-structured entities similar to documents.
### 3.2.2 Vertex Space
The **vertex space** $\mathcal{V}_G$ contains all vertices:
$$
\mathcal{V}_G = \{v \mid v \in V \text{ for some property graph } G\}
$$
Each vertex has:
- A unique identifier: $\text{id}: \mathcal{V}_G \rightarrow \mathbb{N}$
- Properties: $\text{props}: \mathcal{V}_G \rightarrow 2^{\mathcal{F} \times \mathcal{V}}$
- Labels: $\text{labels}: \mathcal{V}_G \rightarrow 2^L$
### 3.2.3 Edge Space
The **edge space** $\mathcal{E}_G$ contains all edges:
$$
\mathcal{E}_G = \{(u, v, t) \mid (u, v) \in E, t = \sigma(u, v)\}
$$
Each edge has:
- A unique identifier: $\text{id}: \mathcal{E}_G \rightarrow \mathbb{N}$
- Source vertex: $\text{src}: \mathcal{E}_G \rightarrow \mathcal{V}_G$
- Target vertex: $\text{tgt}: \mathcal{E}_G \rightarrow \mathcal{V}_G$
- Edge type: $\text{type}: \mathcal{E}_G \rightarrow \mathcal{T}$
- Properties: $\text{props}: \mathcal{E}_G \rightarrow 2^{\mathcal{F} \times \mathcal{V}}$
### 3.2.4 Graph Posting Lists
A **graph posting list** maps a predicate to a set of vertices or edges:
**Vertex posting list**:
$$
P_V: \text{Pred}_V \rightarrow 2^{\mathcal{V}_G}
$$
**Edge posting list**:
$$
P_E: \text{Pred}_E \rightarrow 2^{\mathcal{E}_G}
$$
Examples of graph predicates:
**Label predicate**: Vertices with label $\ell$
$$
P_V(\text{hasLabel}(\ell)) = \{v \in \mathcal{V}_G \mid \ell \in \text{labels}(v)\}
$$
**Property predicate**: Vertices where property $p$ satisfies condition $\phi$
$$
P_V(\text{prop}(p, \phi)) = \{v \in \mathcal{V}_G \mid \phi(\text{props}(v)(p))\}
$$
**Edge type predicate**: Edges of type $t$
$$
P_E(\text{hasType}(t)) = \{e \in \mathcal{E}_G \mid \text{type}(e) = t\}
$$
**Adjacency predicate**: Vertices adjacent to vertex $v$ via edge type $t$
$$
P_V(\text{adj}(v, t)) = \{u \in \mathcal{V}_G \mid (v, u, t) \in \mathcal{E}_G \lor (u, v, t) \in \mathcal{E}_G\}
$$
### 3.2.5 Document-Vertex Correspondence
In Cognica, vertices correspond to documents:
$$
\mathcal{V}_G \cong \mathcal{D}
$$
This isomorphism maps:
- Vertex ID to document ID
- Vertex properties to document fields
- Vertex labels to document type tags
The correspondence enables a document to participate in both relational queries (as a row) and graph queries (as a vertex) without data duplication.
## 3.3 Graph-Posting List Isomorphism
We now prove that graph posting lists satisfy the same algebraic properties as document posting lists.
### 3.3.1 Boolean Algebra Structure
**Theorem 3.1 (Graph Posting List Boolean Algebra)**: The set of graph posting lists $2^{\mathcal{V}_G}$ with operations $\cap$, $\cup$, $\overline{\cdot}$, $\emptyset$, and $\mathcal{V}_G$ forms a Boolean algebra isomorphic to the document posting list algebra $2^{\mathcal{D}}$.
**Proof**: We verify each axiom:
*Closure*: For any graph posting lists $P_1, P_2 \in 2^{\mathcal{V}_G}$:
- $P_1 \cap P_2 \in 2^{\mathcal{V}_G}$ (intersection of vertex sets is a vertex set)
- $P_1 \cup P_2 \in 2^{\mathcal{V}_G}$ (union of vertex sets is a vertex set)
- $\overline{P_1} = \mathcal{V}_G \setminus P_1 \in 2^{\mathcal{V}_G}$ (complement is a vertex set)
*Commutativity*: Inherited from set operations.
*Associativity*: Inherited from set operations.
*Distributivity*: Inherited from set operations.
*Identity*: $P \cap \mathcal{V}_G = P$ and $P \cup \emptyset = P$.
*Complementation*: $P \cap \overline{P} = \emptyset$ and $P \cup \overline{P} = \mathcal{V}_G$.
The isomorphism $\phi: 2^{\mathcal{D}} \rightarrow 2^{\mathcal{V}_G}$ is induced by the document-vertex correspondence:
$$
\phi(P_D) = \{v \in \mathcal{V}_G \mid \text{doc}(v) \in P_D\}
$$
This $\phi$ preserves all Boolean operations:
$$
\phi(P_1 \cap P_2) = \phi(P_1) \cap \phi(P_2)
$$
$$
\phi(P_1 \cup P_2) = \phi(P_1) \cup \phi(P_2)
$$
$$
\phi(\overline{P}) = \overline{\phi(P)}
$$
$\square$
### 3.3.2 Preservation Under Graph Operations
**Theorem 3.2 (Traversal Preserves Boolean Structure)**: Graph traversal operations produce posting lists that participate in Boolean algebra.
**Proof**: Let $\text{traverse}_k(v)$ denote $k$-hop traversal from vertex $v$:
$$
\text{traverse}_k(v) = \{u \in \mathcal{V}_G \mid \text{dist}(v, u) \leq k\}
$$
This is a subset of $\mathcal{V}_G$, hence an element of $2^{\mathcal{V}_G}$.
For combined traversals:
*Conjunction*: Vertices reachable from both $v_1$ and $v_2$:
$$
\text{traverse}_k(v_1) \cap \text{traverse}_k(v_2)
$$
*Disjunction*: Vertices reachable from either $v_1$ or $v_2$:
$$
\text{traverse}_k(v_1) \cup \text{traverse}_k(v_2)
$$
Both results are elements of $2^{\mathcal{V}_G}$, preserving Boolean structure. $\square$
### 3.3.3 Lattice Structure of Graph Queries
Graph queries form a complete lattice ordered by result containment:
**Definition**: For graph queries $Q_1, Q_2$, define $Q_1 \sqsubseteq Q_2$ iff $\text{result}(Q_1) \subseteq \text{result}(Q_2)$ for all graph instances.
**Theorem 3.3 (Graph Query Lattice)**: Graph queries ordered by $\sqsubseteq$ form a complete lattice.
The lattice operations:
- **Meet**: $Q_1 \sqcap Q_2$ returns intersection of results
- **Join**: $Q_1 \sqcup Q_2$ returns union of results
- **Bottom**: Query returning $\emptyset$
- **Top**: Query returning $\mathcal{V}_G$
This lattice structure enables the optimizer to navigate between equivalent graph query formulations.
## 3.4 Graph Algebra Operations
We define the operators that transform graph posting lists.
### 3.4.1 Adjacency Operator
The **adjacency operator** $\alpha$ expands a posting list to include adjacent vertices:
$$
\alpha: 2^{\mathcal{V}_G} \times 2^{\mathcal{T}} \times \{in, out, both\} \rightarrow 2^{\mathcal{V}_G}
$$
For vertex set $S$, edge types $T$, and direction $d$:
$$
\alpha(S, T, out) = \{u \mid \exists v \in S, t \in T: (v, u, t) \in \mathcal{E}_G\}
$$
$$
\alpha(S, T, in) = \{u \mid \exists v \in S, t \in T: (u, v, t) \in \mathcal{E}_G\}
$$
$$
\alpha(S, T, both) = \alpha(S, T, out) \cup \alpha(S, T, in)
$$
**Properties of adjacency**:
*Monotonicity*: $S_1 \subseteq S_2 \implies \alpha(S_1, T, d) \subseteq \alpha(S_2, T, d)$
*Distributivity over union*:
$$
\alpha(S_1 \cup S_2, T, d) = \alpha(S_1, T, d) \cup \alpha(S_2, T, d)
$$
*Edge type union*:
$$
\alpha(S, T_1 \cup T_2, d) = \alpha(S, T_1, d) \cup \alpha(S, T_2, d)
$$
### 3.4.2 Traversal Operator
The **traversal operator** $\tau_G$ performs multi-hop traversal:
$$
\tau_G: 2^{\mathcal{V}_G} \times 2^{\mathcal{T}} \times \mathbb{N} \times \{in, out, both\} \rightarrow 2^{\mathcal{V}_G}
$$
Defined recursively:
$$
\tau_G(S, T, 0, d) = S
$$
$$
\tau_G(S, T, k, d) = \tau_G(S, T, k-1, d) \cup \alpha(\tau_G(S, T, k-1, d), T, d)
$$
This computes the $k$-hop neighborhood: all vertices reachable within $k$ edge traversals.
**Fixed-point characterization**:
$$
\tau_G(S, T, \infty, d) = \mu X. (S \cup \alpha(X, T, d))
$$
The infinite traversal is the least fixed point of the adjacency expansion, representing the transitive closure.
### 3.4.3 Pattern Matching Operator
The **pattern matching operator** $\mu$ finds subgraph isomorphisms:
$$
\mu: \mathcal{G}_P \times \mathcal{V}_G \rightarrow 2^{\mathcal{V}_G}
$$
where $\mathcal{G}_P$ is a pattern graph. The result contains vertices that can serve as the anchor vertex in pattern matches.
**Example pattern**: Find vertices that are both a "person" and have an outgoing "knows" edge to another "person":
```
Pattern: (p1:Person)-[:KNOWS]->(p2:Person)
Anchor: p1
```
$$
\mu(P, v) = \{v \in \mathcal{V}_G \mid \text{Person} \in \text{labels}(v) \land \exists u: (\text{Person} \in \text{labels}(u) \land (v, u, \text{KNOWS}) \in \mathcal{E}_G)\}
$$
Pattern matching reduces to a conjunction of posting list operations:
1. Vertex label constraints: label posting lists
2. Edge existence constraints: adjacency posting lists
3. Property constraints: property posting lists
### 3.4.4 Regular Path Query Operator
**Regular Path Queries** (RPQ) find paths matching a regular expression over edge types:
$$
\text{RPQ}: \mathcal{V}_G \times \mathcal{R} \rightarrow 2^{\mathcal{V}_G}
$$
where $\mathcal{R}$ is the set of regular expressions over edge types.
**Grammar**:
$$
r ::= t \mid r_1 \cdot r_2 \mid r_1 | r_2 \mid r^* \mid r^+ \mid r?
$$
where $t \in \mathcal{T}$ is an edge type.
**Semantics**:
*Base case*: Edge type
$$
\text{RPQ}(v, t) = \alpha(\{v\}, \{t\}, out)
$$
*Concatenation*: Sequential traversal
$$
\text{RPQ}(v, r_1 \cdot r_2) = \bigcup_{u \in \text{RPQ}(v, r_1)} \text{RPQ}(u, r_2)
$$
*Alternation*: Union of paths
$$
\text{RPQ}(v, r_1 | r_2) = \text{RPQ}(v, r_1) \cup \text{RPQ}(v, r_2)
$$
*Kleene star*: Zero or more repetitions
$$
\text{RPQ}(v, r^*) = \mu X. (\{v\} \cup \text{RPQ}(X, r))
$$
*Kleene plus*: One or more repetitions
$$
\text{RPQ}(v, r^+) = \text{RPQ}(v, r \cdot r^*)
$$
*Optional*: Zero or one occurrence
$$
\text{RPQ}(v, r?) = \{v\} \cup \text{RPQ}(v, r)
$$
### 3.4.5 Shortest Path Operator
The **shortest path operator** $\pi_{sp}$ finds vertices connected by minimum-length paths:
$$
\pi_{sp}: \mathcal{V}_G \times \mathcal{V}_G \times 2^{\mathcal{T}} \rightarrow \mathcal{V}_G^* \cup \{\text{null}\}
$$
$$
\pi_{sp}(v, u, T) = \text{argmin}_{p: v \rightsquigarrow u} |p| \text{ where edges in } p \text{ have types in } T
$$
This operator returns a path (sequence of vertices) rather than a posting list. However, the **reachability check** derived from it returns a Boolean:
$$
\text{reachable}(v, u, T) = (\pi_{sp}(v, u, T) \neq \text{null})
$$
And **path existence** produces a posting list:
$$
\text{pathExists}(v, T, \text{maxLen}) = \{u \in \mathcal{V}_G \mid |\pi_{sp}(v, u, T)| \leq \text{maxLen}\}
$$
## 3.5 Operator Composition and Properties
### 3.5.1 Composition Monoid
Graph operators compose to form a monoid analogous to document operators:
$$
(\text{Op}_G, \circ, \text{id}_G)
$$
where:
- $\text{Op}_G$ is the set of graph operators
- $\circ$ is function composition
- $\text{id}_G$ is the identity on $2^{\mathcal{V}_G}$
**Associativity**: $(f \circ g) \circ h = f \circ (g \circ h)$
**Identity**: $f \circ \text{id}_G = \text{id}_G \circ f = f$
### 3.5.2 Algebraic Properties
**Adjacency Idempotence** (for undirected edges):
$$
\alpha(\alpha(S, T, both), T, both) \supseteq \alpha(S, T, both)
$$
Note: This is containment, not equality - repeated adjacency can reach more vertices.
**Traversal Monotonicity**:
$$
k_1 \leq k_2 \implies \tau_G(S, T, k_1, d) \subseteq \tau_G(S, T, k_2, d)
$$
**Traversal Fixed Point**:
$$
\exists k_0: \forall k \geq k_0: \tau_G(S, T, k, d) = \tau_G(S, T, k_0, d)
$$
The fixed point is reached when the traversal saturates (no new vertices discovered).
**Filter-Traversal Interaction**:
$$
\sigma_\phi(\tau_G(S, T, k, d)) \neq \tau_G(\sigma_\phi(S), T, k, d) \text{ in general}
$$
Filtering before traversal restricts starting vertices; filtering after traversal restricts ending vertices. These are semantically different and not interchangeable.
### 3.5.3 Equivalence-Preserving Transformations
Several transformations preserve query semantics:
**Traversal Decomposition**:
$$
\tau_G(S, T, k_1 + k_2, d) = \tau_G(\tau_G(S, T, k_1, d), T, k_2, d)
$$
**Adjacency Distribution**:
$$
\alpha(S_1 \cup S_2, T, d) = \alpha(S_1, T, d) \cup \alpha(S_2, T, d)
$$
**Pattern to Traversal Reduction**:
Simple chain patterns reduce to traversal:
$$
\mu((a)-[t_1]->(b)-[t_2]->(c), a) = \alpha(\alpha(P_V(a), \{t_1\}, out), \{t_2\}, out) \cap P_V(c)
$$
where $P_V(x)$ is the posting list for vertex pattern $x$.
## 3.6 Cross-Paradigm Integration
The power of the unified algebra emerges in cross-paradigm queries.
### 3.6.1 Graph-Relational Integration
**ToGraph Operator**: Convert relational results to graph vertices
$$
\text{ToGraph}: 2^{\mathcal{D}} \rightarrow 2^{\mathcal{V}_G}
$$
Under the document-vertex correspondence, this is the identity:
$$
\text{ToGraph}(P_D) = P_D \text{ (viewing documents as vertices)}
$$
**FromGraph Operator**: Convert graph results to relational rows
$$
\text{FromGraph}: 2^{\mathcal{V}_G} \rightarrow 2^{\mathcal{D}}
$$
Also the identity under correspondence.
**Example Query**: Find customers who purchased products in the same category as their friends:
```sql
SELECT DISTINCT c.name, p.name AS recommended_product
FROM customers c
-- Graph traversal: find friends
JOIN LATERAL (
SELECT friend_id FROM graph_traverse(c.id, 'FRIENDS_WITH', 1)
) f ON true
-- Friend's purchases
JOIN purchases fp ON fp.customer_id = f.friend_id
JOIN products friend_prod ON fp.product_id = friend_prod.id
-- Products in same category
JOIN products p ON p.category = friend_prod.category
-- Exclude already purchased
WHERE NOT EXISTS (
SELECT 1 FROM purchases cp
WHERE cp.customer_id = c.id AND cp.product_id = p.id
);
```
This query seamlessly combines:
- Relational joins (purchases, products)
- Graph traversal (friends)
- Set operations (exclusion)
The posting list representation enables unified optimization.
### 3.6.2 Graph-Vector Integration
**Vertex Embeddings**: Vertices can have vector representations:
$$
\text{embed}_V: \mathcal{V}_G \times \mathcal{F} \rightarrow \mathcal{V}_n
$$
This enables:
**Graph-aware similarity search**:
$$
\text{simSearch}(v, k, T) = \text{kNN}(\text{embed}_V(v), k) \cap \tau_G(\{v\}, T, \infty, both)
$$
Find the $k$ nearest neighbors that are also reachable via edges of type $T$.
**Embedding-based edge prediction**:
$$
\text{predictEdge}(v, t, \epsilon) = \{u \mid \text{sim}(\text{embed}_V(v), \text{embed}_V(u)) > \epsilon \land (v, u, t) \notin \mathcal{E}_G\}
$$
Predict new edges based on embedding similarity.
### 3.6.3 Graph-Text Integration
**Semantic Graph Search**: Combine text relevance with graph structure:
$$
\text{semanticGraphSearch}(q, v, k, T) = \text{topK}(\tau_t(q) \cap \tau_G(\{v\}, T, k, both), \text{BM25})
$$
Find documents matching text query $q$ that are within $k$ hops of vertex $v$ via edges of type $T$.
**Example**: Find research papers mentioning "machine learning" by authors within 2 collaboration hops:
```sql
SELECT p.title, bm25_score(p.abstract) AS relevance
FROM papers p
WHERE MATCH(p.abstract) AGAINST ('machine learning')
AND p.author_id IN (
SELECT vertex_id FROM graph_traverse(:current_author, 'COAUTHORED', 2)
)
ORDER BY relevance DESC
LIMIT 10;
```
### 3.6.4 Unified Query Plan
```mermaid
graph TB
subgraph "Unified Query Processing"
Q[Hybrid Query] --> P[Query Parser]
P --> A[Analyzer]
A --> LP[Logical Plan]
LP --> REL[Relational Predicates]
LP --> FTS[Text Predicates]
LP --> VEC[Vector Predicates]
LP --> GRF[Graph Predicates]
REL --> PL1[Posting List 1]
FTS --> PL2[Posting List 2]
VEC --> PL3[Posting List 3]
GRF --> PL4[Posting List 4]
PL1 --> MERGE[Boolean Merge]
PL2 --> MERGE
PL3 --> MERGE
PL4 --> MERGE
MERGE --> RESULT[Unified Result]
end
```
The query planner treats graph predicates as another source of posting lists, applying the same optimization strategies (predicate pushdown, intersection ordering by selectivity, etc.) across all paradigms.
## 3.7 Implementation Architecture
### 3.7.1 Graph Storage in LSM-Tree
Cognica stores graphs using the same LSM-tree infrastructure as documents:
**Vertex Storage**: Vertices store as documents with graph metadata:
```
Key: vertex:{graph_id}:{vertex_id}
Value: {properties, labels, ...}
```
**Edge Storage**: Edges store with composite keys for efficient traversal:
```
Key: edge:out:{graph_id}:{src_id}:{edge_type}:{tgt_id}
Value: {properties, ...}
Key: edge:in:{graph_id}:{tgt_id}:{edge_type}:{src_id}
Value: {} (reference only)
```
Dual edge storage (out and in) enables efficient traversal in both directions.
### 3.7.2 Adjacency Index
For each edge type, an adjacency index maps vertices to their neighbors:
```
Key: adj:out:{graph_id}:{edge_type}:{src_id}
Value: [tgt_id_1, tgt_id_2, ...] (posting list)
Key: adj:in:{graph_id}:{edge_type}:{tgt_id}
Value: [src_id_1, src_id_2, ...] (posting list)
```
This posting list structure enables:
- Fast single-hop expansion: $O(d)$ where $d$ is the vertex degree
- Efficient intersection with other posting lists
- Skip pointer acceleration for high-degree vertices
### 3.7.3 Label Index
Label indexes support vertex filtering:
```
Key: label:{graph_id}:{label}
Value: [vertex_id_1, vertex_id_2, ...] (posting list)
```
This enables efficient evaluation of label predicates.
### 3.7.4 Traversal Execution
Multi-hop traversal executes as iterative adjacency expansion:
```
Algorithm: BFS_Traversal(start, edge_types, max_depth, direction)
frontier = {start}
visited = {start}
for depth in 1..max_depth:
next_frontier = {}
for v in frontier:
neighbors = adjacency_lookup(v, edge_types, direction)
for u in neighbors:
if u not in visited:
visited.add(u)
next_frontier.add(u)
frontier = next_frontier
if frontier is empty:
break
return visited
```
**Optimization**: For intersection with other posting lists, early termination prunes branches:
```
Algorithm: Filtered_Traversal(start, edge_types, max_depth, direction, filter_posting)
frontier = {start} intersect filter_posting
visited = frontier
for depth in 1..max_depth:
next_frontier = {}
for v in frontier:
neighbors = adjacency_lookup(v, edge_types, direction)
// Early filter application
filtered_neighbors = neighbors intersect filter_posting
for u in filtered_neighbors:
if u not in visited:
visited.add(u)
next_frontier.add(u)
frontier = next_frontier
if frontier is empty:
break
return visited
```
### 3.7.5 Pattern Match Execution
Pattern matching compiles to a join tree of posting list operations:
**Example Pattern**:
```
(a:Person {age > 30})-[:KNOWS]->(b:Person)-[:WORKS_AT]->(c:Company {name = 'Acme'})
```
**Execution Plan**:
1. $P_a$ = label_index(Person) $\cap$ property_filter(age > 30)
2. $P_c$ = label_index(Company) $\cap$ property_filter(name = 'Acme')
3. $P_b$ = adj_in(c, WORKS_AT) $\cap$ label_index(Person)
4. $P_{a,final}$ = adj_in(b, KNOWS) $\cap$ $P_a$
5. Return $P_{a,final}$
The optimizer reorders these operations based on selectivity estimates.
### 3.7.6 SQL Graph Query Interface
Cognica exposes graph operations through SQL table functions, enabling graph traversal within standard SQL queries:
**graph_traverse()** — Multi-hop traversal as a table function:
```sql
-- Find all users reachable within 3 hops via FOLLOWS edges
SELECT t.vertex_id, t.depth, t.path
FROM graph_traverse(
'social_graph', -- graph name
'user_123', -- start vertex
'FOLLOWS', -- edge type
3, -- max depth
'outgoing' -- direction
) t;
-- Combine with relational predicates
SELECT u.name, t.depth
FROM graph_traverse('social_graph', 'user_123', 'FOLLOWS', 2, 'outgoing') t
JOIN users u ON u.id = t.vertex_id
WHERE u.active = true;
```
**Recursive CTE Integration** — Standard SQL recursive queries can express path-finding:
```sql
-- Shortest path via recursive CTE
WITH RECURSIVE paths AS (
SELECT id AS vertex_id, ARRAY[id] AS path, 0 AS depth
FROM vertices WHERE id = 'user_123'
UNION ALL
SELECT e.target_id, p.path || e.target_id, p.depth + 1
FROM paths p
JOIN edges e ON e.source_id = p.vertex_id AND e.type = 'FOLLOWS'
WHERE p.depth < 5 AND NOT e.target_id = ANY(p.path)
)
SELECT * FROM paths WHERE vertex_id = 'user_456'
ORDER BY depth LIMIT 1;
```
**Cypher Query Support** — Optional Cypher syntax via the `cypher()` table function, compatible with Apache AGE:
```sql
-- Cypher via table function
SELECT * FROM cypher('social_graph', $$
MATCH (a:Person {name: 'Alice'})-[:KNOWS*1..3]->(b:Person)
RETURN b.name, b.age
$$) AS (name TEXT, age INTEGER);
```
**Adjacency Cache** — An in-memory cache accelerates repeated traversals by caching adjacency lists for frequently accessed vertices, reducing RocksDB lookups during iterative graph algorithms.
## 3.8 Scored Graph Operations
Graph operations can carry scores for ranking.
### 3.8.1 Edge Weight Scores
Edges may have weights representing relationship strength:
$$
w: \mathcal{E}_G \rightarrow \mathbb{R}^+
$$
**Weighted adjacency** returns scored posting lists:
$$
\alpha_w(S, T, d) = \{(u, w(e)) \mid u \in \alpha(S, T, d), e \text{ connects } S \text{ to } u\}
$$
### 3.8.2 Path Scores
Paths aggregate edge weights:
**Additive path score**:
$$
\text{score}(p) = \sum_{e \in p} w(e)
$$
**Multiplicative path score** (for probability):
$$
\text{score}(p) = \prod_{e \in p} w(e)
$$
**Min path score** (for bottleneck):
$$
\text{score}(p) = \min_{e \in p} w(e)
$$
### 3.8.3 PageRank and Centrality
**PageRank** assigns importance scores to vertices:
$$
\text{PR}(v) = \frac{1 - d}{N} + d \sum_{u \in \text{in}(v)} \frac{\text{PR}(u)}{|\text{out}(u)|}
$$
where $d$ is the damping factor (typically 0.85) and $N$ is the vertex count.
PageRank precomputes as a vertex property, enabling score-aware queries:
```sql
SELECT v.name, v.pagerank
FROM vertices v
WHERE v.id IN (
SELECT vertex_id FROM graph_traverse(:start, 'LINKS_TO', 3)
)
ORDER BY v.pagerank DESC
LIMIT 10;
```
### 3.8.4 Score Combination with Other Paradigms
Graph scores combine with text and vector scores using the frameworks from Chapter 2:
**Example**: Hybrid ranking combining BM25 and PageRank:
$$
\text{score}(d) = \alpha \cdot \text{normalize}(\text{BM25}(d, q)) + (1 - \alpha) \cdot \text{PR}(d)
$$
Or probabilistically:
$$
\text{score}(d) = P_{\text{BM25}}(d) \cdot P_{\text{PR}}(d)
$$
where scores are calibrated to $[0, 1]$.
## 3.9 Query Optimization for Graph Operations
### 3.9.1 Selectivity Estimation
Graph operation selectivity depends on graph structure:
**Adjacency selectivity**:
$$
\text{sel}(\alpha(S, T, d)) \approx |S| \cdot \text{avgDegree}(T, d) / |\mathcal{V}_G|
$$
**Traversal selectivity** (k hops):
$$
\text{sel}(\tau_G(S, T, k, d)) \approx \min(1, |S| \cdot \text{avgDegree}(T, d)^k / |\mathcal{V}_G|)
$$
**Pattern selectivity**: Product of component selectivities:
$$
\text{sel}(\mu(P)) \approx \prod_{c \in \text{constraints}(P)} \text{sel}(c)
$$
### 3.9.2 Join Ordering
Pattern matching is essentially a join problem. The optimizer orders pattern components by selectivity:
1. Start with most selective constraint (smallest posting list)
2. Expand through adjacency with next most selective constraint
3. Continue until pattern is matched
**Example**: For pattern `(a:Person)-[:KNOWS]->(b:Influencer)-[:PROMOTES]->(c:Product)`:
If `Influencer` is rare (high selectivity), start from `b`:
1. $P_b$ = label_index(Influencer)
2. $P_a$ = adj_in(b, KNOWS) $\cap$ label_index(Person)
3. $P_c$ = adj_out(b, PROMOTES) $\cap$ label_index(Product)
### 3.9.3 Traversal Pruning
Early termination strategies for traversal:
**Top-k pruning**: When seeking top-k results by score, maintain a threshold and prune branches that cannot exceed it.
**Filter pushdown**: Apply filters at each traversal step rather than at the end.
**Bidirectional search**: For point-to-point queries, search from both ends and meet in the middle.
## 3.10 Summary
This chapter extended the unified algebra to incorporate graph structures:
1. **Graph type system** formalizes vertices, edges, properties, and labels within the same framework as documents, with explicit document-vertex correspondence enabling zero-duplication storage
2. **Graph-posting list isomorphism** proves that graph operations produce posting lists satisfying the same Boolean algebra as document posting lists, enabling unified optimization
3. **Graph algebra operators** include adjacency ($\alpha$), traversal ($\tau_G$), pattern matching ($\mu$), regular path queries (RPQ), and shortest paths ($\pi_{sp}$), all composing through the same monoid structure as document operators
4. **Cross-paradigm integration** enables queries combining relational predicates, text search, vector similarity, and graph traversal through unified posting list operations
5. **Implementation architecture** stores graphs in LSM-trees with dual-direction edge indexes and label indexes, supporting efficient traversal and pattern matching
6. **Scored graph operations** extend the algebra to handle edge weights, path scores, and centrality measures, combining with text and vector scores through the same frameworks
The following chapter develops query optimization theory, showing how the algebraic properties established in Chapters 2 and 3 enable cost-based optimization across all paradigms.
# Chapter 4: Query Optimization Theory
This chapter develops the theoretical foundations for cost-based query optimization in unified systems. We formalize the query space as a complete lattice, establish equivalence-preserving transformations, build cost models spanning multiple paradigms, and develop selectivity estimation techniques. The chapter concludes with a category-theoretic perspective that reveals the deep structure underlying query optimization.
## 4.1 The Query Optimization Problem
Query optimization is the process of transforming a declarative query into an efficient execution plan. The declarative query specifies *what* data to retrieve; optimization determines *how* to retrieve it.
### 4.1.1 Search Space Explosion
Consider a simple join of five tables:
```sql
SELECT * FROM A, B, C, D, E
WHERE A.x = B.x AND B.y = C.y AND C.z = D.z AND D.w = E.w;
```
The number of possible join orderings is:
$$
\frac{(2(n-1))!}{(n-1)!} = \frac{8!}{4!} = 1680 \text{ for } n = 5
$$
For each ordering, multiple physical implementations exist (nested loop, hash join, merge join). With index choices, the search space grows exponentially.
### 4.1.2 Multi-Paradigm Complexity
In a unified system, optimization spans paradigms:
```sql
SELECT p.title, bm25_score(p.content) as relevance
FROM papers p
WHERE MATCH(p.content) AGAINST ('distributed systems')
AND vector_similarity(p.embedding, ?) > 0.8
AND p.author_id IN (
SELECT vertex_id FROM graph_traverse(:professor, 'ADVISED', 3)
)
ORDER BY relevance DESC
LIMIT 10;
```
This query involves:
- Full-text search (MATCH...AGAINST)
- Vector similarity (embedding comparison)
- Graph traversal (advisor relationships)
- Relational filtering (author constraint)
The optimizer must reason about posting list intersections, index selection across paradigms, and cost trade-offs between different execution strategies.
### 4.1.3 Optimization Objectives
A query optimizer seeks to minimize a cost function:
$$
\text{minimize } C(P) = w_{io} \cdot C_{io}(P) + w_{cpu} \cdot C_{cpu}(P) + w_{mem} \cdot C_{mem}(P)
$$
where:
- $C_{io}(P)$ is I/O cost (disk reads/writes)
- $C_{cpu}(P)$ is CPU cost (operations performed)
- $C_{mem}(P)$ is memory cost (working memory required)
- $w_{*}$ are weights reflecting system characteristics
Different systems weight these differently: OLTP systems prioritize latency (minimize $C_{io}$), OLAP systems prioritize throughput (minimize $C_{cpu}$ per row), memory-constrained systems bound $C_{mem}$.
## 4.2 Query Space as Complete Lattice
We formalize the space of query plans as a mathematical structure amenable to optimization algorithms.
### 4.2.1 Logical Query Plans
A **logical query plan** is a tree of relational operators:
$$
\mathcal{L} = \{\text{Scan}, \text{Filter}, \text{Project}, \text{Join}, \text{Aggregate}, \text{Sort}, \text{Limit}, ...\}
$$
Each operator has:
- Input schema(s)
- Output schema
- Semantic meaning (what it computes)
**Example logical plan**:
```
Limit(10)
Sort(relevance DESC)
Project(title, relevance)
Filter(vector_sim > 0.8)
Join(author_id = vertex_id)
FTSSearch(content, 'distributed systems')
Scan(papers)
GraphTraverse(professor, ADVISED, 3)
```
### 4.2.2 Physical Query Plans
A **physical query plan** specifies concrete implementations:
$$
\mathcal{P} = \{\text{SeqScan}, \text{IndexScan}, \text{NestedLoopJoin}, \text{HashJoin}, \text{MergeJoin}, ...\}
$$
Each physical operator has:
- Implementation algorithm
- Cost characteristics
- Resource requirements
**Example physical plan**:
```
HeapTopK(10, relevance DESC)
Project(title, relevance)
HashJoin(author_id = vertex_id)
PostingListIntersect
WANDSearch(content, 'distributed systems', k=1000)
HNSWSearch(embedding, query_vec, k=1000, threshold=0.8)
BFSTraversal(professor, ADVISED, max_depth=3)
```
### 4.2.3 Plan Equivalence
Two plans are **equivalent** if they produce identical results for all database states:
$$
P_1 \equiv P_2 \iff \forall D: \text{eval}(P_1, D) = \text{eval}(P_2, D)
$$
Equivalence induces equivalence classes on the plan space.
### 4.2.4 Partial Ordering by Cost
Define a partial order on plans by estimated cost:
$$
P_1 \preceq P_2 \iff C(P_1) \leq C(P_2)
$$
Within an equivalence class, the optimizer seeks the minimum element under $\preceq$.
### 4.2.5 Lattice Structure
**Theorem 4.1 (Query Plan Lattice)**: The space of query plans for a given query, ordered by cost, forms a complete lattice.
**Proof sketch**:
- **Bottom**: The optimal plan (minimum cost)
- **Top**: The worst plan (maximum cost)
- **Meet**: Given plans $P_1, P_2$, their meet is the cheaper of any common refinement
- **Join**: The join is the more expensive plan from which both are reachable by cost-reducing transformations
The lattice structure guarantees that local search algorithms (like dynamic programming) can find global optima under appropriate conditions.
### 4.2.6 Plan Space Navigation
Optimization algorithms navigate the lattice:
**Dynamic Programming (DPccp)**: Bottom-up construction of optimal subplans, guaranteed to find global optimum for join ordering.
**Transformation-Based**: Apply equivalence rules to transform plans, hill-climbing toward lower cost.
**Randomized**: Simulated annealing or genetic algorithms for large search spaces.
```mermaid
graph TB
subgraph "Plan Lattice Navigation"
TOP[Worst Plan - Top]
P1[Plan A]
P2[Plan B]
P3[Plan C]
OPT[Optimal Plan - Bottom]
TOP --> P1
TOP --> P2
P1 --> P3
P2 --> P3
P3 --> OPT
end
```
## 4.3 Equivalence-Preserving Transformations
Transformations convert one plan to an equivalent plan, enabling search through the plan space.
### 4.3.1 Selection Pushdown
**Rule**: Push selections through projections when possible.
$$
\sigma_\phi(\pi_A(R)) \equiv \pi_A(\sigma_\phi(R)) \quad \text{if } \text{attrs}(\phi) \subseteq A
$$
**Benefit**: Filter early reduces intermediate result sizes.
**Example**:
```
Before: Project(name, price) -> Filter(price > 100) -> Scan(products)
After: Project(name, price) -> Scan(products) with pushed filter price > 100
```
### 4.3.2 Selection Split and Merge
**Rule**: Conjunctive predicates can split or merge.
$$
\sigma_{\phi_1 \land \phi_2}(R) \equiv \sigma_{\phi_1}(\sigma_{\phi_2}(R))
$$
**Benefit**: Enables independent optimization of each predicate.
### 4.3.3 Selection Commutativity
**Rule**: Independent selections commute.
$$
\sigma_{\phi_1}(\sigma_{\phi_2}(R)) \equiv \sigma_{\phi_2}(\sigma_{\phi_1}(R))
$$
**Benefit**: Enables ordering by selectivity (most selective first).
### 4.3.4 Join Commutativity
**Rule**: Joins are commutative.
$$
R \bowtie S \equiv S \bowtie R
$$
**Benefit**: Choose build/probe sides for hash join based on size.
### 4.3.5 Join Associativity
**Rule**: Joins are associative.
$$
(R \bowtie S) \bowtie T \equiv R \bowtie (S \bowtie T)
$$
**Benefit**: Enables exploring all join orderings.
### 4.3.6 Selection-Join Exchange
**Rule**: Push selection through join.
$$
\sigma_\phi(R \bowtie S) \equiv \sigma_\phi(R) \bowtie S \quad \text{if } \text{attrs}(\phi) \subseteq \text{attrs}(R)
$$
$$
\sigma_\phi(R \bowtie S) \equiv R \bowtie \sigma_\phi(S) \quad \text{if } \text{attrs}(\phi) \subseteq \text{attrs}(S)
$$
**Benefit**: Reduces join input sizes.
### 4.3.7 Projection Pushdown
**Rule**: Push projection through join.
$$
\pi_A(R \bowtie_\theta S) \equiv \pi_A(\pi_{A_R \cup J}(R) \bowtie_\theta \pi_{A_S \cup J}(S))
$$
where $J$ = join attributes, $A_R$ = $A \cap \text{attrs}(R)$, $A_S$ = $A \cap \text{attrs}(S)$.
**Benefit**: Reduces data width in intermediate results.
### 4.3.8 Cross-Paradigm Transformations
Unified systems enable novel transformations:
**FTS-Filter Interchange**:
$$
\sigma_\phi(\text{FTS}(t, R)) \equiv \text{FTS}(t, \sigma_\phi(R)) \quad \text{when } \phi \text{ independent of FTS}
$$
**Vector-Filter Interchange**:
$$
\sigma_\phi(\text{kNN}(v, k, R)) \equiv \text{kNN}(v, k', \sigma_\phi(R)) \quad \text{with adjusted } k'
$$
Note: Vector search may require $k' > k$ to account for filtered-out results.
**Graph-Relational Interchange**:
$$
\text{GraphTraverse}(v, t, k) \cap \sigma_\phi(R) \equiv \text{FilteredTraverse}(v, t, k, \phi)
$$
When filtering can push into traversal.
## 4.4 Cost Model Fundamentals
Cost models estimate execution cost without actually running queries.
### 4.4.1 I/O Cost Model
**Sequential Read Cost**:
$$
C_{seq}(n) = n \cdot c_{seq}
$$
where $n$ = pages read, $c_{seq}$ = cost per sequential page read.
**Random Read Cost**:
$$
C_{rand}(n) = n \cdot c_{rand}
$$
where $c_{rand} \gg c_{seq}$ (typically 10-100x for HDD, 2-10x for SSD).
**Index Scan Cost**:
$$
C_{idx}(R, \phi) = \text{height}(I) \cdot c_{rand} + \text{sel}(\phi) \cdot |R| \cdot c_{rand}
$$
Index lookup plus data page fetches for matching rows.
**Sequential Scan Cost**:
$$
C_{scan}(R) = \text{pages}(R) \cdot c_{seq}
$$
### 4.4.2 CPU Cost Model
**Filter Cost**:
$$
C_{filter}(\phi, n) = n \cdot c_{comp}(\phi)
$$
where $c_{comp}(\phi)$ depends on predicate complexity.
**Hash Table Build Cost**:
$$
C_{build}(R) = |R| \cdot c_{hash}
$$
**Hash Probe Cost**:
$$
C_{probe}(S, R) = |S| \cdot c_{probe}
$$
**Sort Cost**:
$$
C_{sort}(R) = |R| \cdot \log(|R|) \cdot c_{cmp}
$$
### 4.4.3 Memory Cost Model
**Hash Join Memory**:
$$
M_{hash}(R) = |R| \cdot \text{row\_size}(R) \cdot (1 + \text{overhead})
$$
If $M_{hash}(R) > M_{available}$, spill to disk.
**Sort Memory**:
$$
M_{sort}(R) = \min(|R| \cdot \text{row\_size}(R), M_{available})
$$
External sort required when input exceeds memory.
### 4.4.4 Join Cost Models
**Nested Loop Join**:
$$
C_{NLJ}(R, S) = |R| \cdot c_{outer} + |R| \cdot |S| \cdot c_{inner}
$$
**Hash Join**:
$$
C_{HJ}(R, S) = |R| \cdot c_{build} + |S| \cdot c_{probe} + |R \bowtie S| \cdot c_{output}
$$
**Merge Join** (sorted inputs):
$$
C_{MJ}(R, S) = |R| \cdot c_{merge} + |S| \cdot c_{merge}
$$
**Decision Criterion**:
- Use NLJ when $|R|$ small or inner indexed
- Use HJ when memory sufficient for build side
- Use MJ when inputs pre-sorted or sort cost amortized
### 4.4.5 Posting List Operation Costs
**Intersection Cost**:
$$
C_\cap(P_1, P_2) = \min(|P_1|, |P_2|) \cdot c_{seek} + \min(|P_1|, |P_2|) \cdot c_{cmp}
$$
With skip pointers, seeks skip past non-matching regions.
**Union Cost**:
$$
C_\cup(P_1, P_2) = (|P_1| + |P_2|) \cdot c_{merge}
$$
Linear merge of sorted lists.
**FTS Scoring Cost**:
$$
C_{BM25}(P, q) = |P| \cdot |q| \cdot c_{score}
$$
Score computation for each document-term pair.
### 4.4.6 Vector Search Cost
**HNSW Search Cost**:
$$
C_{HNSW}(k, ef) = ef \cdot \log(N) \cdot c_{dist}
$$
where $ef$ = search width, $N$ = index size, $c_{dist}$ = distance computation cost.
**Brute Force Cost**:
$$
C_{brute}(N, d) = N \cdot d \cdot c_{mul}
$$
where $d$ = vector dimension.
### 4.4.7 Graph Traversal Cost
**BFS Cost**:
$$
C_{BFS}(k, d_{avg}) = \sum_{i=1}^{k} d_{avg}^i \cdot c_{expand}
$$
where $k$ = depth, $d_{avg}$ = average degree.
**With Filter**:
$$
C_{filtered\_BFS}(k, d_{avg}, sel) = \sum_{i=1}^{k} (d_{avg} \cdot sel)^i \cdot c_{expand}
$$
Filter selectivity reduces effective degree at each hop.
## 4.5 Cardinality Estimation
Accurate cardinality estimation is critical for cost-based optimization.
### 4.5.1 Base Table Statistics
For each table $R$, maintain:
- $|R|$: Row count
- $V(A, R)$: Distinct values for attribute $A$
- $\min(A, R)$, $\max(A, R)$: Value range
- $\text{hist}(A, R)$: Value histogram
### 4.5.2 Single Predicate Selectivity
**Equality**: $\sigma_{A=v}(R)$
$$
\text{sel}(A = v) = \frac{1}{V(A, R)}
$$
**Range**: $\sigma_{A > v}(R)$
$$
\text{sel}(A > v) = \frac{\max(A, R) - v}{\max(A, R) - \min(A, R)}
$$
**With histogram**: Use histogram bucket frequencies for more accurate estimation.
### 4.5.3 Compound Predicate Selectivity
**Independence Assumption**:
$$
\text{sel}(\phi_1 \land \phi_2) = \text{sel}(\phi_1) \cdot \text{sel}(\phi_2)
$$
**With Correlation**:
$$
\text{sel}(\phi_1 \land \phi_2) = \text{sel}(\phi_1) \cdot \text{sel}(\phi_2) \cdot \text{corr}(\phi_1, \phi_2)
$$
where $\text{corr}(\phi_1, \phi_2)$ captures dependency (1 = independent, >1 = positive correlation, <1 = negative correlation).
### 4.5.4 Join Cardinality
**Foreign Key Join**:
$$
|R \bowtie_{R.fk = S.pk} S| = |R|
$$
Each row in $R$ matches exactly one row in $S$.
**General Equijoin**:
$$
|R \bowtie_{R.A = S.B} S| = \frac{|R| \cdot |S|}{\max(V(A, R), V(B, S))}
$$
Assuming uniform distribution.
**With Statistics**:
$$
|R \bowtie_{R.A = S.B} S| = \sum_{v} \text{freq}(v, R.A) \cdot \text{freq}(v, S.B)
$$
### 4.5.5 FTS Cardinality
**Term Query**:
$$
|\tau_t| = \text{df}(t)
$$
Document frequency from inverted index.
**Conjunction**:
$$
|\tau_{t_1} \cap \tau_{t_2}| \approx \frac{\text{df}(t_1) \cdot \text{df}(t_2)}{N}
$$
Independence assumption.
**With Co-occurrence Statistics**:
$$
|\tau_{t_1} \cap \tau_{t_2}| = \text{codf}(t_1, t_2)
$$
Pre-computed co-occurrence counts.
### 4.5.6 Vector Search Cardinality
**k-NN Search**:
$$
|\text{kNN}(q, k)| = k
$$
By definition.
**Threshold Search**:
$$
|\text{sim}(q, \cdot) > \theta| \approx N \cdot P(\text{sim} > \theta)
$$
Requires distribution model for similarity scores.
### 4.5.7 Graph Traversal Cardinality
**k-Hop Traversal**:
$$
|\tau_G(v, T, k)| \approx \min(N, d_{avg}^k)
$$
Exponential growth bounded by graph size.
**With Selectivity**:
$$
|\tau_G(v, T, k) \cap \sigma_\phi| \approx \min(N \cdot \text{sel}(\phi), d_{avg}^k \cdot \text{sel}(\phi)^k)
$$
Filter applies at each hop.
## 4.6 Selectivity Estimation Techniques
### 4.6.1 Histograms
**Equi-width Histogram**: Divide value range into equal-width buckets.
$$
\text{sel}(A \in [l, u]) = \sum_{b: b \cap [l, u] \neq \emptyset} \frac{|b \cap [l, u]|}{|b|} \cdot \text{freq}(b)
$$
**Equi-depth Histogram**: Each bucket contains equal number of values.
Better for skewed distributions.
**Compressed Histogram**: Store frequent values separately, histogram for remainder.
### 4.6.2 Sampling
**Random Sampling**: Estimate selectivity from sample.
$$
\text{sel}(\phi) \approx \frac{|\sigma_\phi(\text{sample})|}{|\text{sample}|}
$$
**Confidence Interval**:
$$
\text{sel}(\phi) \in \left[\hat{p} - z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, \hat{p} + z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\right]
$$
### 4.6.3 Sketches
**Count-Min Sketch**: Estimate frequency of items.
Space: $O(\frac{1}{\epsilon} \log \frac{1}{\delta})$ for $(1+\epsilon)$ approximation with probability $1-\delta$.
**HyperLogLog**: Estimate cardinality (distinct count).
Space: $O(\log \log N)$ for relative error $\frac{1.04}{\sqrt{m}}$.
### 4.6.4 Machine Learning Approaches
**Query-Driven Learning**: Train models on query workload.
$$
\text{sel}(\phi) = f_\theta(\text{features}(\phi))
$$
Features include: predicate type, column statistics, query structure.
**Learned Cardinality Estimation**: Neural networks trained on actual cardinalities.
## 4.7 The DPccp Algorithm
Dynamic Programming for Connected Subgraph Complement Pairs (DPccp) is the standard algorithm for optimal join ordering.
### 4.7.1 Problem Formulation
Given relations $R_1, ..., R_n$ and join graph $G = (V, E)$ where vertices are relations and edges are join predicates:
**Goal**: Find minimum-cost join tree.
### 4.7.2 Algorithm
```
Algorithm: DPccp(R_1, ..., R_n, G)
// Initialize single relations
for i in 1..n:
opt[{R_i}] = AccessPath(R_i)
// Enumerate subgraph complement pairs
for S in subsets(R_1, ..., R_n) ordered by size:
for (S_1, S_2) in ccp_pairs(S, G):
for each join algorithm J:
cost = C(J, opt[S_1], opt[S_2])
if cost < opt[S].cost:
opt[S] = (J, opt[S_1], opt[S_2], cost)
return opt[{R_1, ..., R_n}]
```
**ccp_pairs(S, G)**: Enumerate pairs $(S_1, S_2)$ where $S_1 \cup S_2 = S$, $S_1 \cap S_2 = \emptyset$, and both $S_1$ and $S_2$ are connected in $G$.
### 4.7.3 Complexity
- Subsets: $O(2^n)$
- CCP pairs per subset: $O(3^n / 2^n)$ on average
- Total: $O(3^n)$
For large $n$, heuristics or randomized algorithms necessary.
### 4.7.4 Extensions for Unified Queries
DPccp extends to multi-paradigm queries by:
1. Treating FTS/Vector/Graph operations as "virtual relations"
2. Modeling posting list intersections as joins
3. Including paradigm-specific cost models
## 4.8 Category-Theoretic Perspective
For readers with category theory background, we sketch the categorical view of query optimization.
### 4.8.1 Category of Queries
Define category $\mathbf{Query}$:
- **Objects**: Query plans (logical or physical)
- **Morphisms**: Equivalence-preserving transformations
This forms a groupoid (every morphism is invertible) since transformations preserve equivalence.
### 4.8.2 Cost Functor
The cost function defines a functor:
$$
C: \mathbf{Query} \rightarrow \mathbf{R}^+
$$
mapping query plans to their estimated costs.
**Functoriality**: Transformations that preserve equivalence should have predictable cost effects:
$$
C(T(Q)) = C(Q) + \Delta(T)
$$
where $\Delta(T)$ is the cost delta of transformation $T$.
### 4.8.3 Natural Transformations as Optimizations
An optimization strategy is a natural transformation:
$$
\eta: \text{Id}_{\mathbf{Query}} \Rightarrow \text{Opt}
$$
where $\text{Opt}$ is the "optimized plan" functor.
Naturality ensures optimization is consistent across equivalent query formulations.
### 4.8.4 Adjunctions in Query Processing
**Free-Forgetful Adjunction**:
$$
F: \mathbf{SQL} \rightleftharpoons \mathbf{Plan} : U
$$
- $F$ (free functor): Compile SQL to logical plan
- $U$ (forgetful functor): Extract SQL semantics from plan
The adjunction captures that plans are "free algebras" over SQL expressions.
**Embedding-Projection Adjunction**:
$$
E: \mathbf{Logical} \rightleftharpoons \mathbf{Physical} : P
$$
- $E$ (embed): Logical plan as abstract physical plan
- $P$ (project): Physical plan's logical semantics
This adjunction formalizes the relationship between logical and physical planning.
### 4.8.5 Monad of Query Execution
Query execution forms a monad:
$$
\text{Exec}: \mathbf{Plan} \rightarrow \mathbf{Plan}
$$
with:
- $\eta_P: P \rightarrow \text{Exec}(P)$ (plan becomes executable)
- $\mu_P: \text{Exec}(\text{Exec}(P)) \rightarrow \text{Exec}(P)$ (flatten nested execution)
The monad laws ensure consistent execution semantics.
## 4.9 Practical Optimization Architecture
### 4.9.1 Two-Phase Optimization
**Phase 1: Logical Optimization**
- Apply transformation rules
- Generate alternative logical plans
- Prune obviously suboptimal plans
**Phase 2: Physical Optimization**
- Select physical operators
- Choose access paths
- Determine join algorithms
```mermaid
graph TB
subgraph "Query Optimization Pipeline"
SQL[SQL Query] --> PARSE[Parser]
PARSE --> AST[AST]
AST --> LOGICAL[Logical Plan]
LOGICAL --> TRANSFORM[Transformation Rules]
TRANSFORM --> ENUM[Plan Enumeration]
ENUM --> COST[Cost Estimation]
COST --> SELECT[Plan Selection]
SELECT --> PHYSICAL[Physical Plan]
PHYSICAL --> CODEGEN[Code Generation]
CODEGEN --> EXEC[Execution]
end
```
### 4.9.2 Rule-Based Optimization
Apply transformation rules in priority order:
1. **Predicate simplification**: Constant folding, contradiction detection
2. **Predicate pushdown**: Move filters toward data sources
3. **Projection pruning**: Remove unused columns
4. **Join elimination**: Remove unnecessary joins (e.g., to unique key)
5. **Subquery unnesting**: Convert correlated subqueries to joins
### 4.9.3 Cost-Based Optimization
After rule-based transformations:
1. Enumerate join orderings (DPccp or heuristic)
2. For each ordering, select physical operators
3. Estimate cost of each plan
4. Select minimum-cost plan
### 4.9.4 Adaptive Optimization
Modern optimizers adapt during execution:
**Adaptive Join Selection**: Switch join algorithm based on runtime cardinalities.
**Adaptive Parallelism**: Adjust parallelism based on observed throughput.
**Re-optimization**: Re-plan query mid-execution if estimates badly wrong.
## 4.10 Cross-Paradigm Optimization
### 4.10.1 Unified Cost Model
Cross-paradigm optimization requires a unified cost model:
$$
C_{total} = C_{relational} + C_{FTS} + C_{vector} + C_{graph} + C_{integration}
$$
The integration cost captures posting list operations that combine paradigm results.
### 4.10.2 Paradigm Selection
Given a predicate expressible in multiple paradigms, choose the cheapest:
**Example**: Find documents where `category = 'electronics'`
Options:
1. Relational: Secondary index scan
2. FTS: Term query on category field
3. Graph: Label-based vertex filter
Cost comparison:
$$
C_{rel} = \text{height}(idx) \cdot c_{rand} + |\sigma| \cdot c_{rand}
$$
$$
C_{FTS} = c_{term\_lookup} + |P_{electronics}| \cdot c_{decode}
$$
$$
C_{graph} = c_{label\_lookup} + |V_{electronics}| \cdot c_{decode}
$$
Select minimum.
### 4.10.3 Intersection Ordering
For conjunctive queries across paradigms:
```sql
WHERE MATCH(content) AGAINST ('database') -- FTS
AND category = 'tech' -- Relational
AND vector_sim(embedding, ?) > 0.8 -- Vector
```
Order intersections by:
1. Selectivity (most selective first)
2. Evaluation cost (cheapest first, tie-breaker)
If FTS selectivity = 0.01, relational selectivity = 0.1, vector selectivity = 0.05:
Order: FTS (0.01) -> Vector (0.05) -> Relational (0.1)
### 4.10.4 Materialization Points
Decide where to materialize intermediate results:
**Eager Materialization**: Materialize after each paradigm operation.
- Pro: Simple, predictable memory
- Con: May materialize large intermediate results
**Lazy Materialization**: Defer materialization until necessary.
- Pro: Avoids unnecessary work
- Con: Complex planning, potential repeated evaluation
**Hybrid**: Materialize based on estimated intermediate sizes.
## 4.11 Summary
This chapter established the theoretical foundations for query optimization:
1. **Query space as lattice** formalizes the search space with partial ordering by cost, enabling systematic exploration toward optimal plans
2. **Equivalence-preserving transformations** include classical rules (selection pushdown, join reordering) extended with cross-paradigm transformations for unified systems
3. **Cost models** span I/O, CPU, and memory costs for relational, FTS, vector, and graph operations, enabling unified cost estimation
4. **Selectivity estimation** uses histograms, sampling, and sketches for cardinality prediction, critical for accurate cost estimation
5. **DPccp algorithm** provides optimal join ordering through dynamic programming over connected subgraph complement pairs
6. **Category-theoretic perspective** reveals query optimization as navigation through a category of plans with cost as a functor and optimization strategies as natural transformations
7. **Cross-paradigm optimization** requires unified cost models, paradigm selection, intersection ordering, and materialization decisions
The following chapters (Part II) examine the storage engine that underlies all these operations, starting with LSM-tree architecture in Chapter 5.
# Chapter 5: LSM-Tree Storage Architecture
This chapter explores the Log-Structured Merge-tree (LSM-tree) storage architecture that forms the foundation of Cognica's persistence layer. We examine the theoretical principles behind LSM-trees, their trade-offs compared to B-tree structures, and the specific implementation choices that optimize for unified query workloads spanning documents, full-text search, and vector operations.
## 5.1 Storage Engine Fundamentals
Every database system must answer a fundamental question: how should data be organized on persistent storage to optimize both read and write operations? The answer to this question shapes the entire system architecture.
### 5.1.1 The Read-Write Trade-off
Storage engines face an inherent tension between read and write performance. Consider two extreme strategies:
**Write-Optimized (Append-Only Log)**:
$$
T_{write} = O(1) \quad \text{(append to end)}
$$
$$
T_{read} = O(n) \quad \text{(scan entire log)}
$$
An append-only log achieves constant-time writes by simply appending new records. However, reads require scanning the entire log to find relevant records, resulting in linear time complexity.
**Read-Optimized (Sorted Array)**:
$$
T_{write} = O(n) \quad \text{(maintain sorted order)}
$$
$$
T_{read} = O(\log n) \quad \text{(binary search)}
$$
A sorted array enables logarithmic-time lookups via binary search. However, maintaining sorted order requires shifting elements on every insert, yielding linear-time writes.
### 5.1.2 B-Trees: The Traditional Solution
B-trees achieve a balance by maintaining sorted data in a tree structure with high fanout:
$$
T_{write} = O(\log_B n) \quad T_{read} = O(\log_B n)
$$
where $B$ is the branching factor (typically 100-1000). B-trees have dominated database storage for decades due to their balanced read/write performance.
However, B-trees suffer from **write amplification** - the ratio of bytes written to storage versus bytes written by the application:
$$
W_{amp}^{B-tree} = O(\log_B n)
$$
Each update may modify multiple tree nodes from leaf to root, with each node requiring a full page write (typically 4-16 KB) even for small changes.
### 5.1.3 LSM-Trees: Write-Optimized Alternative
Log-Structured Merge-trees (LSM-trees), introduced by O'Neil et al. in 1996, take a different approach: buffer writes in memory, then flush sorted runs to disk, and periodically merge runs to maintain read performance.
**Key insight**: Sequential I/O is 100-1000x faster than random I/O on both HDDs and SSDs. LSM-trees convert random writes to sequential writes at the cost of additional read overhead.
$$
W_{amp}^{LSM} = O\left(\frac{L \cdot T}{B}\right)
$$
where $L$ is the number of levels, $T$ is the size ratio between levels, and $B$ is the block size. With proper tuning, LSM write amplification can be 10-100x lower than B-trees.
## 5.2 LSM-Tree Architecture
An LSM-tree consists of multiple components organized hierarchically from fast volatile memory to slow persistent storage.
### 5.2.1 Component Hierarchy
```mermaid
graph TB
subgraph "Memory (Fast)"
M0[Active Memtable]
M1[Immutable Memtable 1]
M2[Immutable Memtable 2]
WAL[Write-Ahead Log]
end
subgraph "Level 0 (Unsorted)"
L0A[SSTable 0-1]
L0B[SSTable 0-2]
L0C[SSTable 0-3]
end
subgraph "Level 1 (Sorted)"
L1A[SSTable 1-1]
L1B[SSTable 1-2]
L1C[SSTable 1-3]
end
subgraph "Level 2 (Sorted)"
L2A[SSTable 2-1]
L2B[SSTable 2-2]
L2C[SSTable 2-3]
L2D[SSTable 2-4]
end
M0 -->|Flush| L0A
M1 -->|Flush| L0B
M2 -->|Flush| L0C
L0A -->|Compact| L1A
L0B -->|Compact| L1B
L0C -->|Compact| L1C
L1A -->|Compact| L2A
L1B -->|Compact| L2B
L1C -->|Compact| L2C
```
**Memtable**: An in-memory sorted data structure (typically a skip list or red-black tree) that buffers incoming writes. When the memtable reaches a size threshold, it becomes immutable and a new active memtable is created.
**Write-Ahead Log (WAL)**: A persistent append-only log that records every write before it enters the memtable. The WAL ensures durability - if the system crashes before a memtable is flushed, the WAL can replay the writes during recovery.
**Sorted String Table (SSTable)**: An immutable, sorted file containing key-value pairs. SSTables are organized into levels, with each level containing increasingly larger amounts of data.
### 5.2.2 Size Ratio and Level Capacity
The size ratio $T$ determines how much larger each level is compared to the previous level:
$$
\text{Size}(L_i) = T \times \text{Size}(L_{i-1})
$$
For a size ratio $T = 10$ and initial size $S_0$:
| Level | Size | Typical Value |
|-------|------|---------------|
| $L_0$ | $S_0$ | 256 MB |
| $L_1$ | $T \cdot S_0$ | 2.5 GB |
| $L_2$ | $T^2 \cdot S_0$ | 25 GB |
| $L_3$ | $T^3 \cdot S_0$ | 250 GB |
| $L_4$ | $T^4 \cdot S_0$ | 2.5 TB |
The total capacity with $L$ levels is:
$$
\text{Total Capacity} = S_0 \cdot \sum_{i=0}^{L-1} T^i = S_0 \cdot \frac{T^L - 1}{T - 1}
$$
### 5.2.3 Write Amplification Analysis
Write amplification measures how many times data is written to storage over its lifetime. In an LSM-tree, data moves through levels via compaction:
$$
W_{amp} = \frac{\text{Total bytes written to storage}}{\text{Bytes written by application}}
$$
For leveled compaction with size ratio $T$:
$$
W_{amp} = O(T \cdot L) = O\left(T \cdot \log_T \frac{N}{S_0}\right)
$$
where $N$ is total data size. With $T = 10$ and 1 TB of data:
$$
W_{amp} \approx 10 \times 4 = 40
$$
Each byte written by the application results in approximately 40 bytes written to storage over its lifetime.
### 5.2.4 Read Amplification Analysis
Read amplification measures how many storage locations must be checked to satisfy a point query:
$$
R_{amp} = \text{Number of locations checked per query}
$$
In the worst case, a key might exist only in the oldest level, requiring checks at every level. With bloom filters (false positive rate $p$), the expected read amplification is:
$$
R_{amp} = 1 + (L - 1) \cdot p
$$
For $L = 4$ levels and $p = 1\%$:
$$
R_{amp} = 1 + 3 \times 0.01 = 1.03
$$
Bloom filters dramatically reduce read amplification by eliminating unnecessary SSTable searches.
### 5.2.5 Space Amplification
Space amplification measures the ratio of storage used to logical data size:
$$
S_{amp} = \frac{\text{Storage space used}}{\text{Logical data size}}
$$
LSM-trees may temporarily store multiple versions of the same key across levels until compaction merges them. In the worst case:
$$
S_{amp} = 1 + \frac{1}{T}
$$
With $T = 10$, space amplification is bounded by 1.1x (10% overhead).
## 5.3 Cognica's RocksDB Integration
Cognica builds its storage layer on RocksDB, a high-performance LSM-tree implementation originally developed at Facebook. This section examines how Cognica configures and extends RocksDB for unified query processing.
### 5.3.1 Storage Engine Architecture
The storage engine provides a layered abstraction over RocksDB:
```mermaid
graph TB
subgraph "Application Layer"
DOC[Document Store]
KV[Key-Value Store]
FTS[Full-Text Index]
VEC[Vector Index]
end
subgraph "Transaction Layer"
TXN[Transaction Manager]
BATCH[Write Batches]
SNAP[Snapshots]
end
subgraph "Storage Abstraction"
SE[Storage Engine]
KVM[Keyspace Manager]
DB[Database Instances]
end
subgraph "RocksDB Layer"
TDB[TransactionDB]
CF[Column Families]
MEM[Memtables]
SST[SSTables]
end
DOC --> TXN
KV --> TXN
FTS --> TXN
VEC --> TXN
TXN --> SE
BATCH --> SE
SNAP --> SE
SE --> TDB
KVM --> DB
DB --> CF
CF --> MEM
MEM --> SST
```
The storage engine initializes RocksDB with carefully tuned parameters:
**Thread Pool Configuration**:
$$
\text{High Priority Threads} = \max(4, \lfloor \text{cores} / 2 \rfloor)
$$
$$
\text{Low Priority Threads} = \max(2, \lfloor \text{cores} / 4 \rfloor)
$$
High-priority threads handle flushes (converting memtables to SSTables), while low-priority threads handle compaction (merging SSTables across levels).
### 5.3.2 Database Category Hierarchy
Cognica organizes data into three database categories:
| Category | ID | Purpose |
|----------|-----|---------|
| System | 0 | Metadata, configuration, schema definitions |
| KeyValue | 1 | Application key-value data with TTL support |
| Document | 2 | Collections, documents, indexes |
Each category operates as a logical partition within the same RocksDB instance, distinguished by key prefixes:
$$
\text{Key} = \text{CategoryID}(1) \| \text{WorkspaceID}(4) \| \text{CollectionID}(4) \| \text{UserKey}
$$
This prefix scheme enables:
- **Isolation**: Different categories never conflict
- **Efficient Scans**: Prefix iterators scan only relevant data
- **Bloom Filters**: Prefix-based bloom filters accelerate lookups
### 5.3.3 Workspace Multi-Tenancy
Cognica supports multi-tenant deployments where each tenant (workspace) has isolated data:
```mermaid
graph LR
subgraph "Single RocksDB Instance"
subgraph "Workspace A"
A1[Collection 1]
A2[Collection 2]
end
subgraph "Workspace B"
B1[Collection 1]
B2[Collection 2]
end
subgraph "Workspace C"
C1[Collection 1]
end
end
```
The key encoding ensures workspace isolation without requiring separate database instances:
$$
\text{Prefix} = \text{DB}(1) \| \text{Category}(1) \| \text{Workspace}(4) \| \text{Collection}(4) \| \text{Index}(4)
$$
Total prefix length: 14 bytes, enabling efficient prefix extraction for bloom filters.
## 5.4 Write Path Implementation
Understanding the write path is essential for optimizing write-heavy workloads common in document ingestion and full-text indexing.
### 5.4.1 Write Flow
```mermaid
sequenceDiagram
participant App as Application
participant TXN as Transaction
participant WAL as Write-Ahead Log
participant MEM as Memtable
participant FLUSH as Flush Thread
participant SST as SSTable
App->>TXN: Put(key, value)
TXN->>WAL: Append Entry
WAL-->>TXN: Ack (Sync/Async)
TXN->>MEM: Insert
MEM-->>App: Success
Note over MEM: Size > Threshold
MEM->>FLUSH: Trigger Flush
FLUSH->>SST: Write Sorted Data
SST-->>FLUSH: Complete
FLUSH->>WAL: Truncate Old Entries
```
### 5.4.2 Write-Ahead Log Configuration
The WAL ensures durability with configurable trade-offs:
| Parameter | Value | Purpose |
|-----------|-------|---------|
| `max_total_wal_size` | 4 GB | Total WAL size limit across all column families |
| `wal_bytes_per_sync` | 128 MB | Bytes between fsync calls |
| `wal_compression` | ZSTD | Compression algorithm for WAL entries |
| `wal_ttl_seconds` | 3600 | Automatic cleanup after 1 hour |
| `recycle_log_file_num` | 16 | Reuse log files to reduce allocation overhead |
**Durability Modes**:
1. **Synchronous** (`sync = true`): Every write waits for fsync. Maximum durability, highest latency.
2. **Asynchronous** (`sync = false`): Writes return immediately. Risk of losing recent writes on crash.
3. **Group Commit**: Multiple writes share a single fsync, amortizing overhead.
Cognica defaults to asynchronous writes with periodic sync (every 128 MB), balancing throughput and durability.
### 5.4.3 Memtable Configuration
The memtable acts as the write buffer, absorbing writes until flushed:
| Parameter | Value | Impact |
|-----------|-------|--------|
| `write_buffer_size` | 256 MB | Size of each memtable |
| `max_write_buffer_number` | 16 | Maximum concurrent memtables |
| `min_write_buffer_number_to_merge` | 1 | Merge threshold before flush |
| `arena_block_size` | 16 MB | Memory allocation granularity |
**Total Write Buffer Capacity**:
$$
\text{Max Memory} = \text{write\_buffer\_size} \times \text{max\_write\_buffer\_number}
$$
$$
= 256\text{ MB} \times 16 = 4\text{ GB}
$$
This allows up to 4 GB of writes to be buffered in memory, enabling high write throughput for bulk ingestion.
### 5.4.4 Memtable Prefix Bloom Filters
Cognica enables prefix bloom filters on memtables to accelerate point queries before data reaches SSTables:
$$
\text{bloom\_size} = \text{memtable\_size} \times \text{prefix\_bloom\_ratio}
$$
$$
= 256\text{ MB} \times 0.2 = 51.2\text{ MB per memtable}
$$
The bloom filter answers the question "might this key exist in this memtable?" with a configurable false positive rate:
$$
p = \left(1 - e^{-kn/m}\right)^k
$$
where $k$ is the number of hash functions, $n$ is the number of keys, and $m$ is the bloom filter size in bits.
## 5.5 Read Path Implementation
The read path must check multiple locations, making optimization critical for query performance.
### 5.5.1 Read Flow
```mermaid
graph TD
Q[Query: Get Key]
Q --> M0{Active Memtable?}
M0 -->|Found| R[Return Value]
M0 -->|Not Found| M1{Immutable Memtables?}
M1 -->|Found| R
M1 -->|Not Found| L0{Level 0 SSTables?}
L0 -->|Found| R
L0 -->|Not Found| L1{Level 1 SSTables?}
L1 -->|Found| R
L1 -->|Not Found| L2{Level 2+ SSTables?}
L2 -->|Found| R
L2 -->|Not Found| NF[Not Found]
```
### 5.5.2 Block Cache Architecture
Cognica employs a multi-level cache hierarchy to minimize disk I/O:
**Primary Block Cache (HyperClockCache)**:
| Parameter | Value | Purpose |
|-----------|-------|---------|
| `cache_capacity` | 16 GB | Total cache size |
| `cache_shard_bits` | 4 | 16 shards for concurrent access |
| `strict_capacity_limit` | true | Never exceed capacity |
The HyperClockCache uses a clock-based eviction algorithm optimized for high concurrency:
$$
\text{Shards} = 2^{\text{shard\_bits}} = 2^4 = 16
$$
$$
\text{Shard Size} = \frac{\text{Total Capacity}}{\text{Shards}} = \frac{16\text{ GB}}{16} = 1\text{ GB}
$$
**Index and Filter Caching**:
```yaml
cache_index_and_filter_blocks: true
cache_index_and_filter_blocks_with_high_priority: true
```
Index and filter blocks receive high priority in the cache because their eviction causes disproportionate performance degradation - every subsequent read must reload them from disk.
### 5.5.3 Bloom Filter Configuration
Cognica uses Ribbon filters, an advanced alternative to traditional Bloom filters:
**Ribbon Filter Advantages**:
- 20-30% less space than Bloom filters for same false positive rate
- Faster construction for large key sets
- Cache-friendly query pattern
**Configuration**:
$$
\text{Bits per Key} \approx -\frac{\ln(p)}{\ln(2)^2}
$$
For false positive rate $p = 1\%$:
$$
\text{Bits per Key} \approx -\frac{\ln(0.01)}{0.48} \approx 9.6 \text{ bits}
$$
### 5.5.4 Read Options Optimization
Cognica configures read operations for optimal performance:
| Option | Value | Impact |
|--------|-------|--------|
| `auto_prefix_mode` | true | Use prefix extractors for bloom filters |
| `verify_checksums` | false | Skip checksum verification for speed |
| `readahead_size` | 512 KB | Sequential read buffer |
| `adaptive_readahead` | true | Dynamically adjust based on access pattern |
| `async_io` | true | Enable asynchronous I/O |
**Read Ahead Strategy**:
For sequential scans, readahead prefetches data before it is requested:
$$
\text{Effective Bandwidth} = \frac{\text{Block Size}}{\text{Seek Time} + \text{Transfer Time}}
$$
With readahead:
$$
\text{Effective Bandwidth} \approx \frac{\text{Readahead Size}}{\text{Seek Time} + \text{Transfer Time}}
$$
A 512 KB readahead can improve sequential read performance by 10-100x compared to reading individual blocks.
## 5.6 Compaction Strategies
Compaction is the process of merging SSTables to maintain read performance and reclaim space from deleted or overwritten keys. The choice of compaction strategy significantly impacts system behavior.
### 5.6.1 Leveled Compaction
Cognica uses leveled compaction, where each level (except L0) contains non-overlapping SSTables:
**Properties**:
- L0: Overlapping SSTables (direct memtable flushes)
- L1+: Non-overlapping, sorted SSTables
- Size ratio $T = 10$ between levels
**Compaction Trigger**:
When level $L_i$ exceeds its size limit:
$$
\text{Size}(L_i) > T^i \times \text{Target Base Size}
$$
SSTables from $L_i$ are merged with overlapping SSTables in $L_{i+1}$.
```mermaid
graph LR
subgraph "Before Compaction"
L1A[SSTable A: keys 1-100]
L1B[SSTable B: keys 50-150]
L2A[SSTable X: keys 1-50]
L2B[SSTable Y: keys 51-100]
L2C[SSTable Z: keys 101-150]
end
subgraph "After Compaction"
L2A2[SSTable X': keys 1-50]
L2B2[SSTable Y': keys 51-100]
L2C2[SSTable Z': keys 101-150]
end
L1A --> L2A2
L1A --> L2B2
L1B --> L2B2
L1B --> L2C2
```
### 5.6.2 Compression Strategy
Cognica employs a tiered compression strategy optimized for the access patterns at each level:
| Level | Algorithm | Rationale |
|-------|-----------|-----------|
| 0-4 | LZ4 | Fast compression/decompression for hot data |
| 5-6 | ZSTD | High compression ratio for cold data |
**Compression Trade-offs**:
$$
\text{Read Latency} = \text{Disk Read Time} + \text{Decompression Time}
$$
For hot data (L0-L4), LZ4's fast decompression minimizes latency:
- LZ4: ~4 GB/s decompression
- ZSTD: ~1 GB/s decompression
For cold data (L5-L6), ZSTD's superior compression ratio reduces storage:
- LZ4: ~2.5x compression
- ZSTD: ~4x compression
**Dictionary Compression**:
ZSTD supports dictionary compression, where common patterns are pre-computed:
```yaml
max_dict_bytes: 32_KB
zstd_max_train_bytes: 3_MB
```
Dictionary compression can improve ratios by 20-50% for structured data like JSON documents.
### 5.6.3 Custom Compaction Filter
Cognica implements a custom compaction filter that runs during compaction to:
1. **Expire TTL Data**: Remove key-value pairs past their time-to-live
2. **Detect Migration**: Identify data requiring schema migration
3. **Validate Structure**: Ensure keys match expected category and database type
**Filter Decision Logic**:
$$
\text{Decision}(key, value) = \begin{cases}
\text{Remove} & \text{if } \text{TTL}(value) < \text{now} \\
\text{Remove} & \text{if } \text{tombstone}(value) \\
\text{Keep} & \text{otherwise}
\end{cases}
$$
The filter runs during compaction, making TTL expiration essentially "free" - data is cleaned up as part of the normal compaction process without additional I/O.
### 5.6.4 Compaction Tuning
Cognica's compaction configuration balances write amplification, space amplification, and read performance:
| Parameter | Value | Purpose |
|-----------|-------|---------|
| `target_file_size_base` | 64 MB | SSTable size at L1 |
| `max_bytes_for_level_base` | 512 MB | Size limit for L1 |
| `level0_file_num_compaction_trigger` | 4 | L0 files before compaction |
| `level0_slowdown_writes_trigger` | 20 | L0 files before write slowdown |
| `level0_stop_writes_trigger` | 36 | L0 files before write stop |
**Write Stall Prevention**:
When L0 accumulates too many files, writes must slow down to allow compaction to catch up:
$$
\text{Write Rate} = \begin{cases}
\text{Full Speed} & \text{if } |L_0| < 4 \\
\text{Throttled} & \text{if } 4 \leq |L_0| < 20 \\
\text{Severely Throttled} & \text{if } 20 \leq |L_0| < 36 \\
\text{Stopped} & \text{if } |L_0| \geq 36
\end{cases}
$$
## 5.7 Transaction Support
Cognica provides ACID transactions through RocksDB's TransactionDB, with extensions for distributed consensus.
### 5.7.1 Transaction Abstraction
The transaction interface supports multiple implementation strategies:
```mermaid
classDiagram
class Transaction {
<>
+put(key, value)
+get(key) value
+del(key)
+commit()
+rollback()
+set_save_point()
+rollback_to_save_point()
}
class SimpleTransaction {
-rdb_txn: RocksDB Transaction
+pessimistic locking
}
class WriteBatchTransaction {
-batch: WriteBatch
+optimistic batching
}
class IndexedWriteBatchTransaction {
-batch: WriteBatch
-index: htrie_map
+read-your-writes
}
Transaction <|-- SimpleTransaction
Transaction <|-- WriteBatchTransaction
Transaction <|-- IndexedWriteBatchTransaction
```
### 5.7.2 Isolation Levels
**Snapshot Isolation**:
Each transaction sees a consistent snapshot of the database at its start time:
$$
\text{Read}(T, k) = \text{Version}(k, \text{start\_time}(T))
$$
Snapshot isolation prevents dirty reads and non-repeatable reads but allows write skew anomalies.
**Serializable Snapshot Isolation (SSI)**:
RocksDB supports SSI through conflict detection:
$$
\text{Conflict}(T_1, T_2) = \text{ReadSet}(T_1) \cap \text{WriteSet}(T_2) \neq \emptyset
$$
When conflicts are detected, one transaction aborts to maintain serializability.
### 5.7.3 Write Batch Optimization
For write-heavy workloads, Cognica uses indexed write batches that buffer writes in memory:
**Benefits**:
- Read-your-writes: Queries see uncommitted changes within the transaction
- Reduced lock contention: No locks until commit
- Atomic commit: All changes apply atomically
**Index Structure**:
The `tsl::htrie_map` provides efficient prefix-based lookups:
$$
T_{lookup} = O(|key|) \text{ (key length, not number of entries)}
$$
This enables efficient iteration over key ranges within a transaction's write set.
### 5.7.4 Savepoints
Savepoints enable partial rollback within a transaction:
```mermaid
sequenceDiagram
participant App as Application
participant TXN as Transaction
App->>TXN: Begin
App->>TXN: Put(A, 1)
App->>TXN: Set Savepoint S1
App->>TXN: Put(B, 2)
App->>TXN: Set Savepoint S2
App->>TXN: Put(C, 3)
Note over App,TXN: Error detected
App->>TXN: Rollback to S1
Note over TXN: Undo C=3, B=2
App->>TXN: Put(D, 4)
App->>TXN: Commit
Note over TXN: Final: A=1, D=4
```
Savepoints are implemented as markers in the write batch, enabling efficient partial undo.
## 5.8 Custom Extensions
Cognica extends RocksDB with custom components for unified query processing.
### 5.8.1 Custom Comparators
Key ordering determines SSTable organization and iteration behavior. Cognica provides two comparators:
**Ascending Comparator** (default):
$$
\text{Compare}(a, b) = \begin{cases}
-1 & \text{if } a < b \text{ (lexicographically)} \\
0 & \text{if } a = b \\
1 & \text{if } a > b
\end{cases}
$$
**Descending Comparator**:
$$
\text{Compare}_{desc}(a, b) = -\text{Compare}_{asc}(a, b)
$$
The descending comparator enables efficient "ORDER BY DESC" queries by storing data in reverse order.
**Key Compression Optimization**:
The comparators implement `FindShortestSeparator()` to minimize index block size:
Given keys $a$ and $b$ where $a < b$, find the shortest $s$ such that $a \leq s < b$.
Example: For $a = \text{"application"}$ and $b = \text{"apply"}$, $s = \text{"applj"}$.
### 5.8.2 Prefix Extraction
Prefix extractors enable bloom filters and prefix-based iteration:
**Capped Prefix Transform** (14 bytes):
$$
\text{Prefix}(key) = key[0:14]
$$
The 14-byte prefix captures:
- Database type (1 byte)
- Category ID (1 byte)
- Collection ID (4 bytes)
- Index ID (4 bytes)
- Workspace ID (4 bytes)
This enables efficient filtering: "Find all documents in collection X" requires only prefix-matching bloom filter lookups.
### 5.8.3 Merge Operators
Merge operators enable atomic read-modify-write operations without read locks:
**Counter Merge Operator**:
$$
\text{Merge}_{counter}(v_{old}, \Delta) = v_{old} + \Delta
$$
Multiple increments merge during compaction:
$$
\text{Merge}(\text{Merge}(v, \Delta_1), \Delta_2) = v + \Delta_1 + \Delta_2
$$
**Clustered Term Index Merge Operator**:
For full-text search, posting lists must merge efficiently:
$$
\text{Merge}_{posting}(P_1, P_2) = P_1 \cup P_2
$$
The clustered term index stores multiple terms per key, requiring custom merge logic to maintain sorted order and handle deletions.
## 5.9 Backup and Recovery
Cognica provides backup and recovery mechanisms built on RocksDB's backup engine.
### 5.9.1 Backup Architecture
```mermaid
graph TB
subgraph "Live Database"
WAL1[WAL Files]
SST1[SSTables]
MAN1[MANIFEST]
end
subgraph "Backup Storage"
META[Backup Metadata]
SHARED[Shared Files]
PRIV[Private Files]
end
SST1 -->|Hard Link or Copy| SHARED
MAN1 -->|Copy| PRIV
WAL1 -->|Optional| PRIV
META -->|Track| SHARED
META -->|Track| PRIV
```
**Incremental Backups**:
Subsequent backups only copy new SSTables:
$$
\text{Backup Size}_n = \text{New SSTables since Backup}_{n-1}
$$
For append-heavy workloads, incremental backups are dramatically smaller than full backups.
### 5.9.2 Point-in-Time Recovery
The backup engine maintains multiple backup versions:
| Backup ID | Timestamp | Files | Size |
|-----------|-----------|-------|------|
| 1 | 2024-01-01 | 100 | 10 GB |
| 2 | 2024-01-02 | 15 | 1.5 GB |
| 3 | 2024-01-03 | 20 | 2 GB |
Recovery restores to any backup point:
$$
\text{Restore}(backup\_id) \rightarrow \text{Database state at backup time}
$$
### 5.9.3 Encryption Support
Cognica supports encryption at rest through RocksDB's encrypted environment:
**Encryption Flow**:
$$
\text{Ciphertext} = E_{key}(\text{Plaintext})
$$
$$
\text{Plaintext} = D_{key}(\text{Ciphertext})
$$
Backups preserve encryption, requiring the same key for restoration.
## 5.10 Performance Characteristics
This section summarizes the performance characteristics of Cognica's LSM-tree storage.
### 5.10.1 Throughput Bounds
**Write Throughput**:
$$
\text{Max Write Throughput} = \min\left(\frac{\text{Memtable Size}}{\text{Flush Time}}, \frac{\text{Disk Bandwidth}}{\text{Write Amp}}\right)
$$
With 256 MB memtables, 100 ms flush time, 500 MB/s disk, and 40x write amp:
$$
\text{Max Write} = \min(2.56 \text{ GB/s}, 12.5 \text{ MB/s}) \approx 12.5 \text{ MB/s sustained}
$$
**Read Throughput**:
$$
\text{Max Read Throughput} = \text{Cache Hit Rate} \times \text{Memory Bandwidth} + (1 - \text{Cache Hit Rate}) \times \text{Disk Bandwidth}
$$
With 99% cache hit rate, 100 GB/s memory, 500 MB/s disk:
$$
\text{Max Read} = 0.99 \times 100 + 0.01 \times 0.5 \approx 99 \text{ GB/s}
$$
### 5.10.2 Latency Distribution
Point query latency depends on data location:
| Location | Latency | Probability |
|----------|---------|-------------|
| Block Cache | 1-10 us | 99% (with good caching) |
| Memtable | 10-100 us | Depends on recency |
| L0 SSTables | 100 us - 1 ms | Low (bloom filters) |
| L1+ SSTables | 1-10 ms | Very low (bloom filters) |
**P99 Latency**:
$$
P_{99} \approx \text{Disk Read Latency} \times (1 - \text{Bloom Filter Effectiveness})
$$
With 1% bloom filter false positive rate and 1 ms disk latency:
$$
P_{99} \approx 1\text{ ms} \times 0.01 = 10 \text{ us}
$$
### 5.10.3 Space Efficiency
**Effective Compression Ratio**:
$$
\text{Compression Ratio} = \frac{\text{Logical Data Size}}{\text{Physical Storage}}
$$
With tiered compression (LZ4 for hot, ZSTD for cold):
$$
\text{Effective Ratio} \approx 0.3 \times 2.5 + 0.7 \times 4.0 = 3.55x
$$
Assuming 30% of data is hot (recent) and 70% is cold (historical).
## 5.11 Summary
This chapter examined the LSM-tree storage architecture underlying Cognica's persistence layer. Key takeaways:
1. **LSM-trees optimize for write throughput** by converting random writes to sequential I/O through the memtable/SSTable hierarchy.
2. **Write amplification** is the primary cost, with leveled compaction yielding $O(T \cdot L)$ amplification. Cognica tunes this through compression tiers and careful level sizing.
3. **Read amplification** is controlled through bloom filters (Ribbon filters in Cognica), achieving near-optimal single-read performance for point queries.
4. **The multi-level cache hierarchy** (block cache, row cache, OS page cache) minimizes disk I/O for hot data.
5. **Transaction support** through RocksDB's TransactionDB provides ACID guarantees with configurable isolation levels.
6. **Custom extensions** (comparators, merge operators, compaction filters) adapt the generic LSM-tree for unified query processing.
The storage layer provides the foundation upon which Cognica builds document storage, full-text indexes, and vector indexes - topics we explore in the following chapters.
# Chapter 6: Document Storage and Schema Management
This chapter examines Cognica's document storage layer, which provides flexible schema management atop the LSM-tree foundation established in Chapter 5. We explore how JSON documents are encoded for efficient storage, how schemas define structure and constraints, and how indexes accelerate queries across diverse access patterns.
## 6.1 The Document Model
Document databases emerged from the recognition that many applications work with semi-structured data that doesn't fit neatly into relational tables. Rather than forcing data into rigid schemas, document databases store self-describing records that can vary in structure.
### 6.1.1 JSON as Universal Data Format
Cognica adopts JSON (JavaScript Object Notation) as its document format. JSON provides:
**Simplicity**: Human-readable syntax with just six data types:
- Objects (key-value maps)
- Arrays (ordered sequences)
- Strings
- Numbers
- Booleans
- Null
**Universality**: Native support in every programming language, HTTP APIs, and configuration systems.
**Nestability**: Documents can contain nested documents and arrays to arbitrary depth.
**Example Document**:
```json
{
"_id": "user_12345",
"name": "Alice Chen",
"email": "alice@example.com",
"profile": {
"bio": "Database enthusiast",
"location": {
"city": "San Francisco",
"country": "USA"
}
},
"tags": ["developer", "researcher"],
"created_at": "2024-01-15T10:30:00Z"
}
```
### 6.1.2 Document vs Relational Trade-offs
The document model trades normalization for locality:
**Relational Model**:
$$
\text{User} \xrightarrow{\text{JOIN}} \text{Profile} \xrightarrow{\text{JOIN}} \text{Location} \xrightarrow{\text{JOIN}} \text{Tags}
$$
Data is normalized across multiple tables, eliminating redundancy but requiring joins for reconstruction.
**Document Model**:
$$
\text{User Document} = \text{User} \cup \text{Profile} \cup \text{Location} \cup \text{Tags}
$$
Related data is embedded within a single document, enabling single-read retrieval at the cost of potential redundancy.
**Access Pattern Optimization**:
| Pattern | Relational | Document |
|---------|------------|----------|
| Read user with profile | 3+ JOINs | 1 read |
| Update user's city | 1 update | Read-modify-write |
| Find users in city | Index scan | Index scan |
| Aggregate across users | Efficient | Efficient |
Documents excel when data is read together more often than updated independently.
### 6.1.3 RapidJSON Integration
Cognica uses RapidJSON, a high-performance JSON library, as its in-memory document representation:
```cpp
class Document : public rapidjson::GenericDocument<
rapidjson::UTF8<>,
DocumentAllocator
> {
// Extended with Cognica-specific operations
};
```
**Performance Characteristics**:
| Operation | Complexity | Notes |
|-----------|------------|-------|
| Parse JSON string | $O(n)$ | Single pass, in-situ possible |
| Access field by name | $O(m)$ | Linear scan, $m$ = object size |
| Access array element | $O(1)$ | Direct index |
| Iterate all fields | $O(m)$ | Sequential scan |
| Serialize to string | $O(n)$ | Single pass |
RapidJSON's DOM (Document Object Model) representation stores parsed JSON in memory, enabling random access and modification.
### 6.1.4 Custom Allocator
Cognica employs a custom memory allocator for document operations:
**Benefits**:
- **Pool allocation**: Reduces malloc/free overhead
- **Arena semantics**: Bulk deallocation when document is destroyed
- **Cache locality**: Related allocations are contiguous
**Allocation Strategy**:
$$
\text{Block Size} = \max(\text{requested}, \text{arena\_block\_size})
$$
Small allocations come from the current arena block; large allocations get dedicated blocks.
## 6.2 Document Encoding
Storing JSON documents directly would be inefficient. Cognica encodes documents into a compact binary format optimized for storage and retrieval.
### 6.2.1 Type Encoding
Each value is prefixed with a type marker:
| Type | Code | Description |
|------|------|-------------|
| Object | `0x01` | Nested document |
| Array | `0x02` | Ordered sequence |
| Null | `0x03` | Null value |
| False | `0x04` | Boolean false |
| True | `0x05` | Boolean true |
| Int64 | `0x06` | 64-bit signed integer |
| UInt64 | `0x07` | 64-bit unsigned integer |
| Double | `0x08` | IEEE 754 double |
| String | `0x09` | UTF-8 string |
**Type-Length-Value (TLV) Encoding**:
$$
\text{Encoded Value} = \text{Type}(1) \| \text{Length}(var) \| \text{Data}(length)
$$
Variable-length encoding uses continuation bits to minimize space for small values:
$$
\text{Encoded Length} = \begin{cases}
1 \text{ byte} & \text{if } length < 128 \\
2 \text{ bytes} & \text{if } length < 16384 \\
\vdots & \vdots
\end{cases}
$$
### 6.2.2 Primitive Encoding
**Integers**:
Signed integers use sign-flip encoding to preserve sort order:
$$
\text{encode}(n) = \begin{cases}
n \oplus \text{0x8000000000000000} & \text{if } n \geq 0 \\
n \oplus \text{0xFFFFFFFFFFFFFFFF} & \text{if } n < 0
\end{cases}
$$
This transforms the two's complement representation so that:
$$
\text{encode}(-1) < \text{encode}(0) < \text{encode}(1)
$$
Lexicographic comparison of encoded bytes yields correct numeric ordering.
**Floating-Point Numbers**:
IEEE 754 doubles require special handling for sortable encoding:
$$
\text{encode}(d) = \begin{cases}
\text{bits}(d) \oplus \text{0x8000000000000000} & \text{if } d \geq 0 \\
\text{bits}(d) \oplus \text{0xFFFFFFFFFFFFFFFF} & \text{if } d < 0
\end{cases}
$$
where $\text{bits}(d)$ interprets the 64-bit IEEE 754 representation as an unsigned integer.
**Strings**:
Strings are encoded with length prefix followed by UTF-8 bytes:
$$
\text{Encoded String} = \text{VarInt}(length) \| \text{UTF8 bytes}
$$
For key comparison, null-terminated encoding is used:
$$
\text{Key String} = \text{UTF8 bytes} \| \text{0x00}
$$
### 6.2.3 Composite Encoding
**Objects**:
Objects encode as sequences of key-value pairs:
$$
\text{Object} = \text{0x01} \| \text{VarInt}(count) \| \text{KV}_1 \| \text{KV}_2 \| ... \| \text{KV}_n
$$
Each key-value pair:
$$
\text{KV} = \text{VarInt}(key\_len) \| \text{key} \| \text{encoded value}
$$
**Arrays**:
Arrays encode as sequences of values:
$$
\text{Array} = \text{0x02} \| \text{VarInt}(count) \| \text{Value}_1 \| \text{Value}_2 \| ... \| \text{Value}_n
$$
### 6.2.4 Document Layout
Complete documents include a header with metadata:
```mermaid
graph LR
subgraph "Document Record"
H[Header]
V[Value Data]
end
subgraph "Header Structure"
TS[Timestamp 8B]
TTL[TTL 4B]
FLAGS[Flags 1B]
end
H --> TS
H --> TTL
H --> FLAGS
```
**Header Fields**:
| Field | Size | Purpose |
|-------|------|---------|
| Timestamp | 8 bytes | Creation/modification time |
| TTL | 4 bytes | Time-to-live in seconds (0 = never expires) |
| Flags | 1 byte | Metadata flags (deleted, migrating, etc.) |
**Space Efficiency**:
Consider encoding the example user document:
| Component | JSON Size | Encoded Size |
|-----------|-----------|--------------|
| Field names | 89 bytes | 89 bytes |
| String values | 78 bytes | 82 bytes |
| Structural overhead | 45 bytes | 15 bytes |
| **Total** | **212 bytes** | **186 bytes** |
Binary encoding typically achieves 10-30% size reduction through eliminated whitespace and compact length encoding.
## 6.3 Schema Definition
While documents can vary in structure, schemas define expectations and constraints that enable optimization and validation.
### 6.3.1 Schema Structure
A Cognica schema specifies:
```yaml
collection: users
workspace: default
primary_key:
fields: [_id]
unique: true
secondary_keys:
- name: email_idx
fields: [email]
unique: true
type: secondary_key
- name: location_idx
fields: [profile.location.country, profile.location.city]
type: secondary_key
- name: content_idx
fields: [profile.bio]
type: full_text_search
comment: "User accounts with profile information"
```
### 6.3.2 Schema Components
**Primary Key**:
Every collection has exactly one primary key that uniquely identifies documents:
$$
\text{PK}: \mathcal{D} \rightarrow \mathcal{K}
$$
The primary key maps each document to a unique key value. Primary keys can be:
- **Single field**: `_id`
- **Composite**: `(tenant_id, user_id)`
- **Auto-generated**: UUID or sequence
**Secondary Keys**:
Secondary keys create additional access paths:
$$
\text{SK}: \mathcal{D} \rightarrow 2^{\mathcal{K}}
$$
Unlike primary keys, secondary keys can map to sets (for non-unique indexes) and support:
- **B-tree indexes**: For range queries and sorting
- **Full-text indexes**: For text search
- **Clustered indexes**: Storing document data with the index
### 6.3.3 Schema Builder Pattern
Schemas are constructed programmatically using the builder pattern:
```cpp
auto schema = SchemaBuilder{}
.set_workspace_id(workspace_id)
.set_collection_id(collection_id)
.set_collection_name("users")
.set_primary_key({"_id"}, PrimaryKeyOptions{.unique = true})
.add_secondary_key("email_idx", {"email"}, SecondaryKeyOptions{
.unique = true,
.type = IndexType::kSecondaryKey
})
.add_secondary_key("content_idx", {"profile.bio"}, SecondaryKeyOptions{
.type = IndexType::kFullTextSearchIndex
})
.set_comment("User accounts")
.build();
```
The builder validates constraints during construction:
- Primary key must have at least one field
- Secondary key names must be unique
- Field paths must be valid dot notation
### 6.3.4 Schema Flexibility
Cognica supports **schema-on-read** semantics: documents can contain fields not defined in the schema. The schema defines:
1. **Indexed fields**: Fields with associated indexes
2. **Type hints**: Expected types for validation
3. **Constraints**: Uniqueness, nullability
Documents may include additional fields that are stored but not indexed. This enables gradual schema evolution without migration.
## 6.4 Key Encoding
Keys must be encoded to preserve ordering in the LSM-tree while supporting composite keys and nullable fields.
### 6.4.1 Primary Key Encoding
Primary keys are encoded with a prefix identifying the collection:
$$
\text{PK Storage Key} = \text{Prefix}(14) \| \text{Encoded PK Fields}
$$
**Prefix Structure**:
| Component | Bytes | Purpose |
|-----------|-------|---------|
| Database Type | 1 | Distinguishes document DB from others |
| Category | 1 | Data category (user data = 2) |
| Workspace ID | 4 | Multi-tenant isolation |
| Collection ID | 4 | Collection identification |
| Index ID | 4 | Primary key index (always 0) |
**Field Encoding**:
For composite primary keys `(field_1, field_2, ...)`:
$$
\text{Encoded PK} = \text{enc}(field_1) \| \text{enc}(field_2) \| ...
$$
Each field is encoded with its type-specific encoding, ensuring lexicographic order matches logical order.
### 6.4.2 Secondary Key Encoding
Secondary keys include both the secondary key fields and the primary key (for uniqueness):
$$
\text{SK Storage Key} = \text{Prefix}(14) \| \text{Encoded SK Fields} \| \text{Encoded PK}
$$
**Example**:
For index `location_idx` on `(country, city)` with primary key `_id`:
```
Key: [prefix][country][city][_id]
[14 bytes][var][var][var]
```
This encoding enables:
- **Prefix scans**: Find all users in a country
- **Range scans**: Find users in countries A-M
- **Exact lookup**: Find user with specific country+city+id
### 6.4.3 Nullable Field Handling
Nullable fields require special encoding to maintain sort order:
$$
\text{enc}_{nullable}(v) = \begin{cases}
\text{0x00} & \text{if } v = \text{null} \\
\text{0x01} \| \text{enc}(v) & \text{otherwise}
\end{cases}
$$
Null values sort before all non-null values (or after, depending on configuration).
### 6.4.4 Sort Order Preservation
The encoding must satisfy:
$$
v_1 < v_2 \implies \text{enc}(v_1) <_{lex} \text{enc}(v_2)
$$
where $<_{lex}$ is lexicographic (byte-wise) comparison.
**Descending Order**:
For descending sorts, the encoding is inverted:
$$
\text{enc}_{desc}(v) = \text{complement}(\text{enc}_{asc}(v))
$$
where complement flips all bits. This reverses the sort order while maintaining the comparison-by-bytes property.
## 6.5 Index Architecture
Indexes are the primary mechanism for accelerating queries. Cognica supports multiple index types optimized for different access patterns.
### 6.5.1 Index Type Hierarchy
```mermaid
classDiagram
class Index {
<>
+get_guid()
+get_index_type()
+get_fields()
+is_unique()
+compute_affinity_score()
}
class PrimaryKey {
+fields: FieldNames
+always unique
}
class SecondaryKey {
+name: string
+fields: FieldNames
+unique: bool
+type: IndexType
}
Index <|-- PrimaryKey
Index <|-- SecondaryKey
```
### 6.5.2 Index Types
| Type | Code | Use Case |
|------|------|----------|
| Primary Key | 0 | Unique document identification |
| Secondary Key | 1 | Traditional B-tree index |
| Clustered Secondary | 2 | Secondary index with embedded data |
| Full-Text Search | 3 | Text search with posting lists |
| Clustered FTS | 4 | FTS with embedded document data |
**Primary Key Index**:
The primary key index stores complete documents:
$$
\text{Key} = \text{PK} \quad \text{Value} = \text{Encoded Document}
$$
**Secondary Key Index**:
Secondary indexes store only the mapping:
$$
\text{Key} = \text{SK} \| \text{PK} \quad \text{Value} = \text{TTL Metadata}
$$
Lookups require two steps:
1. Find PK via secondary index
2. Fetch document via primary key
**Clustered Secondary Index**:
Clustered secondaries embed document data:
$$
\text{Key} = \text{SK} \| \text{PK} \quad \text{Value} = \text{Encoded Document}
$$
This eliminates the second lookup at the cost of storage duplication.
### 6.5.3 Index Descriptor
The `IndexDescriptor` manages all indexes for a collection:
```cpp
class IndexDescriptor {
PrimaryKey primary_key_;
std::vector secondary_keys_;
mutable std::shared_mutex mutex_;
// Operations
auto get_primary_key() const -> const PrimaryKey&;
auto get_secondary_key(IndexID id) const -> const SecondaryKey*;
auto find_by_name(std::string_view name) const -> const SecondaryKey*;
void add_secondary_key(SecondaryKey&& sk);
void remove_secondary_key(IndexID id);
};
```
**Thread Safety**:
The descriptor uses a shared mutex for concurrent access:
- Multiple readers can access concurrently
- Writers acquire exclusive access
- Index additions/removals are atomic
### 6.5.4 Index Statistics
Each index tracks usage statistics for query optimization:
```cpp
struct IndexStatistics {
std::atomic accessed; // Query count
std::atomic added; // Insert count
std::atomic updated; // Update count
std::atomic deleted; // Delete count
std::atomic merged; // Merge operation count
TimePoint accessed_at; // Last query time
TimePoint added_at; // Last insert time
TimePoint updated_at; // Last update time
TimePoint deleted_at; // Last delete time
TimePoint merged_at; // Last merge time
};
```
Statistics inform:
- **Index selection**: Prefer frequently-used indexes
- **Maintenance scheduling**: Identify cold indexes for optimization
- **Capacity planning**: Track growth rates
## 6.6 Collection Operations
Collections are the primary interface for document manipulation, providing ACID operations through the transaction layer.
### 6.6.1 Collection Architecture
```mermaid
graph TB
subgraph "Collection Interface"
C[Collection]
end
subgraph "Context"
CTX[CollectionContext]
SCH[Schema]
PKR[PK Reader]
PKW[PK Writer]
SKR[SK Readers]
SKW[SK Writers]
end
subgraph "Transaction Layer"
TXN[Transaction Manager]
BATCH[Write Batches]
end
subgraph "Storage Layer"
RDB[RocksDB]
end
C --> CTX
CTX --> SCH
CTX --> PKR
CTX --> PKW
CTX --> SKR
CTX --> SKW
PKR --> TXN
PKW --> TXN
SKR --> TXN
SKW --> TXN
TXN --> RDB
BATCH --> RDB
```
### 6.6.2 CRUD Operations
**Insert**:
```cpp
Status Collection::insert(const Document& doc) {
// 1. Extract primary key
auto pk = extract_primary_key(doc);
// 2. Check uniqueness
if (pk_reader_->exists(pk)) {
return Status::AlreadyExists("Duplicate primary key");
}
// 3. Encode document
auto encoded = encode_document(doc);
// 4. Write to primary index
pk_writer_->put(pk, encoded);
// 5. Update secondary indexes
for (auto& sk_writer : sk_writers_) {
auto sk = extract_secondary_key(doc, sk_writer->descriptor());
sk_writer->put(sk, pk);
}
return Status::OK();
}
```
**Find**:
```cpp
Cursor Collection::find(const Document& query) {
// 1. Analyze query
auto plan = query_planner_.plan(query);
// 2. Select best index
auto index = plan.best_index();
// 3. Create cursor
if (index.is_primary_key()) {
return pk_reader_->scan(plan.key_range());
} else {
return sk_readers_[index.id()]->scan(plan.key_range());
}
}
```
**Update**:
```cpp
Status Collection::update(const Document& filter, const Document& updates) {
// 1. Find matching documents
auto cursor = find(filter);
// 2. Apply updates
while (cursor.valid()) {
auto doc = cursor.document();
// 3. Apply update operators
apply_updates(doc, updates);
// 4. Rewrite document
auto pk = extract_primary_key(doc);
pk_writer_->put(pk, encode_document(doc));
// 5. Update secondary indexes if affected fields changed
update_secondary_indexes(old_doc, doc);
cursor.next();
}
return Status::OK();
}
```
**Delete**:
```cpp
Status Collection::remove(const Document& filter) {
auto cursor = find(filter);
while (cursor.valid()) {
auto doc = cursor.document();
auto pk = extract_primary_key(doc);
// 1. Delete from primary index
pk_writer_->del(pk);
// 2. Delete from secondary indexes
for (auto& sk_writer : sk_writers_) {
auto sk = extract_secondary_key(doc, sk_writer->descriptor());
sk_writer->del(sk, pk);
}
cursor.next();
}
return Status::OK();
}
```
### 6.6.3 Batch Operations
For bulk inserts, batch operations amortize overhead:
```cpp
Status Collection::insert_parallel(const std::vector& docs) {
// 1. Partition documents across threads
auto partitions = partition(docs, thread_count_);
// 2. Process partitions in parallel
parallel_for(partitions, [this](auto& partition) {
auto batch = begin_write_batch();
for (auto& doc : partition) {
batch.insert(doc);
}
batch.commit();
});
return Status::OK();
}
```
**Performance Characteristics**:
| Operation | Single | Batch (1000 docs) |
|-----------|--------|-------------------|
| Insert | 100 us | 50 ms (50 us/doc) |
| Index update | 50 us | 25 ms (25 us/doc) |
| Total | 150 us | 75 ms |
| **Throughput** | 6,600/s | 13,300/s |
Batching doubles throughput by amortizing transaction overhead.
### 6.6.4 Transaction Support
Collections support ACID transactions:
```cpp
auto txn = collection.begin_transaction();
try {
txn.insert(doc1);
txn.update(filter, updates);
txn.remove(filter2);
txn.commit();
} catch (...) {
txn.rollback();
}
```
**Isolation Levels**:
| Level | Dirty Read | Non-Repeatable | Phantom |
|-------|------------|----------------|---------|
| Read Uncommitted | Yes | Yes | Yes |
| Read Committed | No | Yes | Yes |
| Repeatable Read | No | No | Yes |
| Serializable | No | No | No |
Cognica defaults to **Snapshot Isolation**, which prevents dirty reads and non-repeatable reads while allowing phantoms in some cases.
## 6.7 Index Reader and Writer
The index reader/writer abstraction separates query and mutation operations.
### 6.7.1 Index Reader Interface
```cpp
class IndexReader {
public:
// Point lookup
virtual auto get(const Slice& key) -> std::optional = 0;
// Existence check
virtual auto exists(const Slice& key) -> bool = 0;
// Range scan
virtual auto scan(const KeyRange& range) -> Cursor = 0;
// Prefix scan
virtual auto scan_prefix(const Slice& prefix) -> Cursor = 0;
// Count
virtual auto count(const KeyRange& range) -> size_t = 0;
};
```
### 6.7.2 Index Writer Interface
```cpp
class IndexWriter {
public:
// Insert
virtual auto put(const Slice& key, const Slice& value) -> Status = 0;
// Delete
virtual auto del(const Slice& key) -> Status = 0;
// Batch operations
virtual auto put_batch(const std::vector& kvs) -> Status = 0;
virtual auto del_batch(const std::vector& keys) -> Status = 0;
};
```
### 6.7.3 Key Codec
The key codec handles encoding and decoding of index keys:
**Primary Key Codec**:
```cpp
struct PrimaryKeyIndexKeyCodec {
static auto encode(
const PrimaryKey& pk_desc,
const Slice& pk
) -> std::string {
std::string key;
// Add 14-byte prefix
append_prefix(key, pk_desc.guid());
// Add encoded primary key fields
key.append(pk.data(), pk.size());
return key;
}
static auto decode(
const PrimaryKey& pk_desc,
const Slice& storage_key
) -> Slice {
// Skip 14-byte prefix
return storage_key.substr(14);
}
};
```
**Secondary Key Codec**:
```cpp
struct SecondaryKeyIndexKeyCodec {
static auto encode(
const PrimaryKey& pk_desc,
const SecondaryKey& sk_desc,
const Slice& pk,
const Document& doc,
bool nullable
) -> std::string {
std::string key;
// Add 14-byte prefix with SK index ID
append_prefix(key, sk_desc.guid());
// Add encoded secondary key fields
for (const auto& field : sk_desc.fields()) {
auto value = doc.find(field);
encode_field(key, value, nullable);
}
// Append primary key for uniqueness
key.append(pk.data(), pk.size());
return key;
}
};
```
### 6.7.4 Index Affinity Score
The query optimizer uses affinity scores to select the best index:
$$
\text{Affinity}(Q, I) = \sum_{f \in \text{fields}(Q) \cap \text{fields}(I)} w(f, I)
$$
where $w(f, I)$ is the weight of field $f$ in index $I$ (higher for earlier positions).
**Scoring Algorithm**:
```cpp
double Index::compute_affinity_score(const FieldNames& query_fields) const {
double score = 0.0;
size_t position = 0;
for (const auto& field : fields_) {
if (query_fields.contains(field)) {
// Higher weight for earlier positions (prefix selectivity)
score += 1.0 / (position + 1);
} else {
// Gap in index prefix reduces usefulness
break;
}
position++;
}
return score;
}
```
## 6.8 Dot Notation and Nested Documents
Cognica supports dot notation for accessing nested fields, enabling queries and indexes on deeply nested data.
### 6.8.1 Path Syntax
Dot notation uses periods to separate nested field names:
| Path | Meaning |
|------|---------|
| `name` | Top-level field |
| `profile.bio` | Nested field |
| `profile.location.city` | Deeply nested field |
| `tags[0]` | Array element |
| `tags[*]` | All array elements |
### 6.8.2 Path Resolution
```cpp
class DotNotationSupport {
public:
// Find nested member
auto find_member(const Document& doc, std::string_view path)
-> std::optional;
// Add nested member (creating intermediate objects)
auto add_member(Document& doc, std::string_view path, Value value)
-> Status;
// Check existence
auto has_member(const Document& doc, std::string_view path)
-> bool;
// Remove nested member
auto remove_member(Document& doc, std::string_view path)
-> Status;
};
```
**Resolution Algorithm**:
```
find_member(doc, "profile.location.city"):
1. Split path: ["profile", "location", "city"]
2. current = doc
3. For each segment:
- If current is object and has segment:
current = current[segment]
- Else: return null
4. Return current
```
### 6.8.3 Nested Index Creation
Indexes on nested fields work identically to top-level fields:
```yaml
secondary_keys:
- name: city_idx
fields: [profile.location.city]
type: secondary_key
```
The index stores the nested value directly, enabling efficient lookups:
```sql
SELECT * FROM users WHERE profile.location.city = 'San Francisco'
```
Uses `city_idx` for O(log n) lookup rather than O(n) full scan.
### 6.8.4 Array Handling
Arrays require special handling for indexing:
**Multi-Key Index**:
For a document with array field:
```json
{"_id": "1", "tags": ["developer", "researcher"]}
```
A multi-key index creates entries for each array element:
$$
\text{Index Entries} = \{(\text{"developer"}, \text{"1"}), (\text{"researcher"}, \text{"1"})\}
$$
**Query Semantics**:
```sql
SELECT * FROM users WHERE tags = 'developer'
```
Matches any document where `tags` contains "developer".
## 6.9 Catalog Management
The catalog stores metadata about collections, indexes, and schemas.
### 6.9.1 Catalog Structure
```mermaid
graph TB
subgraph "Catalog"
WS[Workspaces]
COL[Collections]
IDX[Indexes]
STAT[Statistics]
end
subgraph "Workspace Metadata"
WS --> W1[Workspace 1]
WS --> W2[Workspace 2]
end
subgraph "Collection Metadata"
W1 --> C1[users]
W1 --> C2[products]
W2 --> C3[events]
end
subgraph "Index Metadata"
C1 --> I1[_pk]
C1 --> I2[email_idx]
C1 --> I3[content_idx]
end
```
### 6.9.2 Catalog Operations
| Operation | Description |
|-----------|-------------|
| `create_collection` | Register new collection with schema |
| `drop_collection` | Remove collection and all data |
| `get_collection` | Retrieve collection metadata |
| `list_collections` | Enumerate workspace collections |
| `create_index` | Add secondary index |
| `drop_index` | Remove secondary index |
| `get_index` | Retrieve index metadata |
### 6.9.3 Schema Versioning
Schemas evolve over time. Cognica tracks schema versions:
$$
\text{Schema}_{v+1} = \text{migrate}(\text{Schema}_v, \text{changes})
$$
**Compatible Changes** (no migration needed):
- Adding nullable fields
- Adding secondary indexes
- Adding new collections
**Incompatible Changes** (require migration):
- Changing primary key fields
- Changing field types
- Removing required fields
### 6.9.4 Metadata Persistence
Catalog metadata is stored in the system database category:
$$
\text{Key} = \text{0x00} \| \text{type} \| \text{workspace\_id} \| \text{collection\_id}
$$
| Type | Purpose |
|------|---------|
| `0x01` | Collection schema |
| `0x02` | Index descriptor |
| `0x03` | Statistics |
| `0x04` | Access control |
## 6.10 Query Context and Projection
Query context carries execution state through the query pipeline.
### 6.10.1 Query Context Structure
```cpp
struct QueryContext {
// Execution mode
bool is_single_document;
bool is_streaming;
// Field projection
FieldProjectMap projection;
// Transaction state
Transaction* transaction;
Snapshot* snapshot;
// Statistics
QueryStatistics stats;
};
```
### 6.10.2 Field Projection
Projections limit which fields are returned, reducing I/O and network transfer:
```sql
SELECT name, email FROM users WHERE status = 'active'
```
**Projection Encoding**:
```cpp
struct FieldProjectMap {
enum Mode { kInclude, kExclude };
Mode mode;
std::unordered_set fields;
bool should_include(std::string_view field) const {
bool in_set = fields.contains(field);
return (mode == kInclude) ? in_set : !in_set;
}
};
```
**Projection Optimization**:
For queries touching only indexed fields, the query can be answered from the index alone (covering index):
$$
\text{Covering} \iff \text{projected fields} \subseteq \text{index fields}
$$
Covering queries avoid the primary key lookup entirely.
### 6.10.3 Query Statistics
Each query collects execution statistics:
```cpp
struct QueryStatistics {
size_t documents_scanned;
size_t documents_returned;
size_t index_keys_examined;
size_t bytes_read;
Duration parse_time;
Duration plan_time;
Duration execution_time;
std::string selected_index;
};
```
Statistics enable:
- **Query debugging**: Identify slow queries
- **Index tuning**: Find missing indexes
- **Capacity planning**: Predict resource usage
## 6.11 Summary
This chapter explored Cognica's document storage layer, from JSON representation through binary encoding to index management. Key takeaways:
1. **JSON documents** provide flexible schema with nested structure, encoded efficiently in binary format for storage.
2. **Key encoding** preserves sort order for composite keys, enabling efficient range scans in the LSM-tree.
3. **Multiple index types** (primary, secondary, full-text, clustered) optimize for different access patterns.
4. **Schema management** balances flexibility (schema-on-read) with optimization (indexed fields, constraints).
5. **Collection operations** provide ACID guarantees through the transaction layer, with batch optimization for bulk workloads.
6. **Dot notation** enables seamless access to nested fields, with multi-key indexes for arrays.
7. **Catalog management** tracks metadata with support for schema evolution.
The document layer provides the structured data interface that applications interact with, while the next chapter explores how full-text search indexes enable efficient text queries across document collections.
# Chapter 7: Inverted Index Architecture
This chapter explores the inverted index, the fundamental data structure enabling full-text search. We examine how Cognica implements posting lists, the revolutionary clustered term index optimization that reduces key counts by 62,500x, and the text analysis pipeline that transforms documents into searchable terms.
## 7.1 Information Retrieval Fundamentals
Full-text search differs fundamentally from structured queries. Rather than matching exact values, text search must handle:
- **Vocabulary mismatch**: Users search "car" but documents contain "automobile"
- **Ranking**: Multiple documents match; which is most relevant?
- **Linguistic variation**: "running", "runs", "ran" should match "run"
- **Scale**: Billions of documents, millions of unique terms
### 7.1.1 The Term-Document Matrix
The conceptual foundation of text retrieval is the term-document matrix:
$$
M_{t \times d} = \begin{bmatrix}
m_{1,1} & m_{1,2} & \cdots & m_{1,d} \\
m_{2,1} & m_{2,2} & \cdots & m_{2,d} \\
\vdots & \vdots & \ddots & \vdots \\
m_{t,1} & m_{t,2} & \cdots & m_{t,d}
\end{bmatrix}
$$
where $m_{i,j} = 1$ if term $i$ appears in document $j$, else $0$.
**Problem**: For $t = 1,000,000$ terms and $d = 10,000,000$ documents, this matrix has $10^{13}$ entries - far too large to store.
**Observation**: The matrix is extremely sparse. A typical document contains 100-1000 unique terms out of millions possible. Sparsity is often 99.99%+.
### 7.1.2 Inverted Index as Sparse Representation
The inverted index stores only non-zero entries, organized by term:
$$
\text{InvertedIndex} = \{t_i \mapsto \{d_j \mid m_{i,j} = 1\}\}
$$
Each term maps to a **posting list** - the set of documents containing that term.
**Example**:
| Term | Posting List |
|------|--------------|
| "database" | {1, 5, 12, 47, 103, ...} |
| "query" | {1, 12, 89, 156, ...} |
| "optimization" | {5, 47, 89, 201, ...} |
**Query Processing**:
To find documents containing "database AND query":
$$
\text{Result} = \text{Postings}(\text{"database"}) \cap \text{Postings}(\text{"query"}) = \{1, 12, ...\}
$$
Set intersection replaces matrix multiplication, achieving dramatic speedup.
### 7.1.3 Posting List Complexity
| Operation | Full Matrix | Inverted Index |
|-----------|-------------|----------------|
| Storage | $O(t \times d)$ | $O(\sum_i \lvert P_i \rvert)$ |
| Term lookup | $O(d)$ | $O(1)$ dictionary + $O(\lvert P \rvert)$ |
| AND query | $O(t \times d)$ | $O(\min(\lvert P_1 \rvert, \lvert P_2 \rvert))$ |
| OR query | $O(t \times d)$ | $O(\lvert P_1 \rvert + \lvert P_2 \rvert)$ |
The inverted index transforms text search from infeasible to practical.
## 7.2 Posting List Structure
Cognica's posting lists store rich metadata beyond simple document identifiers, enabling advanced ranking and phrase queries.
### 7.2.1 Posting Entry Components
Each posting entry contains:
```cpp
struct PostingEntry {
DocID doc_id; // Document identifier
std::string pointer; // External document reference
int32_t term_freq; // Occurrences in document
int32_t term_count; // Total terms in field
float field_norm; // Length normalization factor
Positions positions; // Term positions for phrases
Offsets offsets; // Byte offsets for highlighting
};
```
**Field Descriptions**:
| Field | Purpose | Used For |
|-------|---------|----------|
| `doc_id` | Internal identifier | Index lookups |
| `pointer` | External key (e.g., primary key) | Result retrieval |
| `term_freq` | Count in this document | TF-IDF, BM25 scoring |
| `term_count` | Total tokens in field | Length normalization |
| `field_norm` | Pre-computed $1/\sqrt{length}$ | BM25 $b$ parameter |
| `positions` | Token positions [0, 5, 12, ...] | Phrase queries |
| `offsets` | Character/byte ranges | Snippet highlighting |
### 7.2.2 Positions for Phrase Queries
Positions enable phrase matching by recording where each term appears:
**Document**: "The quick brown fox jumps over the lazy dog"
| Term | Positions |
|------|-----------|
| "the" | [0, 6] |
| "quick" | [1] |
| "brown" | [2] |
| "fox" | [3] |
| "jumps" | [4] |
| "over" | [5] |
| "lazy" | [7] |
| "dog" | [8] |
**Phrase Query**: "quick brown fox"
Check if positions are consecutive:
- "quick" at position 1
- "brown" at position 2 (= 1 + 1)
- "fox" at position 3 (= 2 + 1)
Match confirmed - all positions are adjacent.
### 7.2.3 Offsets for Highlighting
Offsets enable precise text highlighting without re-tokenizing:
```cpp
struct Offset {
uint32_t begin; // Start position (byte or char)
uint32_t end; // End position
uint32_t size; // Length (for validation)
};
```
**Example** for "quick" in "The quick brown fox":
| Type | Begin | End |
|------|-------|-----|
| Character | 4 | 9 |
| Byte (UTF-8) | 4 | 9 |
For multi-byte characters, byte and character offsets differ, requiring both for correct highlighting in UTF-8 text.
### 7.2.4 Aggregate Posting List
The complete posting list aggregates all entries for a term:
```cpp
struct Postings {
std::vector doc_ids;
std::vector pointers;
std::vector term_freqs;
std::vector term_counts;
std::vector field_norms;
std::vector positions;
std::vector offsets_bytes;
};
```
This structure-of-arrays layout optimizes cache utilization when scanning only specific fields (e.g., doc_ids for boolean queries, term_freqs for scoring).
## 7.3 The Clustered Term Index
Traditional inverted indexes suffer from key explosion in LSM-tree storage. Cognica's clustered term index addresses this through document clustering, achieving 62,500x key reduction.
### 7.3.1 The Key Explosion Problem
In a traditional inverted index stored in an LSM-tree:
$$
\text{Key} = \text{term} \| \text{doc\_id}
$$
For a corpus with:
- 10 million documents
- 1000 unique terms per document
- Average 100 matching documents per query term
A single query might require:
$$
\text{Keys Accessed} = \text{query terms} \times \text{docs per term} = 5 \times 100,000 = 500,000
$$
Each key requires a separate RocksDB lookup, overwhelming the storage layer.
### 7.3.2 Clustering Solution
The clustered term index groups documents into clusters of 65,536 ($2^{16}$):
$$
\text{cluster\_id}(doc\_id) = \lfloor doc\_id / 65536 \rfloor
$$
$$
\text{offset}(doc\_id) = doc\_id \mod 65536
$$
All posting entries within a cluster are stored in a single key-value pair:
$$
\text{Key} = \text{term} \| \text{cluster\_id}
$$
$$
\text{Value} = \{(\text{offset}_1, \text{entry}_1), (\text{offset}_2, \text{entry}_2), ...\}
$$
### 7.3.3 Key Reduction Analysis
**Traditional Index**:
$$
\text{Keys} = \sum_{t \in \text{terms}} |\text{Postings}(t)|
$$
For 100,000 postings per term across 5 query terms: 500,000 keys.
**Clustered Index**:
$$
\text{Keys} = \sum_{t \in \text{terms}} \lceil |\text{Postings}(t)| / 65536 \rceil
$$
For 100,000 postings spread across clusters: ~8 keys per term, 40 keys total.
**Reduction Factor**:
$$
\text{Reduction} = \frac{500,000}{40} = 12,500\text{x}
$$
In practice, with locality (documents indexed together have similar doc_ids), reduction reaches 62,500x.
### 7.3.4 Clustered Key Format
The key encoding uses a three-layer structure:
```mermaid
graph LR
subgraph Layer1["Layer 1: Index Prefix"]
L1A["collection_id
4 bytes"] --> L1B["index_id
4 bytes"] --> L1C["kClusteredTermIndex
1 byte"]
end
subgraph Layer2["Layer 2: Term Prefix"]
L2A["field
var"] --> L2B["0x00
1 byte"] --> L2C["value
var"] --> L2D["0x00
1 byte"]
end
subgraph Layer3["Layer 3: Cluster Key"]
L3A["cluster_id
4 bytes"]
end
Layer1 ~~~ Layer2 ~~~ Layer3
```
This hierarchical structure enables:
- **Prefix iteration**: Scan all clusters for a term
- **Direct lookup**: Jump to specific cluster
- **Range scans**: Efficient cluster range queries
### 7.3.5 Clustered Value Encoding
Each cluster value contains multiple posting entries:
```mermaid
graph TD
VM["Value Metadata"]
FV["Format Version (1 byte: version 1)"]
EC["Entry Count (varint)"]
E1["Entry 1
- offset_delta (varint, gap-encoded)
- term_freq (varint)
- term_count (varint)
- field_norm (float32)
- positions (gap-encoded array)
- offsets (gap-encoded array)"]
E2["Entry 2
..."]
VM --> FV --> EC --> E1 --> E2
```
**Gap Encoding for Offsets**:
Offsets within a cluster are stored as deltas from previous:
$$
\text{encoded}_i = \text{offset}_i - \text{offset}_{i-1}
$$
Since offsets are sorted, deltas are small positive integers that compress well with variable-length encoding.
### 7.3.6 Performance Comparison
| Metric | Traditional | Clustered | Improvement |
|--------|-------------|-----------|-------------|
| Keys per query | 500,000 | 8 | 62,500x |
| Read latency | 45 ms | 13 ms | 3.5x |
| Write latency | 120 ms | 35 ms | 3.4x |
| Storage overhead | 1.0x | 1.05x | ~5% increase |
The storage overhead comes from cluster metadata, but the dramatic reduction in key operations more than compensates.
## 7.4 Posting List Encoding
Efficient encoding minimizes storage and I/O while enabling fast decoding.
### 7.4.1 Variable-Length Integer Encoding
Small integers are common in posting lists. Variable-length encoding uses fewer bytes for smaller values:
$$
\text{bytes}(n) = \lceil \log_{128}(n + 1) \rceil
$$
| Value Range | Bytes |
|-------------|-------|
| 0 - 127 | 1 |
| 128 - 16,383 | 2 |
| 16,384 - 2,097,151 | 3 |
**Encoding Algorithm**:
```
encode_varint(n):
while n >= 128:
emit(n & 0x7F | 0x80) // Low 7 bits + continuation flag
n >>= 7
emit(n) // Final byte (no continuation)
```
### 7.4.2 Gap Encoding for Positions
Positions are strictly increasing, so deltas (gaps) are always positive and typically small:
**Original positions**: [0, 5, 12, 18, 25, 33]
**Gap-encoded**: [0, 5, 7, 6, 7, 8]
$$
\text{gap}_i = \text{pos}_i - \text{pos}_{i-1}
$$
**Compression Analysis**:
| Encoding | Bytes for example |
|----------|-------------------|
| Raw int32 | 24 bytes |
| Gap + varint | 6 bytes |
| **Compression** | **4x** |
### 7.4.3 Offset Encoding
Offsets encode as (begin_delta, length) pairs:
```cpp
void encode_offsets(const Offsets& offsets) {
encode_varint(offsets.size());
uint32_t prev_end = 0;
for (const auto& offset : offsets) {
encode_varint(offset.begin - prev_end); // Gap from previous end
encode_varint(offset.end - offset.begin); // Length
encode_varint(offset.size); // Redundant but useful
prev_end = offset.end;
}
}
```
This exploits the fact that tokens are typically adjacent or near-adjacent in text.
### 7.4.4 Complete Entry Encoding
A full posting entry encodes as:
```mermaid
graph TD
F1["offset (varint)
Gap from previous entry in cluster"]
F2["term_freq (varint)
Occurrence count"]
F3["term_count (varint)
Field length"]
F4["field_norm (float32)
Pre-computed normalization"]
F5["Positions
- positions_count
- pos_gap_1 (first position)
- pos_gap_2 (delta from pos_1)
- ..."]
F6["Offsets
- offsets_count
- off_begin_1 (first offset begin)
- off_len_1 (first offset length)
- off_size_1 (first offset size)
- off_begin_delta_2 (delta from off_end_1)
- ..."]
F1 --> F2 --> F3 --> F4 --> F5 --> F6
```
**Typical Compression**:
For a posting entry with 3 positions and 3 offsets:
| Component | Raw Size | Encoded Size |
|-----------|----------|--------------|
| Metadata | 16 bytes | 6 bytes |
| Positions | 12 bytes | 4 bytes |
| Offsets | 36 bytes | 10 bytes |
| **Total** | **64 bytes** | **20 bytes** |
Overall compression: **3.2x**
## 7.5 Text Analysis Pipeline
The analysis pipeline transforms raw text into indexable terms through a series of transformations.
### 7.5.1 Pipeline Architecture
```mermaid
graph LR
subgraph "Input"
RAW[Raw Text]
end
subgraph "Character Filtering"
CF[Char Filters]
end
subgraph "Tokenization"
TOK[Tokenizer]
end
subgraph "Token Filtering"
TF1[Lowercase]
TF2[Stop Words]
TF3[Stemming]
TF4[N-grams]
end
subgraph "Output"
TERMS[Terms]
end
RAW --> CF
CF --> TOK
TOK --> TF1
TF1 --> TF2
TF2 --> TF3
TF3 --> TF4
TF4 --> TERMS
```
### 7.5.2 Character Filters
Character filters preprocess text before tokenization:
**HTML Stripping**:
$$
\text{"Hello
"} \rightarrow \text{"Hello"}
$$
**Unicode Normalization** (NFC/NFD):
$$
\text{"cafe\u0301"} \rightarrow \text{"cafe"} \quad \text{(combining accent removed)}
$$
**Pattern Replacement**:
$$
\text{"user@example.com"} \rightarrow \text{"user AT example DOT com"}
$$
### 7.5.3 Tokenizers
Tokenizers split text into individual tokens:
**Standard Tokenizer** (Unicode word boundaries):
```
"The quick-brown fox" -> ["The", "quick", "brown", "fox"]
```
**Whitespace Tokenizer** (simple splitting):
```
"The quick-brown fox" -> ["The", "quick-brown", "fox"]
```
**N-gram Tokenizer** (character n-grams):
```
"fox" with n=2 -> ["fo", "ox"]
```
**Path Hierarchy Tokenizer** (file paths):
```
"/usr/local/bin" -> ["/usr", "/usr/local", "/usr/local/bin"]
```
**ICU Tokenizer** (internationalization):
Handles CJK languages, Thai, and other scripts without whitespace word boundaries.
### 7.5.4 Token Structure
Each token carries metadata:
```cpp
struct Token {
TokenType type; // String, numeric, date, vector
std::string value; // Processed token text
std::string original; // Pre-normalization text
int32_t position; // Position in field
Offset offset_chars; // Character boundaries
Offset offset_bytes; // Byte boundaries (UTF-8)
};
```
The original value enables highlighting even after normalization transforms the text.
### 7.5.5 Token Filters
Token filters transform individual tokens:
**Lowercase Filter**:
$$
\text{"Database"} \rightarrow \text{"database"}
$$
**ASCII Folding** (accent removal):
$$
\text{"cafe"} \rightarrow \text{"cafe"}
$$
**Stop Word Filter** (remove common words):
$$
\text{["the", "quick", "fox"]} \rightarrow \text{["quick", "fox"]}
$$
**Snowball Stemmer** (reduce to root form):
$$
\text{"running"} \rightarrow \text{"run"}
$$
$$
\text{"databases"} \rightarrow \text{"databas"}
$$
**Edge N-gram Filter** (prefix indexing):
$$
\text{"database"} \rightarrow \text{["d", "da", "dat", "data", ...]}
$$
**Double Metaphone** (phonetic matching):
$$
\text{"Smith"} \rightarrow \text{["SM0", "XMT"]}
$$
### 7.5.6 Analyzer Composition
Analyzers combine components into reusable pipelines:
**Standard Analyzer**:
```yaml
analyzer:
type: standard
char_filters: []
tokenizer: standard
token_filters:
- lowercase
- stop_words
```
**Custom Analyzer**:
```yaml
analyzer:
type: custom
char_filters:
- html_strip
tokenizer: standard
token_filters:
- lowercase
- ascii_folding
- snowball:
language: english
```
### 7.5.7 Index vs Query Analysis
Different analyzers can be used at index and query time:
**Index Time**: Full normalization
$$
\text{"Running"} \xrightarrow{\text{index}} \text{"run"}
$$
**Query Time**: Match index normalization
$$
\text{"runs"} \xrightarrow{\text{query}} \text{"run"}
$$
The same stemmer ensures "runs" matches documents containing "running".
## 7.6 Document ID Management
The DocID index maintains bidirectional mappings between external document references and internal numeric identifiers.
### 7.6.1 DocID Structure
```cpp
using DocID = uint64_t;
constexpr DocID kInvalidDocID = std::numeric_limits::max();
```
DocIDs are 64-bit unsigned integers, supporting up to $2^{64}$ documents per index.
### 7.6.2 DocID Index Operations
```cpp
class DocIDIndex {
public:
// Forward mapping: pointer -> DocID
auto get_doc_id(const std::string& pointer) -> std::optional;
// Reverse mapping: DocID -> pointer
auto get_pointer(DocID doc_id) -> std::optional;
// Allocation
auto allocate(const std::string& pointer) -> DocID;
// Deletion
void remove(DocID doc_id);
void remove(const std::string& pointer);
};
```
### 7.6.3 Cluster Assignment
DocIDs are assigned sequentially within each collection, ensuring documents indexed together share clusters:
$$
\text{DocID}_n = \text{DocID}_{n-1} + 1
$$
This locality maximizes cluster efficiency - a batch insert of 100,000 documents uses only:
$$
\text{Clusters} = \lceil 100,000 / 65,536 \rceil = 2 \text{ clusters}
$$
### 7.6.4 DocID Reuse
Deleted DocIDs can be reused to prevent unbounded growth:
```cpp
class DocIDGenerator {
std::queue free_list_;
DocID next_id_;
public:
DocID allocate() {
if (!free_list_.empty()) {
auto id = free_list_.front();
free_list_.pop();
return id;
}
return next_id_++;
}
void release(DocID id) {
free_list_.push(id);
}
};
```
## 7.7 Index Statistics
Statistics enable cost-based query optimization and relevance scoring.
### 7.7.1 Collection Statistics
```cpp
enum class IndexStatsType {
kDocCount, // Total documents in collection
kDocSize, // Total bytes of documents
kFieldCount, // Documents containing field
kFieldSize, // Bytes in field across all docs
kFieldTermCount, // Posting entries for field
kFieldTermSize, // Bytes of postings for field
kTermCount, // Unique terms
kTermSize, // Bytes of term dictionary
kTokenCount, // Total token occurrences
kTokenSize, // Bytes of tokens
};
```
### 7.7.2 Term Statistics
```cpp
struct TermStatistics {
int64_t doc_freq; // Documents containing term
int64_t term_freq; // Total occurrences across collection
};
```
**Document Frequency** (df): Used for IDF calculation in TF-IDF and BM25.
$$
\text{IDF}(t) = \log\left(\frac{N - df(t) + 0.5}{df(t) + 0.5} + 1\right)
$$
**Term Frequency** (tf): Used for collection-wide statistics.
### 7.7.3 Statistics Storage
Statistics are stored in a separate namespace with atomic counters:
```cpp
class ShardedInt64CounterMap {
static constexpr size_t kShardCount = 64;
std::array>, kShardCount> shards_;
public:
void increment(const Key& key, int64_t delta) {
auto& shard = shards_[hash(key) % kShardCount];
shard[key].fetch_add(delta, std::memory_order_relaxed);
}
int64_t get(const Key& key) const {
auto& shard = shards_[hash(key) % kShardCount];
auto it = shard.find(key);
return it != shard.end() ? it->second.load() : 0;
}
};
```
Sharding prevents contention during concurrent indexing.
### 7.7.4 Average Field Length
BM25 requires average field length for normalization:
$$
\text{avgdl} = \frac{\text{total tokens in field}}{\text{documents with field}}
$$
Computed incrementally:
$$
\text{avgdl}_{new} = \frac{\text{avgdl}_{old} \times n + \text{new doc length}}{n + 1}
$$
## 7.8 Fast Update Architecture
The pending list system batches index updates to reduce write amplification.
### 7.8.1 Write Amplification Problem
Without batching, each term generates a separate RocksDB merge operation:
$$
\text{Merges per document} = \text{unique terms in document}
$$
For a document with 500 unique terms: 500 merge operations.
### 7.8.2 Pending List Solution
The pending list accumulates updates in memory before flushing:
```mermaid
graph TB
subgraph "Write Path"
DOC[Document]
ANALYZE[Analyze]
PENDING[Pending List]
FLUSH[Flush]
CLUSTERED[Clustered Index]
end
DOC --> ANALYZE
ANALYZE --> PENDING
PENDING -->|Threshold reached| FLUSH
FLUSH --> CLUSTERED
```
### 7.8.3 Pending Entry Format
```cpp
struct PendingEntry {
enum Operation { kAdd, kRemove };
Operation op;
std::string field;
std::string term_value;
ClusteredPostingEntry entry;
};
```
Entries are grouped by (field, term_value) before flushing.
### 7.8.4 Flush Trigger Conditions
```cpp
struct FTSPendingConfig {
bool fast_update = true;
size_t batch_size_threshold = 64 * 1024; // 64 KB
size_t pending_list_limit = 16 * 1024 * 1024; // 16 MB
size_t pending_entry_limit = 500'000;
size_t cleanup_batch_size = 100'000;
};
```
Flush occurs when:
- Batch size exceeds 64 KB
- Total pending data exceeds 16 MB
- Entry count exceeds 500,000
- Explicit flush requested
### 7.8.5 Background Cleanup
A background process merges pending entries into the main index:
```
cleanup():
1. Scan pending batches in order
2. Decompress and decode entries
3. Group by (field, term_value)
4. For each group:
- Merge add/remove operations
- Issue single merge to clustered index
5. Delete processed pending keys
6. Update statistics
```
### 7.8.6 Write Amplification Reduction
| Scenario | Without Batching | With Batching |
|----------|------------------|---------------|
| 1000 docs, 500 terms each | 500,000 merges | ~1,000 merges |
| Reduction | 1x | **500x** |
The pending list dramatically reduces write amplification for bulk indexing.
## 7.9 Posting List Iteration
Efficient iteration over posting lists is critical for query performance.
### 7.9.1 Iterator Interface
```cpp
class DocIDSetIterator {
public:
// Current position
virtual DocID doc_id() const = 0;
// Validity check
virtual bool is_valid() const = 0;
virtual bool has_next() const = 0;
// Movement
virtual DocID next() = 0;
virtual DocID advance(DocID target) = 0;
// Cost estimation
virtual int64_t cost() const = 0;
};
```
### 7.9.2 Advance Operation
The `advance(target)` operation skips to the first DocID >= target:
$$
\text{advance}(P, target) = \min\{d \in P \mid d \geq target\}
$$
This enables efficient intersection without scanning all entries:
```
intersect(P1, P2):
result = []
while P1.valid() and P2.valid():
if P1.doc_id() == P2.doc_id():
result.append(P1.doc_id())
P1.next()
P2.next()
elif P1.doc_id() < P2.doc_id():
P1.advance(P2.doc_id())
else:
P2.advance(P1.doc_id())
return result
```
### 7.9.3 Specialized Iterators
**Conjunction DISI** (AND queries):
Maintains multiple iterators, advancing the minimum:
```cpp
class ConjunctionDISI : public DocIDSetIterator {
std::vector> iterators_;
DocID advance(DocID target) override {
while (true) {
// Advance all iterators to target
for (auto& it : iterators_) {
target = it->advance(target);
}
// Check if all converged
if (all_equal(iterators_)) {
return target;
}
// Set new target to maximum
target = max_doc_id(iterators_);
}
}
};
```
**Phrase Match DISI**:
Extends conjunction with position checking:
```cpp
class PhraseMatchDISI : public DocIDSetIterator {
bool check_phrase_positions() {
// Load positions for all terms at current doc
// Check if positions are consecutive
for (size_t i = 1; i < positions_.size(); i++) {
if (!has_adjacent_position(positions_[i-1], positions_[i], i)) {
return false;
}
}
return true;
}
};
```
### 7.9.4 Lazy Position Loading
Positions are loaded only when needed:
```cpp
class LazyPostingEntry {
bool positions_loaded_ = false;
Positions positions_;
const Positions& get_positions() {
if (!positions_loaded_) {
positions_ = decode_positions(raw_data_);
positions_loaded_ = true;
}
return positions_;
}
};
```
This optimization avoids decoding positions for documents that fail earlier filters.
## 7.10 Merge Operator
The clustered term index uses RocksDB merge operators for efficient concurrent updates.
### 7.10.1 Merge vs Put
**Put semantics**: Overwrites previous value
$$
\text{Put}(k, v_2) \text{ after } \text{Put}(k, v_1) \Rightarrow \text{Read}(k) = v_2
$$
**Merge semantics**: Combines with previous value
$$
\text{Merge}(k, \Delta_2) \text{ after } \text{Merge}(k, \Delta_1) \Rightarrow \text{Read}(k) = f(v_0, \Delta_1, \Delta_2)
$$
### 7.10.2 Clustered Term Index Merge
For posting lists, merge combines entries from multiple documents:
$$
\text{Merge}(P_1, P_2) = P_1 \cup P_2
$$
With deduplication for the same DocID:
$$
\text{Merge}(e_1, e_2) = \begin{cases}
e_2 & \text{if } e_1.\text{doc\_id} = e_2.\text{doc\_id} \\
\{e_1, e_2\} & \text{otherwise}
\end{cases}
$$
### 7.10.3 Merge Implementation
```cpp
class ClusteredTermIndexMergeOperator : public MergeOperator {
bool FullMerge(
const Slice& key,
const Slice* existing_value,
const std::deque& operands,
std::string* new_value
) override {
// Start with existing entries
auto entries = existing_value
? decode_cluster(*existing_value)
: ClusterEntries{};
// Apply each operand
for (const auto& operand : operands) {
auto delta = decode_delta(operand);
apply_delta(entries, delta);
}
// Encode result
*new_value = encode_cluster(entries);
return true;
}
};
```
### 7.10.4 Partial Merge Optimization
Partial merges combine operands without reading existing value:
```cpp
bool PartialMerge(
const Slice& key,
const Slice& left_operand,
const Slice& right_operand,
std::string* new_value
) override {
// Combine two deltas without reading base value
auto left = decode_delta(left_operand);
auto right = decode_delta(right_operand);
*new_value = encode_delta(merge_deltas(left, right));
return true;
}
```
This reduces read amplification during compaction.
## 7.11 Integration with Document Store
The FTS index integrates with the document store through a unified context.
### 7.11.1 FTS Context
```cpp
class FTSContext {
std::unique_ptr doc_id_index_;
std::unique_ptr term_index_;
std::unique_ptr clustered_term_index_;
std::unique_ptr facet_index_;
std::unique_ptr numeric_index_;
std::unique_ptr query_cache_;
AnalyzerMap analyzer_map_; // Per-field analyzers
SimilarityMap similarity_map_; // Scoring models
FieldOptionsMap field_options_; // Index configuration
};
```
### 7.11.2 Document Indexing Flow
```mermaid
sequenceDiagram
participant App as Application
participant IW as InvertedIndexWriter
participant ANA as Analyzer
participant DID as DocIDIndex
participant CTI as ClusteredTermIndex
App->>IW: insert(document)
IW->>DID: allocate(pointer)
DID-->>IW: doc_id
IW->>ANA: analyze(document)
ANA-->>IW: terms with positions
loop For each field
IW->>CTI: add(term, doc_id, entry)
end
CTI-->>IW: success
IW-->>App: success
```
### 7.11.3 Field Options
Each field can have custom indexing options:
```cpp
struct FieldOptions {
std::string analyzer; // Analyzer name
bool index_positions = true; // Store positions
bool index_offsets = true; // Store offsets
bool store_term_vectors = false;// Store per-doc term stats
float boost = 1.0f; // Field-level boost
};
```
### 7.11.4 Multi-Field Documents
Documents can have multiple searchable fields:
```json
{
"_id": "doc1",
"title": "Database Systems",
"body": "An introduction to database internals...",
"tags": ["database", "systems"]
}
```
Each field is indexed separately with its own analyzer:
| Field | Analyzer | Indexed Terms |
|-------|----------|---------------|
| title | standard | ["database", "systems"] |
| body | english | ["introduct", "databas", "intern", ...] |
| tags | keyword | ["database", "systems"] |
## 7.12 Summary
This chapter explored the inverted index architecture that powers Cognica's full-text search capabilities. Key takeaways:
1. **Inverted indexes** transform the sparse term-document matrix into an efficient structure where each term maps to a posting list of containing documents.
2. **Posting lists** store rich metadata including term frequencies, positions, and offsets, enabling both boolean matching and relevance ranking.
3. **The clustered term index** achieves 62,500x key reduction by grouping documents into clusters of 65,536, storing all postings for a cluster in a single key-value pair.
4. **Gap encoding** and variable-length integers compress posting lists by 3-4x while maintaining fast decoding.
5. **The text analysis pipeline** transforms raw text through character filters, tokenizers, and token filters into normalized, searchable terms.
6. **The pending list architecture** batches index updates to reduce write amplification by up to 500x during bulk indexing.
7. **Merge operators** enable efficient concurrent updates without read-before-write overhead.
8. **Iterator abstraction** with advance operations enables efficient query execution through posting list intersection.
The inverted index provides the foundation for text search; the next part of this book explores query processing, where we'll see how SQL queries are parsed, optimized, and executed against this index structure.
# Chapter 8: SQL Parser and Semantic Analysis
This chapter examines how Cognica transforms SQL text into executable query plans. We explore the parsing infrastructure built on PostgreSQL's battle-tested parser, the Abstract Syntax Tree representation, and the semantic analysis phase that validates and enriches queries with type information.
## 8.1 The SQL Processing Pipeline
SQL is a declarative language - users specify *what* data they want, not *how* to retrieve it. The database must bridge this gap through multiple transformation stages.
### 8.1.1 Pipeline Overview
```mermaid
graph TB
subgraph "Frontend"
SQL[SQL Text]
LEX[Lexical Analysis]
PARSE[Parsing]
AST[Abstract Syntax Tree]
SEM[Semantic Analysis]
end
subgraph "Middle-End"
LOG[Logical Planning]
OPT[Optimization]
PHYS[Physical Planning]
end
subgraph "Backend"
CVM[CVM Bytecode]
EXEC[Execution]
end
SQL --> LEX
LEX --> PARSE
PARSE --> AST
AST --> SEM
SEM --> LOG
LOG --> OPT
OPT --> PHYS
PHYS --> CVM
CVM --> EXEC
```
Each stage transforms the query into a more concrete representation:
| Stage | Input | Output | Purpose |
|-------|-------|--------|---------|
| Lexical Analysis | SQL text | Tokens | Break into words |
| Parsing | Tokens | Parse Tree | Recognize grammar |
| AST Construction | Parse Tree | AST | Structured representation |
| Semantic Analysis | AST | Annotated AST | Type checking, name resolution |
| Logical Planning | Annotated AST | Logical Plan | Relational operators |
| Optimization | Logical Plan | Optimized Plan | Cost-based transformations |
| Physical Planning | Optimized Plan | Physical Plan | Execution strategies |
| Code Generation | Physical Plan | Bytecode | Executable instructions |
### 8.1.2 Why Multiple Stages?
Separation of concerns enables:
**Modularity**: Each stage can be tested and optimized independently.
**Reusability**: The same AST serves multiple backends (CVM, Volcano iterator).
**Extensibility**: New optimizations can be added without modifying parsing.
**Debugging**: Problems can be isolated to specific stages.
## 8.2 Lexical Analysis and Parsing
Cognica leverages libpg_query, a standalone extraction of PostgreSQL's parser, providing full PostgreSQL 17 compatibility.
### 8.2.1 Why PostgreSQL's Parser?
PostgreSQL's SQL parser is:
- **Battle-tested**: 25+ years of production use
- **Standards-compliant**: ANSI SQL with PostgreSQL extensions
- **Feature-rich**: Supports CTEs, window functions, JSON operators, arrays
- **Well-maintained**: Regular updates with PostgreSQL releases
Building a custom parser would require years of effort to match this maturity.
### 8.2.2 libpg_query Integration
The `ParseResult` class wraps libpg_query with RAII semantics:
```cpp
class ParseResult {
public:
// Parse SQL string, returns result with parse tree or error
static auto parse(std::string_view sql) -> ParseResult;
// Access parse tree as JSON
auto get_parse_tree() const -> std::string_view;
// Error information
auto has_error() const -> bool;
auto get_error_message() const -> std::string_view;
auto get_error_line() const -> int32_t;
auto get_error_column() const -> int32_t;
private:
PgQueryParseResult result_; // libpg_query C struct
};
```
**Memory Safety**: The destructor automatically frees libpg_query's internal allocations.
### 8.2.3 Parse Tree Format
libpg_query produces a JSON representation of PostgreSQL's internal parse tree:
```sql
SELECT id, name FROM users WHERE status = 'active'
```
Produces (simplified):
```json
{
"stmts": [{
"stmt": {
"SelectStmt": {
"targetList": [
{"ResTarget": {"val": {"ColumnRef": {"fields": [{"String": {"sval": "id"}}]}}}},
{"ResTarget": {"val": {"ColumnRef": {"fields": [{"String": {"sval": "name"}}]}}}}
],
"fromClause": [
{"RangeVar": {"relname": "users"}}
],
"whereClause": {
"A_Expr": {
"kind": "AEXPR_OP",
"name": [{"String": {"sval": "="}}],
"lexpr": {"ColumnRef": {"fields": [{"String": {"sval": "status"}}]}},
"rexpr": {"A_Const": {"sval": {"sval": "active"}}}
}
}
}
}
}]
}
```
### 8.2.4 Additional libpg_query Features
**Query Fingerprinting**:
```cpp
class FingerprintResult {
static auto fingerprint(std::string_view sql) -> FingerprintResult;
auto get_fingerprint() const -> uint64_t; // Hash for cache key
};
```
Fingerprints normalize queries for caching:
```sql
SELECT * FROM users WHERE id = 1
SELECT * FROM users WHERE id = 2
```
Both produce the same fingerprint, enabling prepared statement caching.
**Tokenization**:
```cpp
class ScanResult {
static auto scan(std::string_view sql) -> ScanResult;
auto get_tokens() const -> std::vector;
};
```
Useful for syntax highlighting and query analysis tools.
**Deparsing**:
```cpp
class DeparseResult {
static auto deparse(const ProtobufParseResult& tree) -> DeparseResult;
auto get_sql() const -> std::string_view;
};
```
Converts parse trees back to normalized SQL, useful for query logging.
## 8.3 Abstract Syntax Tree
The AST provides a type-safe, navigable representation of SQL queries.
### 8.3.1 Node Hierarchy
```mermaid
classDiagram
class Node {
<>
+NodeType type
+get_type()
}
class Expr {
<>
}
class Stmt {
<>
}
Node <|-- Expr
Node <|-- Stmt
Expr <|-- ColumnRef
Expr <|-- Constant
Expr <|-- BinaryExpr
Expr <|-- UnaryExpr
Expr <|-- FunctionCall
Expr <|-- CaseExpr
Expr <|-- SubqueryExpr
Expr <|-- CastExpr
Stmt <|-- SelectStmt
Stmt <|-- InsertStmt
Stmt <|-- UpdateStmt
Stmt <|-- DeleteStmt
Stmt <|-- CreateTableStmt
```
**Node**: Base class with type discriminator.
**Expr**: Expressions that produce values.
**Stmt**: Complete SQL statements.
### 8.3.2 Node Type Enumeration
Cognica supports 56+ node types:
```cpp
enum class NodeType : uint8_t {
// Statements
kSelectStmt, kInsertStmt, kUpdateStmt, kDeleteStmt,
kCreateStmt, kDropStmt, kAlterStmt, kIndexStmt,
kTruncateStmt, kRenameStmt, kTransactionStmt,
kExplainStmt, kAnalyzeStmt, kCopyStmt,
kViewStmt, kCreateMatViewStmt, kRefreshMatViewStmt,
kCreateFunctionStmt, kDropFunctionStmt,
kCreateTriggerStmt, kDropTriggerStmt,
kGrantStmt, kRevokeStmt, kCreateRoleStmt,
kCreatePolicyStmt, kAlterPolicyStmt,
// Expressions
kColumnRef, kConstant, kParamRef,
kBinaryExpr, kUnaryExpr, kFunctionCall,
kCaseExpr, kCastExpr, kSubqueryExpr,
kArrayExpr, kArraySubscript, kArraySlice,
kCoalesceExpr, kNullIfExpr, kTypedLiteral,
// Other
kJoinExpr, kRangeVar, kTargetEntry, kOrderByItem,
kWindowDef, kGroupingSet, kCommonTableExpr,
// ...
};
```
### 8.3.3 Expression Nodes
**ColumnRef** - Table column references:
```cpp
class ColumnRef : public Expr {
std::optional schema_name; // Optional schema
std::optional table_name; // Optional table/alias
std::string column_name; // Column name
};
```
Examples:
- `id` -> `{column_name: "id"}`
- `users.id` -> `{table_name: "users", column_name: "id"}`
- `public.users.id` -> `{schema_name: "public", table_name: "users", column_name: "id"}`
**Constant** - Literal values:
```cpp
class Constant : public Expr {
enum class Type { kNull, kBool, kInt64, kDouble, kString, kArray };
Type value_type;
std::variant<
std::monostate, // NULL
bool, // Boolean
int64_t, // Integer
double, // Floating point
std::string, // String
std::vector // Array
> value;
};
```
**BinaryExpr** - Binary operators:
```cpp
class BinaryExpr : public Expr {
BinaryOpType op;
ExprPtr left;
ExprPtr right;
};
```
With 50+ operator types:
```cpp
enum class BinaryOpType {
// Arithmetic
kAdd, kSubtract, kMultiply, kDivide, kModulo,
// Comparison
kEqual, kNotEqual, kLessThan, kLessEqual,
kGreaterThan, kGreaterEqual,
// Logical
kAnd, kOr,
// String
kLike, kILike, kSimilarTo, kRegexMatch,
// JSON (12 operators)
kJSONExtract, // ->
kJSONExtractText, // ->>
kJSONExtractPath, // #>
kJSONExtractPathText,// #>>
kJSONContains, // @>
kJSONContainedBy, // <@
kJSONKeyExists, // ?
kJSONAnyKeyExists, // ?|
kJSONAllKeysExist, // ?&
kJSONConcat, // ||
kJSONDelete, // -
kJSONDeletePath, // #-
// Array
kArrayContains, kArrayConcat,
// Other
kIn, kNotIn, kBetween, kNotBetween,
kIsDistinctFrom, kIsNotDistinctFrom,
};
```
**FunctionCall** - Function invocations:
```cpp
class FunctionCall : public Expr {
std::string function_name;
ExprList arguments;
bool is_distinct; // COUNT(DISTINCT x)
bool is_star; // COUNT(*)
ExprPtr filter; // FILTER (WHERE ...)
WindowDef window_def; // OVER clause
bool agg_within_group; // WITHIN GROUP (ORDER BY ...)
};
```
### 8.3.4 Statement Nodes
**SelectStmt** - The most complex statement type:
```cpp
class SelectStmt : public Stmt {
// WITH clause
std::vector with_clause;
bool with_recursive;
// SELECT list
std::vector target_list;
bool distinct;
ExprList distinct_on;
// FROM clause
std::vector from_clause;
std::vector from_subqueries;
std::vector from_table_funcs;
std::vector joins;
// Filtering
ExprPtr where_clause;
ExprList group_by;
std::vector grouping_sets;
ExprPtr having_clause;
// Window definitions
std::vector window_clause;
// Ordering and limits
std::vector order_by;
std::optional limit;
std::optional offset;
// Set operations
SetOperation set_op;
SelectStmtPtr set_left;
SelectStmtPtr set_right;
};
```
**JoinExpr** - Join specifications:
```cpp
enum class JoinType {
kInner, kLeft, kRight, kFull, kCross,
kSemi, kAnti // Generated by subquery optimization
};
class JoinExpr {
JoinType join_type;
RangeVar left;
RangeVar right;
ExprPtr join_condition; // ON clause
std::vector using_columns; // USING(...)
bool is_natural; // NATURAL JOIN
};
```
### 8.3.5 Smart Pointer Conventions
The AST uses unique ownership:
```cpp
using NodePtr = std::unique_ptr;
using ExprPtr = std::unique_ptr;
using StmtPtr = std::unique_ptr;
using ExprList = std::vector;
using StmtList = std::vector;
```
Unique pointers ensure:
- No accidental sharing of mutable state
- Automatic cleanup when AST is destroyed
- Clear ownership semantics
## 8.4 AST Builder
The AST builder converts libpg_query's JSON parse tree into Cognica's typed AST.
### 8.4.1 Builder Architecture
```cpp
class ASTBuilder {
public:
// Build single statement (returns nullptr for multiple statements)
auto build(const ParseTree& tree, std::string_view original_sql) -> StmtPtr;
// Build multiple statements
auto build_all(const ParseTree& tree) -> std::vector;
// Error handling
auto has_error() const -> bool;
auto get_last_error() const -> const std::string&;
private:
// Statement builders (27 types)
auto build_select_stmt_(const Value& node) -> StmtPtr;
auto build_insert_stmt_(const Value& node) -> StmtPtr;
auto build_update_stmt_(const Value& node) -> StmtPtr;
auto build_delete_stmt_(const Value& node) -> StmtPtr;
// ... 23 more
// Expression builders (25+ types)
auto build_expr_(const Value& node) -> ExprPtr;
auto build_column_ref_(const Value& node) -> ExprPtr;
auto build_constant_(const Value& node) -> ExprPtr;
auto build_binary_expr_(const Value& node) -> ExprPtr;
auto build_function_call_(const Value& node) -> ExprPtr;
// ... 20 more
};
```
### 8.4.2 Recursive Descent
The builder uses recursive descent to traverse the JSON tree:
```cpp
auto ASTBuilder::build_expr_(const Value& node) -> ExprPtr {
// Determine expression type from JSON structure
if (node.HasMember("ColumnRef")) {
return build_column_ref_(node["ColumnRef"]);
}
if (node.HasMember("A_Const")) {
return build_constant_(node["A_Const"]);
}
if (node.HasMember("A_Expr")) {
return build_binary_expr_(node["A_Expr"]);
}
if (node.HasMember("FuncCall")) {
return build_function_call_(node["FuncCall"]);
}
if (node.HasMember("SubLink")) {
return build_subquery_expr_(node["SubLink"]);
}
// ... handle other expression types
set_error("Unknown expression type");
return nullptr;
}
```
### 8.4.3 Operator Mapping
PostgreSQL operator names must be mapped to Cognica's enum:
```cpp
auto ASTBuilder::parse_operator_(std::string_view op_name) -> BinaryOpType {
static const std::unordered_map mapping = {
// Arithmetic
{"+", BinaryOpType::kAdd},
{"-", BinaryOpType::kSubtract},
{"*", BinaryOpType::kMultiply},
{"/", BinaryOpType::kDivide},
{"%", BinaryOpType::kModulo},
// Comparison
{"=", BinaryOpType::kEqual},
{"<>", BinaryOpType::kNotEqual},
{"!=", BinaryOpType::kNotEqual},
{"<", BinaryOpType::kLessThan},
{"<=", BinaryOpType::kLessEqual},
{">", BinaryOpType::kGreaterThan},
{">=", BinaryOpType::kGreaterEqual},
// JSON operators
{"->", BinaryOpType::kJSONExtract},
{"->>", BinaryOpType::kJSONExtractText},
{"#>", BinaryOpType::kJSONExtractPath},
{"#>>", BinaryOpType::kJSONExtractPathText},
{"@>", BinaryOpType::kJSONContains},
{"<@", BinaryOpType::kJSONContainedBy},
{"?", BinaryOpType::kJSONKeyExists},
{"?|", BinaryOpType::kJSONAnyKeyExists},
{"?&", BinaryOpType::kJSONAllKeysExist},
// Pattern matching
{"~~", BinaryOpType::kLike},
{"~~*", BinaryOpType::kILike},
{"~", BinaryOpType::kRegexMatch},
{"~*", BinaryOpType::kRegexMatchCI},
};
auto it = mapping.find(op_name);
return it != mapping.end() ? it->second : BinaryOpType::kUnknown;
}
```
### 8.4.4 Complex Node Handling
**CASE Expressions**:
```sql
CASE status
WHEN 'active' THEN 1
WHEN 'pending' THEN 2
ELSE 0
END
```
```cpp
auto ASTBuilder::build_case_expr_(const Value& node) -> ExprPtr {
auto result = std::make_unique();
// Simple CASE: CASE expr WHEN value THEN result
if (node.HasMember("arg")) {
result->case_expr = build_expr_(node["arg"]);
}
// WHEN clauses
for (const auto& when : node["args"].GetArray()) {
CaseWhen case_when;
case_when.condition = build_expr_(when["CaseWhen"]["expr"]);
case_when.result = build_expr_(when["CaseWhen"]["result"]);
result->when_clauses.push_back(std::move(case_when));
}
// ELSE clause
if (node.HasMember("defresult")) {
result->else_result = build_expr_(node["defresult"]);
}
return result;
}
```
**Window Functions**:
```sql
SUM(amount) OVER (PARTITION BY category ORDER BY date ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)
```
```cpp
auto ASTBuilder::build_window_def_(const Value& node) -> WindowDef {
WindowDef def;
// PARTITION BY
if (node.HasMember("partitionClause")) {
for (const auto& expr : node["partitionClause"].GetArray()) {
def.partition_by.push_back(build_expr_(expr));
}
}
// ORDER BY
if (node.HasMember("orderClause")) {
for (const auto& item : node["orderClause"].GetArray()) {
def.order_by.push_back(build_order_by_item_(item));
}
}
// Frame specification
def.frame_options = node.HasMember("frameOptions")
? node["frameOptions"].GetInt()
: 0;
if (node.HasMember("startOffset")) {
def.frame_start_offset = build_expr_(node["startOffset"]);
}
if (node.HasMember("endOffset")) {
def.frame_end_offset = build_expr_(node["endOffset"]);
}
return def;
}
```
## 8.5 Semantic Analysis
Semantic analysis validates the AST and enriches it with type information.
### 8.5.1 Validation Tasks
| Task | Description | Example Error |
|------|-------------|---------------|
| Table existence | Verify tables exist | "Table 'foo' not found" |
| Column existence | Verify columns exist | "Column 'bar' not found in 'users'" |
| Type compatibility | Check operator types | "Cannot compare string to integer" |
| Ambiguity resolution | Resolve unqualified names | "Column 'id' is ambiguous" |
| Aggregate validation | Check GROUP BY usage | "Column must appear in GROUP BY" |
### 8.5.2 Semantic Analyzer Interface
```cpp
class SemanticAnalyzer {
public:
explicit SemanticAnalyzer(const SchemaProvider* schema_provider);
// Analyze any statement
auto analyze(const ast::Stmt& stmt) -> Status;
// Analyze SELECT with column resolution
auto analyze_select(const ast::SelectStmt& stmt) -> Status;
// Analyze and resolve column references
auto analyze_and_resolve(ast::SelectStmt* stmt) -> Status;
private:
const SchemaProvider* schema_provider_;
};
```
### 8.5.3 Schema Provider Interface
The analyzer queries schema information through an abstract interface:
```cpp
class SchemaProvider {
public:
virtual ~SchemaProvider() = default;
// Table existence
virtual auto collection_exists(const std::string& name) const -> bool = 0;
// Table enumeration
virtual auto get_collection_names() const
-> std::unordered_set = 0;
// Column enumeration
virtual auto get_column_names(const std::string& table_name) const
-> std::unordered_set = 0;
};
```
This abstraction enables:
- Testing with mock schemas
- Different backends (document DB, relational tables, external tables)
- Schema caching
### 8.5.4 Name Resolution
Unqualified column references must be resolved:
```sql
SELECT id, name FROM users u JOIN orders o ON u.id = o.user_id
```
The analyzer must determine that:
- `id` refers to `u.id` (or error if ambiguous)
- `name` refers to `u.name`
- `user_id` refers to `o.user_id`
**Resolution Algorithm**:
```cpp
auto SemanticAnalyzer::resolve_column_(
const ColumnRef& col,
const std::vector& scopes
) -> ResolvedColumn {
std::vector candidates;
for (const auto& scope : scopes) {
if (col.table_name.has_value()) {
// Qualified: match table name
if (scope.alias == col.table_name || scope.table_name == col.table_name) {
if (scope.has_column(col.column_name)) {
candidates.push_back({scope.alias, col.column_name});
}
}
} else {
// Unqualified: search all tables
if (scope.has_column(col.column_name)) {
candidates.push_back({scope.alias, col.column_name});
}
}
}
if (candidates.empty()) {
throw SemanticError("Column '" + col.column_name + "' not found");
}
if (candidates.size() > 1) {
throw SemanticError("Column '" + col.column_name + "' is ambiguous");
}
return candidates[0];
}
```
### 8.5.5 Scope Management
Queries can have nested scopes (subqueries, CTEs):
```sql
WITH active_users AS (
SELECT id, name FROM users WHERE status = 'active'
)
SELECT a.name, COUNT(o.id)
FROM active_users a
JOIN orders o ON a.id = o.user_id
GROUP BY a.name
```
Scope stack:
1. CTE scope: `active_users(id, name)`
2. FROM scope: `a(id, name), o(id, user_id, amount, ...)`
```cpp
class ScopeManager {
std::vector scope_stack_;
public:
void push_scope(Scope scope) {
scope_stack_.push_back(std::move(scope));
}
void pop_scope() {
scope_stack_.pop_back();
}
auto resolve_column(const std::string& name) -> ResolvedColumn {
// Search from innermost to outermost scope
for (auto it = scope_stack_.rbegin(); it != scope_stack_.rend(); ++it) {
if (auto col = it->find_column(name)) {
return *col;
}
}
throw SemanticError("Column not found: " + name);
}
};
```
### 8.5.6 Aggregate Validation
Aggregates impose constraints on non-aggregated columns:
```sql
-- Valid: name is in GROUP BY
SELECT name, COUNT(*) FROM users GROUP BY name
-- Invalid: email is not in GROUP BY
SELECT name, email, COUNT(*) FROM users GROUP BY name
```
**Validation Algorithm**:
```cpp
auto SemanticAnalyzer::validate_grouping_(
const SelectStmt& stmt
) -> Status {
if (stmt.group_by.empty() && !has_aggregates(stmt.target_list)) {
return Status::OK(); // No grouping, no aggregates
}
std::unordered_set grouped_columns;
for (const auto& expr : stmt.group_by) {
if (auto* col = dynamic_cast(expr.get())) {
grouped_columns.insert(col->column_name);
}
}
for (const auto& target : stmt.target_list) {
if (!is_aggregate(target.expr.get())) {
auto columns = extract_columns(target.expr.get());
for (const auto& col : columns) {
if (grouped_columns.find(col) == grouped_columns.end()) {
return Status::SemanticError(
"Column '" + col + "' must appear in GROUP BY clause"
);
}
}
}
}
return Status::OK();
}
```
## 8.6 Type System
Cognica's type system bridges SQL types and internal representations.
### 8.6.1 SQL Type Categories
| Category | SQL Types | Internal Type |
|----------|-----------|---------------|
| Integer | INTEGER, BIGINT, SMALLINT | int64_t |
| Float | REAL, DOUBLE PRECISION | double |
| String | VARCHAR, CHAR, TEXT | std::string |
| Boolean | BOOLEAN | bool |
| JSON | JSON, JSONB | Document |
| Array | INTEGER[], VARCHAR[] | std::vector |
| Temporal | DATE, TIME, TIMESTAMP | int64_t (epoch) |
### 8.6.2 Type Inference
Expressions have inferred types based on operators and operands:
$$
\text{type}(a + b) = \text{promote}(\text{type}(a), \text{type}(b))
$$
**Type Promotion Rules**:
| Left Type | Right Type | Result Type |
|-----------|------------|-------------|
| INTEGER | INTEGER | INTEGER |
| INTEGER | DOUBLE | DOUBLE |
| INTEGER | STRING | ERROR |
| STRING | STRING | STRING |
| JSON | STRING | JSON |
```cpp
auto infer_binary_type(
BinaryOpType op,
CVMType left_type,
CVMType right_type
) -> CVMType {
switch (op) {
case BinaryOpType::kAdd:
case BinaryOpType::kSubtract:
case BinaryOpType::kMultiply:
case BinaryOpType::kDivide:
// Numeric operations
if (left_type == CVMType::kDouble || right_type == CVMType::kDouble) {
return CVMType::kDouble;
}
return CVMType::kInt64;
case BinaryOpType::kEqual:
case BinaryOpType::kLessThan:
// ... other comparisons
return CVMType::kBool;
case BinaryOpType::kLike:
case BinaryOpType::kConcat:
return CVMType::kString;
case BinaryOpType::kJSONExtract:
return CVMType::kJSON;
case BinaryOpType::kJSONExtractText:
return CVMType::kString;
default:
return CVMType::kUnknown;
}
}
```
### 8.6.3 Implicit Casting
Some type mismatches can be resolved through implicit casts:
```sql
SELECT * FROM users WHERE id = '123' -- String compared to integer
```
The analyzer inserts an implicit cast:
```cpp
auto maybe_insert_cast_(ExprPtr& expr, CVMType target_type) -> Status {
CVMType source_type = infer_type(expr.get());
if (source_type == target_type) {
return Status::OK();
}
if (can_implicit_cast(source_type, target_type)) {
expr = std::make_unique(std::move(expr), target_type);
return Status::OK();
}
return Status::TypeError(
"Cannot convert " + type_name(source_type) +
" to " + type_name(target_type)
);
}
```
**Implicit Cast Rules**:
| From | To | Allowed |
|------|----|---------|
| INTEGER | DOUBLE | Yes |
| STRING | INTEGER | Yes (with validation) |
| STRING | DOUBLE | Yes (with validation) |
| INTEGER | STRING | Yes |
| BOOLEAN | INTEGER | No |
| JSON | STRING | Yes (serialization) |
## 8.7 Function Resolution
The analyzer resolves function calls to their implementations.
### 8.7.1 Function Registry
```cpp
class ScalarFunctionRegistry {
public:
// Function lookup
auto is_registered(const std::string& name) const -> bool;
auto get_function_info(const std::string& name) const -> FunctionInfo;
// Execution
auto evaluate(
const std::string& name,
const std::vector& args
) -> ScalarFunctionResult;
};
struct FunctionInfo {
ScalarFunctionPtr func;
int32_t min_args;
int32_t max_args; // -1 for variadic
FunctionVolatility volatility;
};
```
### 8.7.2 Function Categories
Cognica provides 150+ built-in functions:
**String Functions** (30+):
- `concat`, `upper`, `lower`, `trim`, `ltrim`, `rtrim`
- `substr`, `substring`, `left`, `right`
- `length`, `char_length`, `octet_length`
- `replace`, `translate`, `reverse`
- `like`, `ilike`, `regexp_match`
- `split_part`, `string_to_array`
**Math Functions** (20+):
- `abs`, `ceil`, `floor`, `round`, `trunc`
- `sqrt`, `cbrt`, `power`, `exp`, `ln`, `log`
- `sin`, `cos`, `tan`, `asin`, `acos`, `atan`
- `mod`, `div`, `gcd`, `lcm`
**DateTime Functions** (25+):
- `now`, `current_date`, `current_time`, `current_timestamp`
- `extract`, `date_part`, `date_trunc`
- `age`, `date_add`, `date_sub`
- `to_char`, `to_date`, `to_timestamp`
**JSON Functions** (30+):
- `jsonb_extract_path`, `jsonb_extract_path_text`
- `jsonb_set`, `jsonb_insert`, `jsonb_delete_path`
- `jsonb_agg`, `jsonb_object_agg`
- `jsonb_array_length`, `jsonb_array_elements`
- `jsonb_typeof`, `jsonb_pretty`
**Aggregate Functions**:
- `count`, `sum`, `avg`, `min`, `max`
- `array_agg`, `string_agg`, `jsonb_agg`
- `bool_and`, `bool_or`, `every`
- `stddev`, `variance`, `corr`, `covar_pop`
### 8.7.3 Function Volatility
Functions are classified by side effects:
```cpp
enum class FunctionVolatility {
kImmutable, // Pure function, can be constant-folded
kStable, // Same within transaction, can cache per-query
kVolatile // May return different results each call
};
```
**Examples**:
| Function | Volatility | Reason |
|----------|------------|--------|
| `abs(x)` | Immutable | Deterministic |
| `now()` | Stable | Same within transaction |
| `random()` | Volatile | Different each call |
| `nextval()` | Volatile | Side effects |
Volatility affects optimization:
```sql
-- Immutable: can fold
SELECT * FROM t WHERE x = abs(-5) -- Becomes: WHERE x = 5
-- Stable: can cache
SELECT * FROM t WHERE created_at > now() - interval '1 day'
-- Volatile: cannot optimize
SELECT * FROM t WHERE x = random() -- Must evaluate per row
```
### 8.7.4 Overload Resolution
Functions can be overloaded by argument types:
```cpp
auto resolve_overload(
const std::string& name,
const std::vector& arg_types
) -> const FunctionOverload* {
auto& overloads = registry_.get_overloads(name);
// Exact match first
for (const auto& overload : overloads) {
if (types_match_exactly(overload.param_types, arg_types)) {
return &overload;
}
}
// Implicit conversion match
for (const auto& overload : overloads) {
if (types_match_with_conversion(overload.param_types, arg_types)) {
return &overload;
}
}
return nullptr; // No matching overload
}
```
## 8.8 Query Rewriting
Before optimization, certain query patterns are rewritten for efficiency.
### 8.8.1 View Expansion
Views are expanded inline:
```sql
-- View definition
CREATE VIEW active_users AS
SELECT id, name, email FROM users WHERE status = 'active';
-- Query
SELECT name FROM active_users WHERE email LIKE '%@example.com';
-- Expanded
SELECT name FROM (
SELECT id, name, email FROM users WHERE status = 'active'
) AS active_users WHERE email LIKE '%@example.com';
```
### 8.8.2 Star Expansion
`SELECT *` is expanded to explicit columns:
```sql
-- Original
SELECT * FROM users
-- Expanded
SELECT id, name, email, status, created_at FROM users
```
This ensures consistent column ordering and enables projection pushdown.
### 8.8.3 Subquery Flattening
Certain subqueries can be flattened:
```sql
-- Original
SELECT * FROM (SELECT id, name FROM users WHERE status = 'active') t
WHERE t.name LIKE 'A%'
-- Flattened
SELECT id, name FROM users
WHERE status = 'active' AND name LIKE 'A%'
```
### 8.8.4 IN to JOIN Transformation
IN subqueries often become joins:
```sql
-- Original
SELECT * FROM orders
WHERE user_id IN (SELECT id FROM users WHERE status = 'active')
-- Transformed
SELECT o.* FROM orders o
INNER JOIN users u ON o.user_id = u.id
WHERE u.status = 'active'
```
This enables join optimization algorithms.
### 8.8.5 Cypher Query Rewriting
When the parser encounters a `cypher()` function call in a FROM clause, it rewrites the call into a standard SQL subselect. The `cypher()` function accepts a graph name, a Cypher query string, and optional parameters, and the parser delegates to a Cypher rewriter that transforms the Cypher query into an equivalent SQL subquery operating on the graph's underlying collections.
```sql
-- Original: Cypher function call in FROM
SELECT v.name, v.age
FROM cypher('social', 'MATCH (n:Person) RETURN n.name, n.age') AS v(name, age)
-- Rewritten: SQL subselect over graph collections
SELECT v.name, v.age
FROM (SELECT ... FROM __graph_social_vertices ... ) AS v(name, age)
```
The rewriter enforces several constraints:
- `cypher()` is only valid in FROM clauses; using it in SELECT or WHERE produces a clear error.
- Mutating Cypher queries (CREATE, DELETE, SET) cannot appear directly in JOINs or multi-source FROM clauses — they must be wrapped in a CTE to ensure deterministic execution order.
- The function requires an explicit column definition list so the parser knows the output schema.
```cpp
auto ASTBuilder::build_cypher_range_subselect_(
const JsonValue& node, bool disallow_mutating_join_usage)
-> std::unique_ptr;
```
The `is_cypher_range_function_()` helper inspects the parse tree to determine whether a RangeFunction node refers to `cypher()`, enabling the parser to route the node through the Cypher rewriting path instead of the standard table function path.
### 8.8.6 Expression-Based LIMIT and OFFSET
LIMIT and OFFSET clauses now accept arbitrary expressions in addition to integer literals. The parser builds AST expression nodes for these clauses while retaining the integer shortcut for constant values:
```sql
-- Constant LIMIT (both limit and limit_expr are set)
SELECT * FROM users LIMIT 10
-- Expression LIMIT (only limit_expr is set; limit is empty)
SELECT * FROM users LIMIT (SELECT count(*) / 10 FROM users)
```
The AST representation reflects this dual approach:
```cpp
class SelectStmt : public Stmt {
ExprPtr limit_expr; // Full expression tree
ExprPtr offset_expr; // Full expression tree
std::optional limit; // Constant shortcut (if available)
std::optional offset; // Constant shortcut (if available)
};
```
When a constant value is available, the planner uses it directly for cost estimation and optimization. When only the expression is present, the physical plan evaluates it at execution time during the `open()` phase of `PhysicalLimit`. The expression must not reference query columns — it is evaluated against an empty document context, and references to row-dependent expressions produce a clear error.
## 8.9 Error Handling
Good error messages are crucial for usability.
### 8.9.1 Error Categories
```cpp
enum class ErrorCategory {
kSyntax, // Parse errors
kSemantic, // Type/name errors
kExecution, // Runtime errors
kInternal // Bug in database
};
```
### 8.9.2 Error Context
Errors include context for debugging:
```cpp
struct QueryError {
ErrorCategory category;
std::string message;
std::optional line;
std::optional column;
std::optional hint;
std::optional detail;
};
```
**Example**:
```sql
SELECT * FROM users WHERE sttaus = 'active'
```
```
ERROR: Column 'sttaus' not found in table 'users'
LINE 1: SELECT * FROM users WHERE sttaus = 'active'
^
HINT: Did you mean 'status'?
```
### 8.9.3 Error Recovery
The parser attempts error recovery for better diagnostics:
```sql
SELECT id name FROM users WHERE status =
```
Rather than failing at the first error (`id name` missing comma), the parser continues to report the incomplete WHERE clause.
## 8.10 Summary
This chapter examined Cognica's SQL parsing and semantic analysis infrastructure. Key takeaways:
1. **PostgreSQL compatibility** via libpg_query provides battle-tested SQL parsing with full support for CTEs, window functions, JSON operators, and more.
2. **The AST** provides a type-safe, navigable representation with 56+ node types covering all SQL constructs.
3. **AST building** recursively transforms JSON parse trees using 70+ specialized builder methods.
4. **Semantic analysis** validates queries by checking table/column existence, resolving ambiguous references, and validating aggregate usage.
5. **The type system** bridges SQL types to internal representations with inference rules and implicit casting.
6. **Function resolution** matches 150+ built-in functions with overload resolution and volatility classification.
7. **Query rewriting** transforms views, expands stars, flattens subqueries, and rewrites Cypher graph queries into SQL subselects before optimization.
8. **Expression-based LIMIT/OFFSET** extends the parser to accept arbitrary expressions, not only integer literals, enabling dynamic pagination and computed bounds.
The parsed and validated AST feeds into the logical planner, where the declarative SQL is transformed into a tree of relational operators — the subject of our next chapter.
# Chapter 9: Logical Planning and Optimization
## 9.1 Introduction to Query Optimization
Query optimization represents one of the most intellectually challenging problems in database systems. Given a declarative SQL query, the optimizer must find an efficient execution strategy from an exponentially large search space of possible plans. The quality of this decision directly impacts query performance by orders of magnitude.
Cognica implements a sophisticated multi-phase optimization pipeline that combines rule-based transformations with cost-based decisions. This chapter explores the logical planning phase, where queries are transformed into an algebraic representation and optimized through a series of rewrite rules before being converted to physical execution plans.
### 9.1.1 The Optimization Problem
Consider a simple three-way join query:
```sql
SELECT o.order_id, c.name, p.title
FROM orders o
JOIN customers c ON o.customer_id = c.id
JOIN products p ON o.product_id = p.id
WHERE c.country = 'US' AND o.total > 100
```
Even for this modest query, the optimizer must decide:
1. **Join Order**: Should we join `orders-customers` first, then `products`? Or `orders-products` first? With $n$ tables, there are $\frac{(2n-2)!}{(n-1)!}$ possible bushy join trees.
2. **Join Algorithm**: Hash join, merge join, nested loop, or index nested loop for each join?
3. **Access Paths**: Full table scan or index scan for each table? Which index if multiple are available?
4. **Filter Placement**: Apply `c.country = 'US'` before or after the join?
5. **Sort Strategy**: In-memory sort, external sort, or leverage index ordering?
The number of possible plans grows combinatorially. For a 10-way join, there are over 17 billion possible join orderings alone. The optimizer must navigate this space efficiently to find a good plan quickly.
### 9.1.2 Optimization Pipeline Architecture
Cognica's query optimizer implements a five-phase pipeline:
```mermaid
flowchart TB
subgraph Phase1["Phase 1: Plan Construction"]
AST[SQL AST] --> LPB[LogicalPlanBuilder]
LPB --> LP1[Logical Plan]
end
subgraph Phase2["Phase 2: Rule-Based Optimization"]
LP1 --> AO[ASTOptimizer]
AO --> R1[MergeFiltersRule]
AO --> R2[PredicatePushdownRule]
AO --> R3[PredicateOrderingRule]
AO --> R4[TopKPushdownRule]
AO --> R5[RemoveRedundantSkipRule]
R1 & R2 & R3 & R4 & R5 --> LP2[Optimized Logical Plan]
end
subgraph Phase3["Phase 3: Statistics"]
LP2 --> CE[CardinalityEstimator]
STATS[(Statistics
Histograms)] --> CE
CE --> LP3[Annotated Plan]
end
subgraph Phase4["Phase 4: Physical Planning"]
LP3 --> PPB[PhysicalPlanBuilder]
APE[AccessPathEnumerator] --> PPB
SS[StrategySelectors] --> PPB
PPB --> PP1[Physical Plan]
end
subgraph Phase5["Phase 5: Memory Allocation"]
PP1 --> MBA[MemoryBudgetAllocator]
MBA --> PP2[Executable Plan]
end
style Phase1 fill:#e1f5fe
style Phase2 fill:#fff3e0
style Phase3 fill:#e8f5e9
style Phase4 fill:#fce4ec
style Phase5 fill:#f3e5f5
```
Each phase serves a distinct purpose:
| Phase | Component | Purpose |
|-------|-----------|---------|
| 1 | LogicalPlanBuilder | Convert AST to relational algebra |
| 2 | ASTOptimizer | Apply transformation rules |
| 3 | CardinalityEstimator | Estimate result sizes |
| 4 | PhysicalPlanBuilder | Select algorithms and access paths |
| 5 | MemoryBudgetAllocator | Distribute memory, predict spills |
## 9.2 Logical Plan Representation
The logical plan represents a query as a tree of relational algebra operators. Unlike the physical plan (which specifies concrete algorithms), the logical plan describes *what* operations to perform without specifying *how*.
### 9.2.1 Operator Types
Cognica defines fourteen logical operator types:
```mermaid
classDiagram
class LogicalOperator {
<>
+type() LogicalOpType
+children() vector~LogicalOpPtr~
+estimated_rows() optional~double~
+clone() LogicalOpPtr
}
class ScanOp {
+collection_name: string
+index_hint: optional~string~
}
class FilterOp {
+predicate: Expression
}
class ProjectOp {
+projections: vector~Projection~
}
class SortOp {
+sort_keys: vector~SortKey~
}
class LimitOp {
+limit_expr: ExprPtr
+offset_expr: ExprPtr
+limit: optional~int64_t~
+offset: optional~int64_t~
}
class GroupOp {
+group_keys: vector~Expression~
+aggregates: vector~AggregateSpec~
}
class JoinOp {
+join_spec: JoinSpec
}
class UnionOp {
+union_spec: UnionSpec
}
class SubqueryOp {
+alias: optional~string~
+colnames: vector~string~
+lateral: bool
}
class TableFuncOp {
+function_name: string
+arguments: vector~ExprPtr~
+output_columns: vector~string~
}
class UnnestOp {
+unnest_expr: ExprPtr
}
class SearchOp {
+search_type: SearchType
+field_queries: map
}
LogicalOperator <|-- ScanOp
LogicalOperator <|-- FilterOp
LogicalOperator <|-- ProjectOp
LogicalOperator <|-- SortOp
LogicalOperator <|-- LimitOp
LogicalOperator <|-- GroupOp
LogicalOperator <|-- JoinOp
LogicalOperator <|-- UnionOp
LogicalOperator <|-- SubqueryOp
LogicalOperator <|-- TableFuncOp
LogicalOperator <|-- UnnestOp
LogicalOperator <|-- SearchOp
```
Each operator type corresponds to a relational algebra operation:
| Operator | Algebra | SQL Clause |
|----------|---------|------------|
| `ScanOp` | $R$ | `FROM table` |
| `FilterOp` | $\sigma_\theta(R)$ | `WHERE condition` |
| `ProjectOp` | $\pi_{a_1,...,a_n}(R)$ | `SELECT columns` |
| `SortOp` | $\tau_{k_1,...,k_n}(R)$ | `ORDER BY` |
| `LimitOp` | $\lambda_{n}(R)$ | `LIMIT n` / `LIMIT expr` |
| `GroupOp` | $\gamma_{G,F}(R)$ | `GROUP BY` |
| `JoinOp` | $R \bowtie_\theta S$ | `JOIN ... ON` |
| `UnionOp` | $R \cup S$ | `UNION` |
| `SubqueryOp` | $\rho_{alias}(Q)$ | `FROM (SELECT ...) AS t` |
| `TableFuncOp` | $f(args)$ | `FROM func(...)` |
| `UnnestOp` | $\mu(e)$ | `FROM unnest(array)` |
| `SearchOp` | $\mathcal{S}_q(R)$ | Full-text search |
**Expression-Based LIMIT/OFFSET**: The `LimitOp` supports both constant values and arbitrary expressions for `LIMIT` and `OFFSET`. When the limit or offset is a constant integer literal, the planner stores it directly in the `limit` / `offset` optional fields. When the value comes from a parameter, a subquery, or an arithmetic expression, the planner stores the unevaluated AST expression in `limit_expr` / `offset_expr` and defers evaluation to execution time. This dual representation lets the optimizer reason about constant limits during planning (for TopK pushdown and cardinality estimation) while still supporting dynamic limits such as `LIMIT $1`.
### 9.2.2 Plan Tree Structure
A logical plan forms a tree where:
- Leaf nodes are `ScanOp` operators (data sources)
- Internal nodes are transformation operators
- The root produces the final query result
For the query:
```sql
SELECT name, total FROM orders WHERE status = 'shipped' ORDER BY total DESC LIMIT 10
```
The logical plan tree is:
```mermaid
flowchart TB
L[LimitOp
count=10] --> S[SortOp
total DESC]
S --> P[ProjectOp
name, total]
P --> F[FilterOp
status='shipped']
F --> SC[ScanOp
orders]
style L fill:#ffcdd2
style S fill:#c8e6c9
style P fill:#bbdefb
style F fill:#fff9c4
style SC fill:#e1bee7
```
### 9.2.3 Operator Properties
Each logical operator maintains properties that guide optimization:
**Cardinality Estimate**: The `estimated_rows_` field stores the predicted output row count. This is populated during cardinality estimation and used for cost calculations.
**Schema Propagation**: Operators track their output schema, enabling the optimizer to verify that downstream operators reference valid columns.
**Ordering Properties**: Some operators (Sort, certain scans) produce ordered output. Tracking this enables sort elimination optimizations.
## 9.3 Rule-Based Optimization
The first optimization phase applies a set of transformation rules that improve the plan regardless of data statistics. These rules implement algebraic equivalences that always (or almost always) improve performance.
### 9.3.1 Optimization Rule Framework
Cognica implements an extensible rule framework:
```cpp
class OptimizationRule {
public:
virtual ~OptimizationRule() = default;
virtual auto apply(LogicalOpPtr root) -> std::pair = 0;
virtual auto name() const -> std::string = 0;
};
```
Each rule's `apply()` method returns:
1. The transformed plan (possibly unchanged)
2. A boolean indicating whether any transformation occurred
The `ASTOptimizer` manages rule application:
```cpp
class ASTOptimizer {
public:
auto optimize(LogicalPlan plan) -> std::pair;
private:
std::vector> rules_;
uint32_t max_passes_ = 10;
};
```
Rules are applied iteratively until a fixed point (no rule makes changes) or the maximum iteration count is reached. This fixed-point iteration ensures that rules can trigger each other—for example, filter merging might enable additional pushdown opportunities.
### 9.3.2 MergeFiltersRule
The first rule merges consecutive filter operators into a single filter with a conjunctive predicate.
**Transformation**:
$$
\sigma_{\theta_1}(\sigma_{\theta_2}(R)) \Rightarrow \sigma_{\theta_1 \land \theta_2}(R)
$$
**Example**:
Before:
```
Filter(price > 100)
Filter(category = 'electronics')
Scan(products)
```
After:
```
Filter(price > 100 AND category = 'electronics')
Scan(products)
```
**Benefits**:
1. Reduces operator overhead (fewer virtual function calls)
2. Enables better predicate evaluation order
3. Improves index matching (composite predicates may match composite indexes)
The implementation recursively processes the tree bottom-up:
```cpp
auto MergeFiltersRule::merge_recursive_(LogicalOpPtr node) -> LogicalOpPtr {
// First, recursively process children
for (size_t i = 0; i < node->children().size(); ++i) {
node->set_child(i, merge_recursive_(node->children()[i]));
}
// If this is a filter with a filter child, merge them
if (node->type() == LogicalOpType::kFilter) {
auto* filter = static_cast(node.get());
if (!filter->children().empty() &&
filter->children()[0]->type() == LogicalOpType::kFilter) {
auto* child_filter = static_cast(filter->children()[0].get());
// Create AND of both predicates
auto merged = ast::make_and(filter->predicate(), child_filter->predicate());
filter->set_predicate(std::move(merged));
// Skip the child filter
filter->set_child(0, child_filter->children()[0]);
}
}
return node;
}
```
### 9.3.3 PredicatePushdownRule
Predicate pushdown moves filter operations closer to data sources, reducing the number of rows that flow through the plan.
**Transformation**:
$$
\pi_L(\sigma_\theta(R)) \Rightarrow \pi_L(\sigma_\theta(R)) \text{ (no change)}
$$
$$
\sigma_\theta(\pi_L(R)) \Rightarrow \pi_L(\sigma_\theta(R)) \text{ (if } \theta \text{ only references } L\text{)}
$$
**Pushdown Compatibility Matrix**:
| Parent Operator | Can Push Through? | Condition |
|-----------------|-------------------|-----------|
| Project | Yes | If predicate columns are in projection |
| Sort | Yes | Always |
| Limit | Yes | Always |
| Skip | Yes | Always |
| Group | No | Predicate references aggregates |
| Join | Partial | Only predicates on single table |
| Search | No | Changes semantics |
**Example**:
Before:
```
Project(name, price)
Sort(price DESC)
Filter(category = 'books')
Scan(products)
```
After:
```
Project(name, price)
Sort(price DESC)
Filter(category = 'books')
Scan(products)
```
In this case, the filter is already at the optimal position. But consider:
Before:
```
Filter(category = 'books')
Project(name, price, category)
Sort(price DESC)
Scan(products)
```
After:
```
Project(name, price, category)
Sort(price DESC)
Filter(category = 'books')
Scan(products)
```
**Benefits**:
1. Reduces data volume early in the pipeline
2. Enables index utilization (filter at scan level can use indexes)
3. Reduces memory pressure for blocking operators
### 9.3.4 PredicateOrderingRule
Within a compound predicate, the evaluation order affects performance. For short-circuit evaluation:
- In AND clauses: evaluate most selective predicates first
- In OR clauses: evaluate least selective predicates first
**Selectivity Heuristics**:
The rule uses hardcoded selectivity estimates when statistics are unavailable:
| Predicate Type | Estimated Selectivity |
|----------------|----------------------|
| Equality (`=`) | 0.01 (1%) |
| Inequality (`!=`) | 0.99 (99%) |
| Range (`<`, `>`, `<=`, `>=`) | 0.33 (33%) |
| `IN` clause | 0.01 per value, max 0.30 |
| Pattern match (`LIKE`) | 0.15 (15%) |
| Existence check | 0.90 (90%) |
**Compound Selectivity**:
For AND:
$$
S(\theta_1 \land \theta_2) = S(\theta_1) \times S(\theta_2)
$$
For OR:
$$
S(\theta_1 \lor \theta_2) = 1 - (1 - S(\theta_1))(1 - S(\theta_2))
$$
For NOT:
$$
S(\lnot \theta) = 1 - S(\theta)
$$
**Example**:
Before:
```sql
WHERE status != 'deleted' AND id = 12345
```
After:
```sql
WHERE id = 12345 AND status != 'deleted'
```
The equality predicate (1% selectivity) is evaluated first because it will short-circuit more often than the inequality (99% selectivity).
### 9.3.5 TopKPushdownRule
When a query has both `ORDER BY` and `LIMIT`, the optimizer can avoid sorting all rows by using a heap-based Top-K algorithm.
**Transformation**:
$$
\lambda_k(\tau_{keys}(R)) \Rightarrow \text{TopK}_k^{keys}(R) \text{ (when } k \leq 1000\text{)}
$$
**Example**:
Before:
```
Limit(10)
Sort(score DESC)
Scan(articles) -- 10 million rows
```
After (conceptually):
```
TopKSort(10, score DESC)
Scan(articles)
```
**Algorithm Comparison**:
| Algorithm | Time Complexity | Space Complexity |
|-----------|-----------------|------------------|
| Full Sort | $O(n \log n)$ | $O(n)$ |
| Top-K Heap | $O(n \log k)$ | $O(k)$ |
For $n = 10^7$ and $k = 10$:
- Full sort: ~233 million comparisons
- Top-K heap: ~33 million comparisons
The rule annotates the Sort operator with `estimated_rows = k`, signaling to the physical planner to use a heap-based strategy.
### 9.3.6 RemoveRedundantSkipRule
A `Skip(0)` operator has no effect and can be removed:
$$
\delta_0(R) \Rightarrow R
$$
This rule handles edge cases where query builders generate trivial offsets.
### 9.3.7 Fixed-Point Iteration
Rules are applied iteratively because one transformation may enable another:
```cpp
auto ASTOptimizer::optimize(LogicalPlan plan) -> std::pair {
auto stats = OptimizationStats {};
auto root = plan.take_root();
for (uint32_t pass = 0; pass < max_passes_; ++pass) {
auto any_changed = false;
for (auto& rule : rules_) {
auto [new_root, changed] = rule->apply(std::move(root));
root = std::move(new_root);
if (changed) {
any_changed = true;
++stats.rules_applied;
}
}
if (!any_changed) {
break; // Fixed point reached
}
}
plan.set_root(std::move(root));
return {std::move(plan), stats};
}
```
Typical queries converge within 2-3 passes. The maximum of 10 passes handles pathological cases.
### 9.3.8 Logical Plan Optimization Pipeline
After the AST-level rule-based optimizer produces an initial logical plan, the `PlanOptimizer` applies a second round of transformations that operate directly on the logical plan tree. These passes require access to the full relational algebra structure and cannot be expressed as simple AST rewrite rules.
The `PlanOptimizer` executes a fixed sequence of passes:
```
1. simplify_predicates_ -- Boolean algebra simplification
2. estimate_selectivity_ -- Annotate filters and joins with selectivity
3. pushdown_predicates_ -- Push filters below joins and projections
4. unnest_subqueries_ -- Convert correlated subqueries to joins
5. eliminate_common_subexpressions_ -- Factor out repeated expressions
6. expand_join_or_to_union_ -- Rewrite OR-linked join predicates
7. reorder_joins_ -- Cost-based multi-way join enumeration
8. prune_columns_ -- Remove unused columns from scans
```
Unlike the fixed-point iteration of the AST optimizer, the plan optimizer runs each pass exactly once in the order shown above. The ordering is significant: selectivity estimates must be available before predicate pushdown can make cost-aware decisions, and subquery unnesting must complete before join reordering can consider the newly introduced join nodes.
### 9.3.9 Subquery Unnesting
Subquery unnesting (also called subquery decorrelation) converts correlated scalar subqueries and `EXISTS` / `IN` / `NOT EXISTS` subqueries into equivalent join operations. This transformation is important because a correlated subquery would otherwise require re-execution for every outer row, producing $O(n \times m)$ work, whereas a join can be executed in $O(n + m)$ with a hash join.
**EXISTS Subquery to Semi-Join**:
A filter of the form `EXISTS (SELECT 1 FROM S WHERE S.fk = R.pk)` is converted to a semi-join:
$$
\sigma_{\text{EXISTS}(Q)}(R) \Rightarrow R \ltimes_{\theta} S
$$
**IN Subquery to Semi-Join**:
A predicate `R.col IN (SELECT S.col FROM S WHERE ...)` is rewritten similarly:
$$
\sigma_{col \in Q}(R) \Rightarrow R \ltimes_{R.col = S.col} S
$$
**NOT EXISTS to Anti-Join**:
$$
\sigma_{\lnot\text{EXISTS}(Q)}(R) \Rightarrow R \rhd_{\theta} S
$$
**Source Resolution**: The unnesting pass must determine the data source for the right side of the generated join. Earlier versions only supported plain table references in the subquery's `FROM` clause. The current implementation uses a unified source resolution strategy that handles three kinds of FROM sources:
1. **Table references** — the subquery selects from a named table, producing a `LogicalScan` node.
2. **Derived tables** — the subquery's FROM clause is itself a subquery (`FROM (SELECT ...) AS t`), producing a `LogicalSubquery` node that wraps a recursively built logical plan.
3. **Table-valued functions** — the subquery references a table function (`FROM generate_series(1, 10)`), producing a `LogicalTableFunc` node.
This generality allows unnesting to work uniformly regardless of the subquery's internal structure. The join condition is then constructed from the correlation predicate, and any remaining WHERE-clause predicates from the subquery are placed as a filter above the join's right child.
### 9.3.10 Common Subexpression Elimination
Common subexpression elimination (CSE) identifies expression subtrees that appear more than once in a query's projection list and factors them into a single computation. The result is referenced by subsequent operators through a generated column name, avoiding redundant evaluation.
**Example**:
```sql
SELECT price * quantity AS line_total,
price * quantity * tax_rate AS tax_amount
FROM orders
```
The expression `price * quantity` appears twice. CSE factors it into a pre-computation step, and the outer projection references the computed column instead of re-evaluating the multiplication.
**Exclusions**: Not all repeated expressions are candidates for CSE. The optimizer skips:
- **Aggregate and window functions** — these have special evaluation semantics that cannot be hoisted into a pre-projection layer.
- **`SELECT *` queries** — wildcard projections are incompatible with the intermediate projection node that CSE inserts, because the column set is not fully known at optimization time.
- **Field access and array slicing expressions** — `FieldAccess` and `ArraySlice` nodes over lateral or correlated values are context-sensitive; hoisting them into a CSE layer can change evaluation semantics when the outer row varies.
### 9.3.11 Column Pruning
The final optimization pass removes columns from scan operators that are not referenced by any downstream operator. This reduces I/O and memory consumption, which is significant for wide tables where only a few columns are needed.
The pass walks the plan tree top-down, collecting the set of columns referenced by each operator's expressions (projection lists, filter predicates, join conditions, sort keys, group-by keys, and aggregate arguments). Scan operators are then annotated with this minimal column set so that the physical planner can push the projection into the storage layer.
## 9.4 Cardinality Estimation
Accurate cardinality estimates are crucial for cost-based optimization. A wrong estimate can lead to catastrophically bad plan choices.
### 9.4.1 The Cardinality Estimation Problem
Consider estimating the result size of:
```sql
SELECT * FROM orders WHERE status = 'pending' AND total > 1000
```
We need to estimate:
1. How many rows satisfy `status = 'pending'`?
2. How many rows satisfy `total > 1000`?
3. How many rows satisfy both?
The naive approach assumes independence:
$$
|R_{\theta_1 \land \theta_2}| = |R| \times S(\theta_1) \times S(\theta_2)
$$
But predicates are often correlated. If `status = 'pending'` implies recent orders, and recent orders tend to be smaller, the predicates are negatively correlated.
### 9.4.2 CardinalityEstimate Structure
Cognica represents cardinality estimates with:
```cpp
struct CardinalityEstimate {
double rows; // Estimated row count
double row_width = 512.0; // Average bytes per row
auto memory_estimate() const -> double {
return rows * row_width;
}
auto apply_selectivity(double selectivity) -> CardinalityEstimate& {
rows *= selectivity;
return *this;
}
};
```
The `row_width` field enables memory estimation for buffer sizing and spill prediction.
### 9.4.3 Operator Cardinality Propagation
Each operator type has specific propagation rules:
**Scan**:
$$
|Scan(R)| = |R|
$$
The base table cardinality comes from statistics.
**Filter**:
$$
|Filter_\theta(R)| = |R| \times S(\theta)
$$
**Sort, Project**:
$$
|Sort(R)| = |Project(R)| = |R|
$$
Rows unchanged (though `row_width` may change for Project).
**Limit**:
$$
|Limit_k(R)| = \min(|R|, k)
$$
When the limit is an expression rather than a constant, cardinality estimation falls back to the child estimate because the actual value is unknown until execution time.
**Offset**:
$$
|Offset_k(R)| = \max(0, |R| - k)
$$
**Group**:
$$
|Group_{G,F}(R)| = NDV(G)
$$
Where $NDV(G)$ is the number of distinct values in the grouping columns.
**Join**:
$$
|R \bowtie_\theta S| = |R| \times |S| \times S_{join}(\theta)
$$
**Union**:
$$
|R \cup S| = |R| + |S|
$$
### 9.4.4 Selectivity Estimation
The `CardinalityEstimator` class provides selectivity estimation for various predicate types:
**Equality Selectivity**:
With statistics:
$$
S(col = v) = \frac{1}{NDV(col)}
$$
Without statistics:
$$
S(col = v) = 0.01 \text{ (default)}
$$
**Range Selectivity**:
With histogram:
$$
S(col < v) = \frac{\sum_{b: upper_b < v} count_b + \text{interpolate}(b_v)}{|R|}
$$
Without histogram:
$$
S(col < v) = 0.30 \text{ (default)}
$$
**IN Selectivity**:
$$
S(col \in \{v_1, ..., v_k\}) = \min(k \times S(col = v_i), 0.30)
$$
**Pattern Selectivity**:
$$
S(col \sim pattern) = 0.15 \text{ (default)}
$$
### 9.4.5 Correlation Handling
The naive independence assumption often produces severe underestimates. Cognica uses a damping factor for conjunctive predicates:
$$
S(\theta_1 \land \theta_2) = \sqrt{S(\theta_1) \times S(\theta_2)}
$$
This geometric mean provides a middle ground between:
- Full independence: $S(\theta_1) \times S(\theta_2)$ (often too low)
- Full correlation: $\min(S(\theta_1), S(\theta_2))$ (often too high)
**Example**:
For `status = 'active' AND country = 'US'`:
- $S(status = 'active') = 0.3$
- $S(country = 'US') = 0.2$
- Independence: $0.3 \times 0.2 = 0.06$
- Damped: $\sqrt{0.3 \times 0.2} = 0.245$
### 9.4.6 Default Constants
When statistics are unavailable, the estimator uses conservative defaults:
```cpp
static constexpr double kDefaultEqualitySelectivity = 0.01;
static constexpr double kDefaultRangeSelectivity = 0.30;
static constexpr double kDefaultInSelectivity = 0.05;
static constexpr double kDefaultJoinSelectivity = 0.1;
static constexpr double kDefaultGroupReduction = 0.1;
```
These defaults are calibrated to avoid catastrophic underestimates while remaining reasonably selective.
## 9.5 Statistics and Histograms
Accurate cardinality estimation requires statistics about data distribution. Cognica maintains comprehensive statistics including histograms, distinct value counts, and correlation information.
### 9.5.1 Histogram Structure
Cognica uses equi-depth (equi-height) histograms:
```cpp
struct HistogramBucket {
double lower_bound; // Inclusive
double upper_bound; // Inclusive
int64_t count; // Values in bucket
int64_t distinct; // Distinct values in bucket
};
```
**Equi-Depth Property**: Each bucket contains approximately the same number of values:
$$
count_i \approx \frac{|R|}{B}
$$
Where $B$ is the number of buckets (default: 100).
**Advantages over Equi-Width**:
- Better handles skewed distributions
- Uniform error bounds across the value range
- Naturally adapts to data density
### 9.5.2 Histogram-Based Selectivity
**Equality Selectivity**:
```cpp
auto Histogram::estimate_equality_selectivity(double value) const -> double {
auto bucket_idx = find_bucket_(value);
if (bucket_idx >= buckets_.size()) {
return 0.0; // Value outside range
}
const auto& bucket = buckets_[bucket_idx];
// Assume uniform distribution within bucket
return static_cast(bucket.count) /
static_cast(bucket.distinct * total_count_);
}
```
**Range Selectivity**:
$$
S(lower \leq col < upper) = \frac{\sum_{b \in range} count_b}{|R|}
$$
With linear interpolation for partial buckets:
```cpp
auto Histogram::interpolate_bucket_fraction_(
size_t bucket_idx, double lower, double upper) const -> double {
const auto& b = buckets_[bucket_idx];
auto bucket_width = b.upper_bound - b.lower_bound;
if (bucket_width <= 0) {
return 1.0; // Single-value bucket
}
auto effective_lower = std::max(lower, b.lower_bound);
auto effective_upper = std::min(upper, b.upper_bound);
return (effective_upper - effective_lower) / bucket_width;
}
```
### 9.5.3 Multi-Column Statistics
For correlated columns, single-column statistics are insufficient. Cognica supports several multi-column statistics:
**N-Distinct Statistics**:
Tracks distinct value counts for column combinations:
```cpp
// Key: "col1,col2,col3" (sorted, comma-separated)
// Value: distinct count for the combination
std::unordered_map multi_column_ndistinct;
```
**Functional Dependencies**:
Tracks when one column determines another:
```cpp
// Key: "A->B" (A determines B)
// Value: dependency strength [0.0, 1.0]
std::unordered_map functional_dependencies;
```
A value of 1.0 indicates perfect functional dependency; lower values indicate partial dependency.
**2D Histograms**:
For pairs of numeric columns with significant correlation:
```cpp
struct Histogram2D {
std::vector x_boundaries;
std::vector y_boundaries;
std::vector> grid; // Row-major counts
};
```
### 9.5.4 Statistics Collection
Statistics are collected by `IndexStatisticsCollector`:
```cpp
struct Statistics {
// Basic cardinality
int64_t total_keys;
int64_t distinct_values;
HyperLogLog hll; // Approximate distinct count
// Range information
std::unordered_map min_values;
std::unordered_map max_values;
std::unordered_map min_string_values;
std::unordered_map max_string_values;
// Null counts
std::unordered_map null_counts;
// Histograms
std::unordered_map histograms;
// Multi-column statistics
std::unordered_map multi_column_ndistinct;
std::unordered_map functional_dependencies;
std::unordered_map histograms_2d;
// Staleness tracking
TimePoint last_updated;
bool is_stale;
int64_t last_analyzed_write_count;
};
```
## 9.6 Cost Model
The cost model estimates the resource consumption of execution plans, enabling the optimizer to compare alternatives.
### 9.6.1 Cost Structure
Cognica uses a multi-dimensional cost model:
```cpp
struct Cost {
double cpu_cost; // CPU cycles estimate
double io_cost; // I/O operations
double memory_cost; // Memory pressure
double spill_cost; // Disk spill overhead
auto total() const -> double {
return cpu_cost +
io_cost * kIOCostWeight +
memory_cost * kMemoryCostWeight +
spill_cost * kSpillCostWeight;
}
};
```
**Cost Weights**:
```cpp
static constexpr double kIOCostWeight = 10.0;
static constexpr double kMemoryCostWeight = 0.1;
static constexpr double kSpillCostWeight = 5.0;
```
The weights reflect modern hardware characteristics:
- I/O is heavily weighted because disk access is orders of magnitude slower than CPU
- Memory cost is lightly weighted because it represents pressure, not direct time
- Spill cost represents the penalty of exceeding memory budgets
### 9.6.2 Cost Constants
The cost model uses calibrated constants:
```cpp
// CPU costs
static constexpr double kCPUTupleProcessing = 0.01;
static constexpr double kCPUComparison = 0.0001;
static constexpr double kCPUHashOp = 0.0005;
// I/O costs
static constexpr double kHeapAccessCost = 0.5;
static constexpr double kSpillIOCost = 0.01; // Per byte
// Memory constants
static constexpr double kCompressionFactor = 0.5; // LZ4 typical
static constexpr double kHashTableOverhead = 1.5;
static constexpr size_t kMergeWidth = 16;
static constexpr size_t kDefaultBlockSize = 8192;
static constexpr size_t kAggStateSize = 256;
```
### 9.6.3 Operator Cost Formulas
**Sequential Scan**:
$$
C_{seq}(R) = |R| \times (cpu_{tuple} + io_{page} \times \frac{1}{tuples_{per\_page}})
$$
**Index Scan**:
$$
C_{idx}(R, sel) = |R| \times sel \times (cpu_{tuple} + io_{random} + io_{heap})
$$
The random I/O cost reflects the non-sequential access pattern.
**Sort**:
In-memory:
$$
C_{sort}(R) = |R| \times \log_2(|R|) \times cpu_{cmp}
$$
External:
$$
C_{ext\_sort}(R) = C_{sort}(R) + \frac{2 \times |R| \times width}{block} \times (io_{write} + io_{read}) \times \lceil \log_M(runs) \rceil
$$
Where $M$ is the merge width and $runs$ is the number of initial sorted runs.
**Hash Join**:
$$
C_{hash}(R, S) = |R| \times cpu_{hash} \times k_{build} + |S| \times cpu_{hash} \times k_{probe}
$$
Where $k_{build} = 1.0$ and $k_{probe} = 1.2$ (probe is slightly more expensive due to collision handling).
**Nested Loop Join**:
$$
C_{nl}(R, S) = |R| \times |S| \times cpu_{tuple} \times k_{nl}
$$
Where $k_{nl} = 10.0$ reflects the high cost of repeated inner scans.
### 9.6.4 Spill Prediction
The cost model predicts when operations will exceed memory budgets:
```cpp
auto CostEstimator::will_spill(double data_size, size_t memory_budget) const -> bool {
return data_size > static_cast(memory_budget);
}
```
When spill is predicted, the cost model adds spill overhead:
$$
C_{spill} = \frac{data\_size}{block\_size} \times io_{spill} \times (1 + \frac{1}{compression})
$$
## 9.7 Join Ordering
Join ordering is often the most impactful optimization decision. The order of joins can change query execution time by orders of magnitude.
### 9.7.1 The Join Ordering Problem
For $n$ tables, the number of possible join trees is:
$$
T(n) = \frac{(2n-2)!}{(n-1)!}
$$
| Tables | Join Trees |
|--------|------------|
| 2 | 1 |
| 3 | 12 |
| 4 | 120 |
| 5 | 1,680 |
| 10 | 17,643,225,600 |
Exhaustive enumeration is infeasible for large queries. Dynamic programming reduces the complexity to $O(3^n)$ while guaranteeing optimality.
### 9.7.2 Join Graph Representation
Cognica represents join relationships as a graph:
```cpp
struct JoinVertex {
std::string identifier; // Table alias
LogicalPlanPtr plan; // Subplan for this table
double cardinality; // Estimated rows
};
struct JoinGraphEdge {
size_t left_idx; // Left table index
size_t right_idx; // Right table index
ast::Expression* condition; // Join predicate
ast::JoinType type; // Inner, left, right, full
double selectivity; // Join selectivity
};
```
**Example**:
For the query:
```sql
SELECT * FROM A
JOIN B ON A.x = B.x
JOIN C ON B.y = C.y
JOIN D ON A.z = D.z
```
```mermaid
graph LR
A((A)) --- |x=x| B((B))
B --- |y=y| C((C))
A --- |z=z| D((D))
```
### 9.7.3 DPccp Algorithm
Cognica implements the DPccp (Dynamic Programming with Connected-Complement-Pairs) algorithm from Moerkotte and Neumann. This algorithm generates optimal bushy join trees without cross products.
**Key Insight**: A valid join plan connects a subset $S$ of tables with its complement $\bar{S}$ only if there exists a join predicate between them.
**Algorithm Structure**:
```cpp
auto JoinOrderEnumerator::enumerate(const JoinGraph& graph) -> LogicalPlanPtr {
// Initialize DP table with single-table plans
for (size_t i = 0; i < graph.vertex_count(); ++i) {
auto singleton = 1ULL << i;
dp_table_[singleton] = {
.subset = singleton,
.plan = graph.vertices()[i].plan,
.cost = 0.0,
.cardinality = graph.vertices()[i].cardinality
};
}
// Enumerate connected subgraph-complement pairs
enumerate_csg_cmp_pairs_(graph);
// Return plan for all tables
auto all_tables = (1ULL << graph.vertex_count()) - 1;
return dp_table_[all_tables].plan;
}
```
**CSG-CMP Enumeration**:
The algorithm enumerates pairs $(S_1, S_2)$ where:
1. $S_1$ and $S_2$ are both connected subgraphs
2. $S_1 \cap S_2 = \emptyset$
3. There exists an edge between $S_1$ and $S_2$
```cpp
void JoinOrderEnumerator::enumerate_csg_cmp_pairs_(const JoinGraph& graph) {
auto n = graph.vertex_count();
// For each starting vertex
for (size_t i = 0; i < n; ++i) {
auto start = 1ULL << i;
// Enumerate connected subgraphs containing vertex i
enumerate_csg_rec_(graph, start, start, i);
}
}
void JoinOrderEnumerator::enumerate_csg_rec_(
const JoinGraph& graph,
uint64_t current,
uint64_t excluded,
size_t min_vertex) {
// For current subgraph, enumerate complements
enumerate_cmp_(graph, current);
// Extend subgraph with neighbors
auto neighbors = get_neighborhood_(graph, current, excluded, min_vertex);
for (auto neighbor : neighbors) {
auto extended = current | neighbor;
enumerate_csg_rec_(graph, extended, excluded | neighbor, min_vertex);
}
}
```
**Complement Enumeration**:
```cpp
void JoinOrderEnumerator::enumerate_cmp_(const JoinGraph& graph, uint64_t s1) {
// Find edges crossing from s1 to complement
auto crossing = graph.get_crossing_edges(s1);
if (crossing.empty()) {
return; // No valid complement
}
// Build complement subgraphs
auto complement_vertices = get_complement_vertices_(graph, s1, crossing);
for (auto s2 : enumerate_connected_subsets_(graph, complement_vertices)) {
emit_csg_cmp_(graph, s1, s2);
}
}
```
**Plan Emission**:
```cpp
void JoinOrderEnumerator::emit_csg_cmp_(
const JoinGraph& graph, uint64_t s1, uint64_t s2) {
// Skip if either subset doesn't have a plan yet
if (dp_table_.find(s1) == dp_table_.end() ||
dp_table_.find(s2) == dp_table_.end()) {
return;
}
auto& plan1 = dp_table_[s1];
auto& plan2 = dp_table_[s2];
// Create join plan and estimate cost
auto join_plan = create_join_plan_(graph, s1, s2, plan1, plan2);
auto cost = estimate_join_cost_(plan1, plan2);
auto cardinality = estimate_join_cardinality_(graph, s1, s2, plan1, plan2);
// Store if better than existing plan
auto combined = s1 | s2;
store_plan_(combined, std::move(join_plan), cost, cardinality);
}
```
### 9.7.4 Join Cost Estimation
The join cost depends on the algorithm chosen:
```cpp
static constexpr double kHashBuildCostPerRow = 1.0;
static constexpr double kHashProbeCostPerRow = 1.2;
static constexpr double kNestLoopCostPerRow = 10.0;
static constexpr double kSortMergeCostPerRow = 2.0;
```
**Hash Join Cost**:
$$
C_{hash}(R, S) = |R| \times k_{build} + |S| \times k_{probe}
$$
The smaller relation is typically chosen as the build side.
**Join Cardinality**:
$$
|R \bowtie S| = |R| \times |S| \times S_{join}
$$
Join selectivity is estimated from:
1. Key cardinalities (for equi-joins on keys)
2. Default selectivity (0.1) when statistics are unavailable
### 9.7.5 Bushy vs Left-Deep Plans
DPccp generates bushy plans, which can be more efficient than left-deep plans:
**Left-Deep Plan**:
```mermaid
graph TD
J3[Join] --> J2[Join]
J3 --> D((D))
J2 --> J1[Join]
J2 --> C((C))
J1 --> A((A))
J1 --> B((B))
```
**Bushy Plan**:
```mermaid
graph TD
J3[Join] --> J1[Join]
J3 --> J2[Join]
J1 --> A((A))
J1 --> B((B))
J2 --> C((C))
J2 --> D((D))
```
Bushy plans enable parallelism and can reduce intermediate result sizes.
### 9.7.6 Join Reordering Safety Guard
The multi-way join enumeration algorithm (DPccp) assumes that the inputs to each join are independent, side-effect-free relations whose evaluation order does not affect semantics. This assumption holds for plain table scans but breaks down for several classes of join inputs:
- **Lateral joins** — the right side of a `LATERAL` join references columns from the left side, creating a data dependency that the enumerator's bitmask representation cannot express.
- **Table-valued functions** — function calls like `generate_series()` or `unnest()` may depend on correlated outer values and must remain in their original position.
- **Subquery inputs** — derived tables (`FROM (SELECT ...) AS t`) may contain correlated references or side effects.
The optimizer guards against incorrect reordering by inspecting the join chain before invoking DPccp. It walks the left-spine of nested inner joins and checks two conditions:
1. All joins in the chain must be `INNER` joins (outer and semi-joins have non-commutative semantics).
2. Every right-side input of each join must be a plain `LogicalScan` node.
If either condition is violated, the optimizer skips multi-way enumeration and falls back to pairwise join optimization, which preserves the original tree shape and its associated evaluation order.
## 9.8 Access Path Selection
Access path selection determines how to read data from tables—full scan, index scan, or index-only scan.
### 9.8.1 Access Path Types
```cpp
enum class AccessPathType {
kSeqScan, // Full table scan
kIndexScan, // Index scan with heap fetch
kIndexOnlyScan, // Covering index scan
kPrimaryKeyLookup // Direct point lookup
};
```
### 9.8.2 Access Path Enumeration
The `AccessPathEnumerator` generates all viable access paths:
```cpp
auto AccessPathEnumerator::enumerate(const ast::Expression* filter)
-> std::vector {
auto paths = std::vector {};
// Sequential scan is always available
paths.push_back(create_seq_scan_());
// Check for primary key equality
if (is_pk_equality_(filter)) {
paths.push_back(create_pk_lookup_(filter));
}
// Check each available index
for (const auto& index : indexes_) {
if (auto bounds = extract_index_bounds_(filter, index)) {
paths.push_back(create_index_scan_(index, *bounds));
// Check for covering index
if (is_covering_index_(index)) {
paths.push_back(create_index_only_scan_(index, *bounds));
}
}
}
// Sort by cost
std::sort(paths.begin(), paths.end(),
[](const auto& a, const auto& b) {
return a.cost.total() < b.cost.total();
});
return paths;
}
```
### 9.8.3 Cost Comparison
**Sequential Scan**:
- Cost: Proportional to table size
- Best when: High selectivity (reading most of table) or no suitable index
**Index Scan**:
- Cost: Index traversal + heap fetches
- Best when: Low selectivity (reading small fraction of table)
**Index-Only Scan**:
- Cost: Index traversal only
- Best when: Index contains all required columns (covering index)
**Primary Key Lookup**:
- Cost: Single point lookup
- Best when: Equality predicate on primary key
**Break-Even Analysis**:
For an index scan to beat a sequential scan:
$$
|R| \times sel \times (C_{idx} + C_{heap}) < |R| \times C_{seq}
$$
Solving for selectivity:
$$
sel < \frac{C_{seq}}{C_{idx} + C_{heap}}
$$
With typical constants ($C_{seq} = 1.0$, $C_{idx} = 0.1$, $C_{heap} = 4.0$):
$$
sel < \frac{1.0}{4.1} \approx 24\%
$$
Index scans typically win when selecting less than ~25% of rows.
## 9.9 Strategy Selection
After determining access paths, the optimizer selects algorithms for each operator.
### 9.9.1 Sort Strategy Selection
```cpp
enum class SortStrategy {
kNoSort, // Input already sorted
kTopKHeap, // Heap-based for small limits
kInMemorySort, // QuickSort in memory
kExternalSort, // External merge sort
kIndexScan // Use index ordering
};
```
**Selection Logic**:
```cpp
auto SortStrategySelector::select(const SortConfig& config) -> SortStrategy {
// Check if input is already sorted
if (is_sorted_by_(config.input_ordering, config.sort_keys)) {
return SortStrategy::kNoSort;
}
// Check for index providing order
for (const auto& idx : config.available_indexes) {
if (provides_ordering_(idx, config.sort_keys)) {
return SortStrategy::kIndexScan;
}
}
// Check for TopK optimization
if (config.limit_hint && *config.limit_hint <= kTopKThreshold) {
return SortStrategy::kTopKHeap;
}
// Choose between in-memory and external
auto data_size = config.input_rows * config.row_width;
if (data_size <= config.memory_budget) {
return SortStrategy::kInMemorySort;
}
return SortStrategy::kExternalSort;
}
```
### 9.9.2 Join Strategy Selection
```cpp
enum class JoinStrategy {
kHashJoin, // Hash build/probe
kIndexNestedLoop, // Index lookup on inner
kMergeJoin, // Sort-merge
kNestedLoop // Simple nested loop
};
```
**Selection Logic**:
```cpp
auto JoinStrategySelector::select(const JoinConfig& config) -> JoinStrategy {
// Check for index nested loop (small outer, index on inner)
if (config.left_rows <= kINLOuterThreshold) {
for (const auto& idx : config.right_indexes) {
if (matches_join_keys_(idx, config.join_keys)) {
return JoinStrategy::kIndexNestedLoop;
}
}
}
// Check if both sides are sorted on join keys
if (is_sorted_on_(config.left_ordering, config.join_keys) &&
is_sorted_on_(config.right_ordering, config.join_keys)) {
return JoinStrategy::kMergeJoin;
}
// Default to hash join
auto smaller_side = std::min(config.left_rows, config.right_rows);
auto hash_size = smaller_side * config.row_width * kHashTableOverhead;
if (hash_size <= config.memory_budget) {
return JoinStrategy::kHashJoin;
}
// Fallback to nested loop (should be rare)
return JoinStrategy::kNestedLoop;
}
```
### 9.9.3 Aggregate Strategy Selection
```cpp
enum class AggregateStrategy {
kStreamAggregate, // Streaming on sorted input
kHashAggregate, // Hash-based grouping
kSortAggregate // Sort then stream
};
```
**Selection Logic**:
```cpp
auto AggregateStrategySelector::select(const AggregateConfig& config)
-> AggregateStrategy {
// Check if input is sorted on group keys
if (is_sorted_on_(config.input_ordering, config.group_keys)) {
return AggregateStrategy::kStreamAggregate;
}
// Estimate hash table size
auto groups = config.estimated_groups.value_or(
config.input_rows * kDefaultGroupReduction);
auto hash_size = groups * kBytesPerGroupState;
if (hash_size <= config.memory_budget) {
return AggregateStrategy::kHashAggregate;
}
// Sort then stream
return AggregateStrategy::kSortAggregate;
}
```
## 9.10 Memory Budget Allocation
The final optimization phase allocates memory budgets to operators and predicts spill behavior.
### 9.10.1 Budget Distribution
Memory is distributed proportionally to operator data requirements:
```cpp
void MemoryBudgetAllocator::allocate(PhysicalPlan& plan) {
// Identify memory-consuming operators
auto blocking_ops = find_blocking_operators_(plan);
// Calculate total data volume
auto total_data = 0.0;
for (const auto& op : blocking_ops) {
total_data += op->properties().cardinality.memory_estimate();
}
// Distribute budget proportionally
for (auto& op : blocking_ops) {
auto data_size = op->properties().cardinality.memory_estimate();
auto weight = data_size / total_data;
auto budget = static_cast(weight * total_budget_);
// Ensure minimum budget
budget = std::max(budget, kMinBudget);
op->properties().memory_budget = budget;
op->properties().will_spill = (data_size > budget);
}
}
```
### 9.10.2 Spill Prediction
When an operator's estimated data exceeds its budget, spill is predicted:
```cpp
struct OperatorProperties {
CardinalityEstimate cardinality;
std::vector ordering;
size_t memory_budget;
bool will_spill;
std::optional spill_options;
bool preserves_ordering;
};
```
Spill options configure the spill behavior:
```cpp
struct SpillOptions {
std::string spill_directory;
size_t spill_block_size = 64 * 1024;
CompressionType compression = CompressionType::kLZ4;
};
```
### 9.10.3 Minimum Budget Guarantee
Each operator receives at least 16 MB:
```cpp
static constexpr size_t kMinBudget = 16 * 1024 * 1024;
```
This ensures operators can make progress even with limited memory. Below this threshold, the constant overhead of spilling exceeds any benefit from reduced memory usage.
## 9.11 Optimizer Configuration
The optimizer behavior is controlled by configuration options:
```cpp
struct OptimizerConfig {
// Memory configuration
size_t memory_budget = 256 * 1024 * 1024; // 256 MB default
std::string spill_directory;
// Optimization passes
uint32_t max_optimization_passes = 10;
// Feature flags
bool enable_filter_pushdown = true;
bool enable_topk_optimization = true;
bool enable_index_selection = true;
bool enable_cost_based_join = true;
// Strategy forcing (for testing)
std::optional force_sort_strategy;
std::optional force_join_strategy;
std::optional force_aggregate_strategy;
};
```
**Feature Flags**: Individual optimizations can be disabled for debugging or when they cause regressions.
**Strategy Forcing**: For testing and debugging, specific strategies can be forced regardless of cost estimates.
## 9.12 Optimization Statistics
The optimizer collects detailed statistics about its operation:
```cpp
struct OptimizerStats {
// Timing (microseconds)
int64_t parse_time_us;
int64_t logical_build_time_us;
int64_t rule_optimization_time_us;
int64_t cardinality_estimation_time_us;
int64_t physical_build_time_us;
int64_t total_time_us;
// Rule application
uint32_t rules_applied;
uint32_t filters_pushed_down;
uint32_t topk_optimizations;
// Plan characteristics
double estimated_rows;
double estimated_cost;
std::string selected_access_path;
std::string selected_sort_strategy;
std::string selected_join_strategy;
std::string selected_aggregate_strategy;
// Memory
size_t total_memory_allocated;
size_t operators_will_spill;
};
```
These statistics enable:
1. Performance monitoring of the optimizer itself
2. Plan explanation for users
3. Regression detection in optimizer changes
4. Debugging of plan quality issues
## 9.13 Summary
Cognica's logical planning and optimization system implements a sophisticated multi-phase pipeline:
1. **Logical Plan Construction**: Converts SQL AST to relational algebra tree with support for subqueries, table-valued functions, UNNEST, and expression-based LIMIT/OFFSET.
2. **Rule-Based Optimization**: Applies algebraic transformations (filter merging, pushdown, predicate ordering, TopK optimization).
3. **Plan Optimization**: An eight-pass pipeline that simplifies predicates, estimates selectivity, pushes down predicates, unnests correlated subqueries into joins, eliminates common subexpressions, expands OR-linked join predicates, reorders multi-way joins, and prunes unused columns.
4. **Cardinality Estimation**: Estimates result sizes using histograms and statistics.
5. **Physical Planning**: Selects access paths and execution algorithms.
6. **Memory Allocation**: Distributes memory budgets and predicts spills.
Key innovations include:
- **Damped Selectivity**: Uses geometric mean for correlated predicates
- **DPccp Join Ordering**: Optimal bushy tree generation without cross products, with a safety guard that preserves lateral, table function, and subquery inputs
- **Subquery Unnesting**: Converts correlated EXISTS/IN/NOT EXISTS subqueries into semi-joins and anti-joins, supporting table references, derived tables, and table-valued functions as subquery sources
- **Common Subexpression Elimination**: Factors out repeated expression subtrees to avoid redundant computation
- **Multi-Dimensional Cost Model**: Balances CPU, I/O, memory, and spill costs
- **Integrated Spill Prediction**: Memory-aware planning from the start
The optimizer balances plan quality against optimization time, using heuristics where exact solutions are intractable while guaranteeing optimality for critical decisions like join ordering.
# Chapter 10: Physical Planning and Execution Strategies
## 10.1 From Logical to Physical Plans
The transition from logical to physical planning marks a critical boundary in query processing. While logical plans describe *what* operations to perform using abstract relational algebra, physical plans specify *how* to execute those operations using concrete algorithms and data structures.
This chapter explores Cognica's physical planning system, which transforms optimized logical plans into executable physical plans by making three fundamental decisions:
1. **Access Path Selection**: How to retrieve data (sequential scan, index scan, or point lookup)
2. **Algorithm Selection**: Which algorithm to use for each operation (hash join vs. merge join, quicksort vs. external sort)
3. **Resource Allocation**: How to distribute memory budgets and handle spill to disk
### 10.1.1 Physical Plan Properties
A physical plan differs from a logical plan in several important ways:
| Property | Logical Plan | Physical Plan |
|----------|--------------|---------------|
| Abstraction | Relational algebra | Executable algorithms |
| Cost awareness | None | Full cost model |
| Memory awareness | None | Budget allocation |
| Ordering | Implicit | Explicit tracking |
| Parallelism | Unspecified | Execution model defined |
Physical operators carry rich metadata that guides execution:
```cpp
struct OperatorProperties {
CardinalityEstimate cardinality {}; // Expected output rows
std::vector ordering {}; // Output sort order
size_t memory_budget = 0; // Allocated memory
bool will_spill = false; // Spill prediction
std::optional spill_options {};
bool preserves_ordering = false; // Ordering guarantee
};
```
### 10.1.2 Physical Planning Architecture
The physical planning process follows a systematic flow:
```mermaid
flowchart TB
subgraph Input
LP[Logical Plan]
STATS[(Statistics)]
CONFIG[Config]
end
subgraph PhysicalPlanBuilder
APE[Access Path
Enumerator]
SS[Strategy
Selectors]
CE[Cost
Estimator]
MBA[Memory Budget
Allocator]
end
subgraph Output
PP[Physical Plan]
end
LP --> APE
STATS --> APE
STATS --> CE
CONFIG --> MBA
APE --> SS
CE --> SS
SS --> MBA
MBA --> PP
style Input fill:#e3f2fd
style PhysicalPlanBuilder fill:#fff3e0
style Output fill:#e8f5e9
```
## 10.2 Physical Operator Types
Cognica defines a comprehensive set of physical operators, each representing a specific execution algorithm.
### 10.2.1 Operator Classification
Physical operators are classified by their function:
```cpp
enum class PhysicalOpType : uint8_t {
// === Scan Operators ===
kSeqScan, // Full collection sequential scan
kIndexScan, // B-tree index range scan with heap fetch
kIndexOnlyScan, // Covering index scan (no heap fetch)
kPrimaryKeyLookup, // Direct point lookup by primary key
// === Filter Operators ===
kFilter, // Interpreted predicate evaluation
kBytecodeFilter, // CVM bytecode compiled filter
// === Sort Operators ===
kSort, // In-memory quicksort
kTopKSort, // Heap-based top-K selection
kExternalSort, // External merge sort (disk spill)
kIndexSort, // Leverage index ordering
// === Limit/Skip ===
kLimit, // Output row cap (constant or expression)
// === Aggregate Operators ===
kHashAggregate, // Hash-based grouping
kSortAggregate, // Sort-based grouping
kStreamAggregate, // Streaming on sorted input
// === Join Operators ===
kHashJoin, // Build/probe hash join
kIndexNestedLoop, // Index lookup on inner table
kNestedLoopJoin, // Simple nested loop
kMergeJoin, // Sort-merge join
// === Set Operators ===
kUnionAll, // Concatenate without dedup
kHashUnion, // Union with hash deduplication
// === Transform Operators ===
kProject, // Field selection/transformation
kLiteral, // Constant field injection
// === Subquery and Table Function Operators ===
kSubquery, // Derived table (inline view)
kTableFunc, // Table-valued function (e.g., generate_series)
kUnnest, // Array unnesting to rows
// === Search Operators ===
kFullTextSearch, // Full-text search
kVectorSearch // Vector similarity search
};
```
### 10.2.2 Operator Hierarchy
All physical operators inherit from a common base class:
```mermaid
classDiagram
class PhysicalOperator {
<>
+type() PhysicalOpType
+cost() Cost
+to_cursor(ctx) Cursor
+clone() PhysicalOpPtr
+explain() string
+properties() OperatorProperties
+children() vector~PhysicalOpPtr~
}
class SeqScanOp {
+collection_name: string
+scan_filter: Expression
}
class IndexScanOp {
+index_name: string
+start_key: KeyBound
+end_key: KeyBound
+direction: ScanDirection
}
class HashJoinOp {
+join_type: JoinType
+join_condition: Expression
+build_from_right: bool
}
class PhysicalSortOp {
+sort_keys: vector~SortKey~
}
class HashAggregateOp {
+group_keys: vector~Expression~
+aggregates: vector~AggregateSpec~
}
PhysicalOperator <|-- SeqScanOp
PhysicalOperator <|-- IndexScanOp
PhysicalOperator <|-- HashJoinOp
PhysicalOperator <|-- PhysicalSortOp
PhysicalOperator <|-- HashAggregateOp
```
### 10.2.3 Scan Operators
Scan operators provide the foundation for data access:
**Sequential Scan** (`SeqScanOp`):
- Reads all documents in a collection
- Optionally applies a pushed-down filter
- Cost: $O(n)$ where $n$ is collection size
- Best when: High selectivity or no suitable index
**Index Scan** (`IndexScanOp`):
- Uses B-tree index to locate matching documents
- Fetches documents from heap storage
- Cost: $O(k \log n + k)$ where $k$ is result size
- Best when: Low selectivity with matching index
**Index-Only Scan** (`IndexOnlyScanOp`):
- Reads data directly from index (covering index)
- No heap fetch required
- Cost: $O(k \log n)$
- Best when: Index contains all required columns
**Primary Key Lookup** (`PrimaryKeyLookupOp`):
- Direct point lookup by primary key
- Cost: $O(1)$ per lookup
- Best when: Equality predicate on primary key
### 10.2.4 Join Operators
Join operators combine data from multiple sources:
**Hash Join** (`HashJoinOp`):
```
Phase 1 (Build): Hash smaller relation into hash table
Phase 2 (Probe): Scan larger relation, probe hash table
```
- Build cost: $O(|R|)$
- Probe cost: $O(|S|)$
- Memory: $O(\min(|R|, |S|))$
- Best when: No useful indexes, adequate memory
**Index Nested Loop** (`IndexNestedLoopOp`):
```
For each row in outer:
Look up matching rows via index on inner
```
- Cost: $O(|R| \times \log |S|)$
- Memory: $O(1)$
- Best when: Small outer table, index on inner join key
**Merge Join** (`MergeJoinOp`):
```
Advance both sorted inputs in lockstep
Emit matching pairs
```
- Cost: $O(|R| + |S|)$ (assuming pre-sorted)
- Memory: $O(1)$
- Best when: Both inputs already sorted on join key
**Nested Loop Join** (`NestedLoopJoinOp`):
```
For each row in outer:
For each row in inner:
If predicate matches, emit
```
- Cost: $O(|R| \times |S|)$
- Memory: $O(1)$
- Best when: Very small relations or fallback
### 10.2.5 Aggregate Operators
Aggregate operators compute grouped summaries:
**Hash Aggregate** (`HashAggregateOp`):
- Groups rows using hash table keyed by group columns
- Maintains aggregate state per group
- Cost: $O(n)$
- Memory: $O(g)$ where $g$ is number of groups
**Stream Aggregate** (`StreamAggregateOp`):
- Requires input sorted on group keys
- Processes groups in single pass
- Cost: $O(n)$
- Memory: $O(1)$ (single group state)
**Sort Aggregate** (`SortAggregateOp`):
- Sorts input, then streams
- Cost: $O(n \log n)$ for sort + $O(n)$ for aggregate
- Memory: $O(n)$ for sort
### 10.2.6 Sort Operators
Sort operators order result sets:
**In-Memory Sort** (`PhysicalSortOp`):
- QuickSort implementation
- Cost: $O(n \log n)$
- Memory: $O(n)$
- Best when: Data fits in memory
**Top-K Sort** (`TopKSortOp`):
- Heap-based selection
- Cost: $O(n \log k)$
- Memory: $O(k)$
- Best when: Small LIMIT clause
**External Sort** (`ExternalSortOp`):
- External merge sort with disk spill
- Cost: $O(n \log n)$ comparisons + I/O
- Memory: Configurable budget
- Best when: Data exceeds memory
**Index Sort** (`IndexSortOp`):
- Leverages index ordering
- Cost: $O(0)$ (no actual sort)
- Best when: Index provides required ordering
### 10.2.7 Subquery and Table Function Operators
**Subquery** (`PhysicalSubquery`):
The physical subquery operator wraps a complete physical plan that was produced from a derived table (`FROM (SELECT ...) AS t`). When the logical `SubqueryOp` carries column aliases (the `AS t(col1, col2)` syntax), the physical planner inserts a rename projection layer above the child plan. This rename layer is a `PhysicalProject` (or `BatchProject` in batch mode) whose target list maps each original output column to its alias name. This approach is more general than mutating column names on a specific child node type, because it works regardless of whether the child is a `PhysicalValues`, a sort, a join, or any other operator.
After inserting the optional rename layer, the subquery alias is propagated to the outermost physical node via `set_output_alias()`, which allows parent join operators to build `CompositeRow` objects with properly qualified field names.
**Table Function** (`PhysicalTableFunc`):
Table-valued functions such as `generate_series()`, `json_each()`, or graph query functions produce a set of rows from their arguments. The physical operator stores the function name, argument expressions, and declared output columns. At execution time, the operator evaluates its arguments, invokes the registered table function implementation, and produces rows through the standard iterator interface.
Table function operators participate in the transactional execution context. The `ExecutionContext` carries an optional `DocumentDB` handle and `DocumentDBTransaction` pointer so that write-capable table functions (such as Cypher mutation functions) can participate in the active SQL transaction instead of falling back to global database state.
**Unnest** (`PhysicalUnnest`):
The unnest operator expands an array-valued expression into a set of rows, one per element. It is the physical counterpart of `UNNEST(array_expr)` in the FROM clause.
### 10.2.8 Limit Operator
The `PhysicalLimit` operator supports both constant and expression-based limits. When the logical plan contains a constant `LIMIT` or `OFFSET`, the values are stored directly. When the value comes from a parameter or expression, the operator stores the unevaluated AST expression in `limit_expr` / `offset_expr` and evaluates it at `open()` time to produce an effective limit and offset. This allows parameterized queries like `LIMIT $1 OFFSET $2` to work correctly without recompilation.
## 10.3 Execution Models
Cognica implements a hybrid execution architecture supporting three execution models, each optimized for different scenarios.
### 10.3.1 Iterator (Volcano) Model
The primary execution model follows the classic Volcano iterator pattern:
```mermaid
sequenceDiagram
participant Client
participant Project
participant Filter
participant Scan
Client->>Project: next()
Project->>Filter: next()
Filter->>Scan: next()
Scan-->>Filter: row1
Filter-->>Project: row1
Project-->>Client: projected_row1
Client->>Project: next()
Project->>Filter: next()
Filter->>Scan: next()
Scan-->>Filter: row2
Note over Filter: row2 fails predicate
Filter->>Scan: next()
Scan-->>Filter: row3
Filter-->>Project: row3
Project-->>Client: projected_row3
```
Each physical operator implements the `Cursor` interface:
```cpp
class Cursor {
public:
virtual auto next() -> Document* = 0;
virtual auto has_next() const -> bool = 0;
virtual void close() = 0;
};
```
The `to_cursor()` method converts a physical operator to an executable cursor:
```cpp
class PhysicalOperator {
public:
virtual auto to_cursor(ExecutionContext& ctx) -> std::unique_ptr = 0;
};
```
**Cursor Implementations**:
| Cursor Type | Purpose |
|-------------|---------|
| `ScanCursor` | Iterates over collection |
| `FilterCursor` | Applies predicate to input |
| `ProjectCursor` | Transforms fields |
| `SortCursor` | Buffers and sorts input |
| `LimitCursor` | Caps output count |
| `HashJoinOperator` | Executes hash join |
| `UnionCursor` | Combines multiple inputs |
**Advantages**:
- Simple, composable design
- Low memory overhead for streaming operations
- Natural pipelining of operators
**Disadvantages**:
- High per-row overhead from virtual function calls
- Poor cache utilization (row-at-a-time processing)
### 10.3.2 CVM Bytecode Execution
For SQL queries, Cognica compiles physical plans to CVM (Cognica Virtual Machine) bytecode:
```mermaid
flowchart LR
PP[Physical Plan] --> PL[PlanLowering]
PL --> BC[BytecodeModule]
BC --> INT[Interpreter]
INT --> RES[Results]
style PP fill:#e3f2fd
style BC fill:#fff3e0
style RES fill:#e8f5e9
```
**Bytecode Generation Example**:
For a simple filter query:
```sql
SELECT name FROM users WHERE age > 21
```
Generated bytecode:
```
CURSOR_OPEN slot_0, "users"
LABEL loop:
CURSOR_NEXT R0, slot_0
JUMP_NULL R0, done
; Filter: age > 21
FIELD_GET R1, R0, "age"
LOAD_CONST R2, 21
CMP_GT R3, R1, R2
JUMP_IF_FALSE R3, loop
; Project: name
FIELD_GET R4, R0, "name"
EMIT_ROW R4
JUMP loop
LABEL done:
CURSOR_CLOSE slot_0
HALT
```
**Lowering Result Structure**:
```cpp
struct PlanLoweringResult {
bool success = false;
std::unique_ptr module;
std::vector output_columns;
uint8_t cursor_slots_used = 0;
uint8_t registers_used = 0;
// Index queries for cursor initialization
std::unordered_map index_queries;
// Registered subqueries
std::vector> subqueries;
// Optimization hints for external tables
std::unordered_map limit_hints;
std::unordered_map offset_hints;
};
```
**Register Allocation**:
The lowering phase manages a limited set of virtual registers:
```cpp
class PlanLowering {
private:
static constexpr size_t kMaxRegisters = 32;
std::bitset register_in_use_;
std::bitset register_reserved_;
std::deque allocation_order_; // FIFO for eviction
uint8_t next_cursor_slot_ = 0;
auto allocate_register_() -> uint8_t;
auto allocate_cursor_slot_() -> uint8_t;
auto evict_register_() -> uint8_t; // Spill to stack if needed
};
```
**Advantages**:
- Eliminates virtual function overhead
- Computed-goto dispatch for fast interpretation
- Amenable to JIT compilation
### 10.3.3 Vectorized Execution
For columnar data processing, Cognica supports vectorized execution using Apache Arrow:
```cpp
class VectorizedContext {
auto get_batch(uint8_t reg) -> ColumnBatch*;
auto get_column(uint8_t batch_reg, size_t col_idx) -> ColumnData*;
};
class ColumnBatch {
size_t row_count_;
std::vector columns_;
};
```
Vectorized operations process data in batches (typically 1024-4096 rows), enabling:
- SIMD instruction utilization
- Better cache locality
- Reduced interpretation overhead
**Vectorized Lowering Methods**:
```cpp
class PlanLowering {
void lower_seq_scan_vectorized_(const SeqScanOp* op);
void lower_filter_vectorized_(const FilterOp* op);
void lower_project_vectorized_(const ProjectOp* op);
void lower_batch_hash_join_(const HashJoinOp* op);
void lower_batch_sort_(const SortOp* op);
};
```
## 10.4 Physical Plan Builder
The `PhysicalPlanBuilder` transforms logical plans into physical plans through a recursive descent process.
### 10.4.1 Builder Configuration
```cpp
class PhysicalPlanBuilder final {
public:
struct Config {
std::string collection_name;
const IndexDescriptor* index_desc = nullptr;
const IndexStatisticsManager* stats_mgr = nullptr;
const LSMCostModel* lsm_cost = nullptr;
int64_t total_rows = 0;
double avg_row_width = 512.0;
size_t memory_budget = 256 * 1024 * 1024; // 256 MB
std::vector output_fields {};
};
explicit PhysicalPlanBuilder(Config config);
auto build(const LogicalPlan& logical_plan) -> PhysicalPlan;
private:
auto build_(const LogicalOperator* op) -> PhysicalOpPtr;
auto build_scan_(const ScanOp* scan) -> PhysicalOpPtr;
auto build_filter_(const FilterOp* filter) -> PhysicalOpPtr;
auto build_sort_(const SortOp* sort) -> PhysicalOpPtr;
auto build_group_(const GroupOp* group) -> PhysicalOpPtr;
auto build_join_(const JoinOp* join) -> PhysicalOpPtr;
};
```
### 10.4.2 Recursive Plan Construction
The builder processes the logical plan tree bottom-up:
```cpp
auto PhysicalPlanBuilder::build_(const LogicalOperator* op) -> PhysicalOpPtr {
switch (op->type()) {
case LogicalOpType::kScan:
return build_scan_(static_cast(op));
case LogicalOpType::kFilter:
return build_filter_(static_cast(op));
case LogicalOpType::kSort:
return build_sort_(static_cast(op));
case LogicalOpType::kGroup:
return build_group_(static_cast(op));
case LogicalOpType::kJoin:
return build_join_(static_cast(op));
// ... other operator types
}
}
```
### 10.4.3 Access Path Selection
For scan operators, the builder enumerates and costs all viable access paths:
```cpp
auto PhysicalPlanBuilder::build_scan_(const ScanOp* scan) -> PhysicalOpPtr {
auto enumerator = AccessPathEnumerator {
.collection_name = config_.collection_name,
.index_desc = config_.index_desc,
.stats_mgr = config_.stats_mgr,
.lsm_cost = config_.lsm_cost,
.total_rows = config_.total_rows,
.avg_row_width = config_.avg_row_width,
.output_fields = config_.output_fields
};
// Get filter from parent if available
auto filter = get_pushed_filter_();
// Select lowest-cost access path
auto best_path = enumerator.select_best(filter);
return best_path.op;
}
```
### 10.4.4 Strategy Selection Integration
For blocking operators, the builder consults strategy selectors:
```cpp
auto PhysicalPlanBuilder::build_sort_(const SortOp* sort) -> PhysicalOpPtr {
// Build child first
auto child = build_(sort->children()[0].get());
auto child_props = child->properties();
// Configure strategy selection
auto sort_config = SortConfig {
.input_rows = child_props.cardinality.rows,
.row_width = child_props.cardinality.row_width,
.sort_keys = sort->sort_keys(),
.limit_hint = current_limit_hint_,
.memory_budget = remaining_memory_budget_,
.input_ordering = child_props.ordering,
.available_indexes = get_available_indexes_()
};
// Select strategy
auto strategy = SortStrategySelector::select(sort_config);
// Create appropriate physical operator
switch (strategy) {
case SortStrategy::kNoSort:
return child; // Already sorted
case SortStrategy::kTopKHeap:
return std::make_unique(
*current_limit_hint_, sort->sort_keys(), std::move(child));
case SortStrategy::kInMemorySort:
return std::make_unique(
sort->sort_keys(), std::move(child));
case SortStrategy::kExternalSort:
return std::make_unique(
sort->sort_keys(), std::move(child), spill_config_);
case SortStrategy::kIndexScan:
return create_index_sort_(sort->sort_keys());
}
}
```
### 10.4.5 Scan Projection Pushdown
After the physical plan tree is constructed, the planner performs a top-down pass to propagate column references down to scan operators. The goal is to ensure that each scan reads only the columns that are actually needed by the operators above it, reducing I/O and memory consumption.
The `set_scan_projections_()` method walks the physical plan tree and accumulates the set of referenced columns at each operator by inspecting:
- **Filter** operators: columns referenced in the predicate expression.
- **Project** operators: columns referenced in the target list.
- **Join** operators (Hash, NestedLoop, Merge): columns referenced in the join condition.
- **Sort** operators: columns referenced in ORDER BY expressions.
- **Aggregate** operators (Hash, Stream): columns in GROUP BY keys, aggregate function arguments, aggregate ordering clauses, and HAVING predicates.
- **DistinctOn** operators: columns in the DISTINCT ON expression list.
- **Unnest** operators: columns referenced in the unnest expression.
- **TableFunc** operators: columns referenced in function arguments.
When the pass reaches a `PhysicalSeqScan` or `PhysicalIndexScan`, it sets the scan's projection list to the accumulated column set.
**LATERAL Dependency Propagation**: Lateral joins introduce a special challenge for projection pushdown. The right side of a lateral join may reference columns from the left side that do not appear in the join condition itself. To handle this, the planner uses a separate `collect_plan_referenced_columns_()` method that recursively collects all column references from the entire right subtree of a lateral nested-loop join. These columns are added to the needed-columns set before the pass descends into the left child, ensuring that the left scan includes all columns the right side depends on.
## 10.5 Access Path Enumeration
Access path enumeration generates all viable ways to read data from a table.
### 10.5.1 Access Path Structure
```cpp
struct AccessPath {
PhysicalOpPtr op; // The physical operator
optimizer::Cost cost; // Estimated cost
bool is_covering; // True if index-only scan possible
};
```
### 10.5.2 Enumeration Process
The `AccessPathEnumerator` systematically considers each access method:
```cpp
auto AccessPathEnumerator::enumerate(const ast::Expression* filter)
-> std::vector {
auto paths = std::vector {};
// Sequential scan is always available (fallback)
paths.push_back(create_seq_scan_());
// Check for primary key equality condition
if (is_pk_equality_(filter)) {
paths.push_back(create_pk_lookup_(filter));
}
// Consider each available index
for (const auto& index : indexes_) {
// Extract bounds that match this index
if (auto bounds = extract_index_bounds_(filter, index)) {
// Standard index scan (with heap fetch)
paths.push_back(create_index_scan_(index, *bounds));
// Index-only scan (if covering)
if (is_covering_index_(index)) {
paths.push_back(create_index_only_scan_(index, *bounds));
}
}
}
// Sort by cost (cheapest first)
std::sort(paths.begin(), paths.end(),
[](const auto& a, const auto& b) {
return a.cost.total() < b.cost.total();
});
return paths;
}
```
### 10.5.3 Index Bound Extraction
The enumerator analyzes filter predicates to extract index bounds:
```cpp
auto AccessPathEnumerator::extract_index_bounds_(
const ast::Expression* filter,
const IndexInfo& index) -> std::optional {
if (!filter) {
return std::nullopt;
}
auto bounds = IndexBounds {};
// Collect equality conditions on index columns
auto equalities = collect_equality_fields_(filter);
// Match against index prefix
for (const auto& idx_col : index.columns) {
if (auto it = equalities.find(idx_col); it != equalities.end()) {
bounds.add_equality(idx_col, it->second);
} else {
break; // Index prefix broken
}
}
// Check for range condition on next column
if (bounds.prefix_length() < index.columns.size()) {
auto next_col = index.columns[bounds.prefix_length()];
if (auto range = extract_range_(filter, next_col)) {
bounds.set_range(*range);
}
}
return bounds.is_useful() ? std::optional(bounds) : std::nullopt;
}
```
### 10.5.4 Cost Comparison
The cost model determines which access path wins:
**Sequential Scan Cost**:
$$
C_{seq} = |R| \times (cpu_{tuple} + \frac{io_{page}}{tuples_{per\_page}})
$$
**Index Scan Cost**:
$$
C_{idx} = |R| \times sel \times (cpu_{tuple} + io_{random} + io_{heap})
$$
**Index-Only Scan Cost**:
$$
C_{idx\_only} = |R| \times sel \times (cpu_{tuple} + io_{random})
$$
**Primary Key Lookup Cost**:
$$
C_{pk} = k \times (cpu_{lookup} + io_{point})
$$
Where $k$ is the number of keys to look up.
**Break-Even Analysis**:
Index scan beats sequential scan when:
$$
sel < \frac{cpu_{tuple} + io_{seq}}{cpu_{tuple} + io_{random} + io_{heap}}
$$
With typical values:
$$
sel < \frac{0.01 + 0.1}{0.01 + 4.0 + 0.5} \approx 2.4\%
$$
For highly selective queries (< ~2.5% of rows), index scans typically win.
## 10.6 Strategy Selection
Strategy selectors choose the best algorithm for each operator type based on input characteristics, available resources, and physical properties.
### 10.6.1 Sort Strategy Selection
```cpp
enum class SortStrategy {
kNoSort, // Input already sorted
kTopKHeap, // Heap-based for small LIMIT
kInMemorySort, // QuickSort in memory
kExternalSort, // External merge sort
kIndexScan // Use index ordering
};
```
**Selection Logic**:
```mermaid
flowchart TB
START([Sort Required]) --> CHECK_SORTED{Input
Already Sorted?}
CHECK_SORTED -->|Yes| NO_SORT[kNoSort]
CHECK_SORTED -->|No| CHECK_INDEX{Index Provides
Ordering?}
CHECK_INDEX -->|Yes| INDEX_SORT[kIndexScan]
CHECK_INDEX -->|No| CHECK_LIMIT{LIMIT <= 1000?}
CHECK_LIMIT -->|Yes| TOPK[kTopKHeap]
CHECK_LIMIT -->|No| CHECK_MEM{Data Fits
in Memory?}
CHECK_MEM -->|Yes| IN_MEM[kInMemorySort]
CHECK_MEM -->|No| EXTERNAL[kExternalSort]
style NO_SORT fill:#c8e6c9
style INDEX_SORT fill:#c8e6c9
style TOPK fill:#bbdefb
style IN_MEM fill:#bbdefb
style EXTERNAL fill:#ffcdd2
```
**Algorithm Complexity Comparison**:
| Strategy | Time | Space | I/O |
|----------|------|-------|-----|
| TopK Heap | $O(n \log k)$ | $O(k)$ | 0 |
| In-Memory | $O(n \log n)$ | $O(n)$ | 0 |
| External | $O(n \log n)$ | $O(B)$ | $O(\frac{n}{B} \log_M \frac{n}{B})$ |
Where $B$ is buffer size and $M$ is merge width.
### 10.6.2 Join Strategy Selection
```cpp
enum class JoinStrategy {
kHashJoin, // Build/probe hash join
kIndexNestedLoop, // Index lookup on inner
kMergeJoin, // Sort-merge join
kNestedLoop // Simple nested loop
};
```
**Selection Logic**:
```mermaid
flowchart TB
START([Join Required]) --> CHECK_INL{Outer Small?
Index on Inner?}
CHECK_INL -->|Yes| INL[kIndexNestedLoop]
CHECK_INL -->|No| CHECK_SORTED{Both Inputs
Sorted on Keys?}
CHECK_SORTED -->|Yes| MERGE[kMergeJoin]
CHECK_SORTED -->|No| CHECK_MEM{Hash Table
Fits in Memory?}
CHECK_MEM -->|Yes| HASH[kHashJoin]
CHECK_MEM -->|No| GRACE{Use Grace
Hash Join}
GRACE --> HASH_SPILL[kHashJoin
with Spill]
style INL fill:#c8e6c9
style MERGE fill:#c8e6c9
style HASH fill:#bbdefb
style HASH_SPILL fill:#ffcdd2
```
**Threshold Constants**:
```cpp
static constexpr double kINLOuterThreshold = 1000.0;
static constexpr double kHashBuildCostPerRow = 1.0;
static constexpr double kHashProbeCostPerRow = 1.2;
static constexpr double kNestLoopCostPerRow = 10.0;
static constexpr double kSortMergeCostPerRow = 2.0;
```
### 10.6.3 Aggregate Strategy Selection
```cpp
enum class AggregateStrategy {
kStreamAggregate, // Input sorted on group keys
kHashAggregate, // Hash-based grouping
kSortAggregate // Sort then stream
};
```
**Selection Logic**:
```cpp
auto AggregateStrategySelector::select(const AggConfig& config) -> AggregateStrategy {
// If input already sorted on group keys, stream is optimal
if (is_sorted_on_(config.input_ordering, config.group_keys)) {
return AggregateStrategy::kStreamAggregate;
}
// Estimate hash table memory requirement
auto estimated_groups = config.estimated_groups.value_or(
config.input_rows * kDefaultGroupReduction);
auto hash_memory = estimated_groups * kBytesPerGroupState;
// Hash aggregate if fits in memory
if (hash_memory <= config.memory_budget) {
return AggregateStrategy::kHashAggregate;
}
// Sort-aggregate for large group counts
return AggregateStrategy::kSortAggregate;
}
```
**Memory Constants**:
```cpp
static constexpr size_t kBytesPerGroupState = 256; // Per-group state
static constexpr double kDefaultGroupReduction = 0.1; // 10% reduction
```
## 10.7 Memory Management
Memory management is critical for physical execution. Cognica implements sophisticated memory budgeting with automatic spill handling.
### 10.7.1 Memory Budget Allocation
The `MemoryBudgetAllocator` distributes memory across operators:
```cpp
struct OperatorMemoryBudget {
size_t allocated = 0;
bool will_spill = false;
double spill_fraction = 0.0;
SpillOptions spill_options {};
};
class MemoryBudgetAllocator final {
public:
auto allocate(const ParsedPipeline& pipeline,
const std::vector& estimates)
-> std::vector;
private:
size_t total_budget_;
std::string spill_directory_;
};
```
**Allocation Strategy**:
1. **Identify Blocking Operators**: Sort, Join, Aggregate require memory
2. **Calculate Requirements**: Estimate memory per operator
3. **Weighted Distribution**: Allocate proportionally with priority weighting
4. **Apply Constraints**: Enforce minimum (16 MB) and maximum bounds
5. **Configure Spill**: Set up spill options for operators exceeding budget
**Priority Weights**:
```cpp
static constexpr double kSortPriority = 1.5; // Sort is memory-hungry
static constexpr double kGroupPriority = 1.2; // Groups vary widely
static constexpr double kJoinPriority = 1.0; // Base priority
```
### 10.7.2 Spill Configuration
When operators cannot fit data in memory, they spill to disk:
```cpp
struct SpillConfig {
size_t memory_limit = 256 * 1024 * 1024; // 256 MB
size_t spill_batch_size = 10000; // Rows per spill batch
int32_t max_merge_width = 16; // Merge fan-in
std::filesystem::path temp_directory {};
CompressionType compression = CompressionType::kNone;
};
```
### 10.7.3 Spillable Data Structures
Cognica implements spill-aware versions of key data structures:
**SpillableSortBuffer**:
- Implements external merge sort
- Writes sorted runs to disk when memory exceeded
- Merges runs during final output phase
**SpillableHashTable**:
- Uses Grace Hash Join partitioning
- Partitions data by hash of key
- Spills partitions independently
- Reloads partitions on demand during probe
**SpillableAggTable**:
- Hash aggregate with partition spill
- Maintains partial aggregates in memory
- Spills partitions when full
- Merges partitions during finalization
```mermaid
flowchart TB
subgraph Memory["In-Memory Phase"]
INSERT[Insert Rows] --> PARTITION[Hash Partition]
PARTITION --> CHECK{Memory
Exceeded?}
CHECK -->|No| INSERT
end
subgraph Spill["Spill Phase"]
CHECK -->|Yes| SELECT[Select Largest
Partition]
SELECT --> WRITE[Write to Disk]
WRITE --> CLEAR[Clear Memory]
CLEAR --> INSERT
end
subgraph Merge["Merge Phase"]
DONE([Input Complete]) --> LOAD[Load Partition]
LOAD --> PROCESS[Process Partition]
PROCESS --> MORE{More
Partitions?}
MORE -->|Yes| LOAD
MORE -->|No| OUTPUT[Output Results]
end
style Memory fill:#e8f5e9
style Spill fill:#fff3e0
style Merge fill:#e3f2fd
```
### 10.7.4 Memory Tracking
Each operator tracks its memory usage:
```cpp
class MemoryTracker {
public:
void allocate(size_t bytes);
void deallocate(size_t bytes);
auto current_usage() const -> size_t;
auto peak_usage() const -> size_t;
auto budget() const -> size_t;
auto should_spill() const -> bool;
private:
std::atomic current_ = 0;
std::atomic peak_ = 0;
size_t budget_ = 0;
};
```
## 10.8 Cost Estimation
The cost model guides all physical planning decisions.
### 10.8.1 Cost Structure
```cpp
struct Cost {
double cpu_cost = 0.0; // CPU cycles estimate
double io_cost = 0.0; // I/O operations
double memory_cost = 0.0; // Memory pressure
double spill_cost = 0.0; // Disk spill overhead
auto total() const -> double {
return cpu_cost +
io_cost * kIOCostWeight +
memory_cost * kMemoryCostWeight +
spill_cost * kSpillCostWeight;
}
};
```
**Weight Constants**:
```cpp
static constexpr double kIOCostWeight = 10.0; // I/O is expensive
static constexpr double kMemoryCostWeight = 0.1; // Memory is pressure
static constexpr double kSpillCostWeight = 5.0; // Spill is costly
```
The weights reflect modern hardware characteristics:
- Disk I/O is 10x more expensive than CPU operations
- Memory pressure translates to cache misses
- Spill incurs both I/O and CPU overhead
### 10.8.2 Cost Formulas
**Sequential Scan**:
$$
C_{seq} = n \times (cpu_{tuple} + \frac{io_{page}}{rows_{per\_page}})
$$
**Index Scan**:
$$
C_{idx} = k \times (cpu_{lookup} + io_{random} + io_{heap})
$$
Where $k = n \times selectivity$.
**Hash Join**:
$$
C_{hash} = |build| \times cpu_{hash} \times k_{build} + |probe| \times cpu_{hash} \times k_{probe}
$$
**Sort (In-Memory)**:
$$
C_{sort} = n \times \log_2(n) \times cpu_{cmp}
$$
**Sort (External)**:
$$
C_{ext} = C_{sort} + \frac{2 \times n \times width}{block} \times io_{spill} \times \lceil \log_M(runs) \rceil
$$
**Hash Aggregate**:
$$
C_{agg} = n \times cpu_{hash} + g \times cpu_{agg}
$$
Where $g$ is the number of groups.
### 10.8.3 Cost Constants
```cpp
// CPU costs
static constexpr double kCPUTupleProcessing = 0.01;
static constexpr double kCPUComparison = 0.0001;
static constexpr double kCPUHashOp = 0.0005;
// I/O costs
static constexpr double kSeqPageCost = 1.0;
static constexpr double kRandomPageCost = 4.0;
static constexpr double kHeapAccessCost = 0.5;
static constexpr double kSpillIOCost = 0.01; // Per byte
// Memory constants
static constexpr double kHashTableOverhead = 1.5;
static constexpr double kCompressionFactor = 0.5; // LZ4 typical
static constexpr size_t kMergeWidth = 16;
static constexpr size_t kDefaultBlockSize = 8192;
```
### 10.8.4 Spill Prediction
The cost estimator predicts when operations will exceed memory:
```cpp
auto CostEstimator::will_spill(int64_t rows,
double row_width,
size_t memory_budget) const -> bool {
auto data_size = static_cast(rows) * row_width;
return data_size > static_cast(memory_budget);
}
```
When spill is predicted, additional I/O cost is added:
$$
C_{spill} = \frac{data\_size}{block\_size} \times io_{spill} \times \frac{1}{compression}
$$
## 10.9 Physical Ordering
Physical operators track output ordering properties, enabling sort elimination and merge join selection.
### 10.9.1 Ordering Representation
```cpp
struct PhysicalOrdering {
std::string column;
SortOrder order; // kAscending or kDescending
};
```
An operator's output ordering is a list of `PhysicalOrdering` entries representing the sort guarantee.
### 10.9.2 Ordering Propagation
Different operators affect ordering differently:
| Operator | Ordering Effect |
|----------|-----------------|
| SeqScan | None (arbitrary order) |
| IndexScan | Index key order |
| Sort | Produces specified order |
| Filter | Preserves input order |
| Project | Preserves input order |
| HashJoin | Destroys order |
| MergeJoin | Preserves outer order |
| HashAggregate | Destroys order |
| StreamAggregate | Preserves group key order |
### 10.9.3 Ordering Satisfaction
The optimizer checks if an operator's output satisfies a required ordering:
```cpp
auto OperatorProperties::ordering_satisfies(
const std::vector& required) const -> bool {
if (required.size() > ordering.size()) {
return false;
}
for (size_t i = 0; i < required.size(); ++i) {
if (ordering[i].column != required[i].column ||
ordering[i].order != required[i].order) {
return false;
}
}
return true;
}
```
This enables sort elimination when input is already sorted.
## 10.10 Explain Plans
Physical plans can be explained for debugging and optimization analysis.
### 10.10.1 Explain Output
Each physical operator implements an `explain()` method:
```cpp
auto HashJoinOp::explain() const -> std::string {
auto ss = std::ostringstream {};
ss << "HashJoin [" << join_type_to_string(join_type_) << "]";
ss << " on (" << condition_to_string(join_condition_) << ")";
ss << " build_from=" << (build_from_right_ ? "right" : "left");
ss << " rows=" << properties_.cardinality.rows;
ss << " cost=" << cost_.total();
if (properties_.will_spill) {
ss << " [SPILL]";
}
return ss.str();
}
```
### 10.10.2 Plan Tree Visualization
```
HashJoin [INNER] on (orders.customer_id = customers.id)
build_from=right rows=50000 cost=12500.5
|-- SeqScan(orders) rows=100000 cost=1000.0
|-- IndexScan(customers) using pk_customers rows=10000 cost=500.0
range: [1, 10000]
```
### 10.10.3 Execution Statistics
After execution, operators report actual statistics:
```cpp
struct ExecutionStats {
int64_t rows_processed = 0;
int64_t rows_output = 0;
int64_t bytes_read = 0;
int64_t bytes_written = 0;
int64_t execution_time_us = 0;
int64_t memory_peak_bytes = 0;
int64_t spill_bytes = 0;
};
```
Comparing estimated vs. actual statistics helps identify cardinality estimation errors.
## 10.11 CVM Integration
Physical plans are lowered to CVM bytecode for efficient execution. However, certain plan shapes are not yet supported by the CVM lowering path and fall back to the Volcano (iterator) executor automatically.
**CVM Fallback Conditions**: The executor checks `should_use_cvm_for_plan()` before attempting CVM lowering. Even when the CVM is otherwise enabled, the executor falls back to the Volcano path when the physical plan contains table-valued function or UNNEST operators inside an outer (correlated) context. The CVM subquery execution path does not correctly preserve standalone table-function row production when an outer context is installed, which manifests as a single NULL row for shapes like `ARRAY(SELECT ... FROM unnest(...))`. The fallback check walks the physical plan tree looking for `kTableFunc` or `kUnnest` nodes and, if any are found while an outer context is active, routes the query to the iterator executor.
### 10.11.1 Lowering Pipeline
```mermaid
flowchart LR
subgraph PhysicalPlan
PP[Physical Plan Tree]
end
subgraph Lowering
PL[PlanLowering]
IL[IRBuilder]
OPT[IR Optimizer]
CG[Code Generator]
end
subgraph Output
BC[BytecodeModule]
end
PP --> PL
PL --> IL
IL --> OPT
OPT --> CG
CG --> BC
style PhysicalPlan fill:#e3f2fd
style Lowering fill:#fff3e0
style Output fill:#e8f5e9
```
### 10.11.2 Operator Lowering
Each physical operator has a corresponding lowering method:
```cpp
class PlanLowering {
void lower_seq_scan_(const SeqScanOp* op);
void lower_index_scan_(const IndexScanOp* op);
void lower_filter_(const PhysicalFilterOp* op);
void lower_hash_join_(const HashJoinOp* op);
void lower_sort_(const PhysicalSortOp* op);
void lower_hash_aggregate_(const HashAggregateOp* op);
void lower_project_(const ProjectOp* op);
void lower_limit_(const LimitOp* op);
};
```
### 10.11.3 Cursor Slot Management
Scan operators use cursor slots to manage iteration state:
```cpp
// Allocate slot
auto slot = allocate_cursor_slot_();
// Open cursor
emit_(Opcode::CURSOR_OPEN, slot, collection_name);
// Iteration loop
emit_label_(loop_label);
emit_(Opcode::CURSOR_NEXT, result_reg, slot);
emit_(Opcode::JUMP_NULL, result_reg, done_label);
// ... process row ...
emit_(Opcode::JUMP, loop_label);
// Close cursor
emit_label_(done_label);
emit_(Opcode::CURSOR_CLOSE, slot);
```
### 10.11.4 Join Lowering
Hash join lowering generates build and probe phases:
```cpp
void PlanLowering::lower_hash_join_(const HashJoinOp* op) {
// Lower build side
lower_(op->build_child());
// Build phase: populate hash table
emit_(Opcode::HASH_BUILD_START, hash_table_reg);
emit_label_(build_loop);
emit_(Opcode::CURSOR_NEXT, build_row, build_cursor);
emit_(Opcode::JUMP_NULL, build_row, build_done);
emit_(Opcode::HASH_BUILD_INSERT, hash_table_reg, build_row, key_reg);
emit_(Opcode::JUMP, build_loop);
emit_label_(build_done);
// Lower probe side
lower_(op->probe_child());
// Probe phase: scan and look up
emit_label_(probe_loop);
emit_(Opcode::CURSOR_NEXT, probe_row, probe_cursor);
emit_(Opcode::JUMP_NULL, probe_row, probe_done);
emit_(Opcode::HASH_PROBE, match_reg, hash_table_reg, probe_row, key_reg);
emit_(Opcode::JUMP_NULL, match_reg, probe_loop);
// Emit matched row
emit_(Opcode::ROW_COMBINE, result_reg, probe_row, match_reg);
emit_(Opcode::EMIT_ROW, result_reg);
emit_(Opcode::JUMP, probe_loop);
emit_label_(probe_done);
emit_(Opcode::HASH_TABLE_FREE, hash_table_reg);
}
```
### 10.11.5 Nested CTE Discovery
Before physical planning begins, the executor performs a recursive discovery pass over the AST to find and materialize CTEs defined in nested subqueries. A top-level query may contain derived tables or join subqueries that themselves define `WITH` clauses. If these nested CTEs are not discovered and materialized before the main plan is built, references to them will fail during execution.
The `prepare_nested_subquery_ctes_()` method walks the AST recursively:
1. For each `SelectStmt`, it inspects `from_subqueries` and `joins` for nested subqueries and descends into them.
2. When a nested statement contains a `WITH` clause, the method builds a logical plan for that statement, executes its CTEs, and registers the materialized results.
3. The CTE name scope is saved and restored around each nested discovery to prevent name collisions between sibling subqueries.
4. `SetOperationStmt` nodes are handled similarly, recursing into both `left_query` and `right_query`.
This discovery runs before optimization and physical planning for all three execution paths (Volcano, CVM, and Acero), ensuring that CTE materialization is complete regardless of which execution model is selected.
## 10.12 Summary
Cognica's physical planning system transforms logical query plans into efficient executable plans through several key mechanisms:
1. **Access Path Enumeration**: Systematically evaluates sequential scans, index scans, index-only scans, and point lookups to find the cheapest data access method.
2. **Strategy Selection**: Chooses optimal algorithms for sort (heap, quicksort, external), join (hash, merge, nested loop), and aggregate (hash, stream, sort) operations based on input characteristics and available resources.
3. **Hybrid Execution Model**: Supports iterator-based execution for simplicity, CVM bytecode for performance, and vectorized execution for columnar workloads. The executor automatically falls back from CVM to the Volcano iterator model when the plan contains table functions or UNNEST operators inside a correlated outer context.
4. **Memory-Aware Planning**: Distributes memory budgets across operators, predicts spill behavior, and configures spill options proactively.
5. **Physical Property Tracking**: Maintains ordering and cardinality information to enable optimizations like sort elimination and merge join selection.
6. **Cost-Based Decisions**: Uses a calibrated cost model considering CPU, I/O, memory, and spill costs to guide all planning decisions.
7. **Scan Projection Pushdown**: Propagates needed-column sets top-down through the physical plan tree, with special handling for LATERAL join dependencies, so that scan operators read only the columns required by downstream operators.
8. **Nested CTE Discovery**: Recursively walks the AST before planning to discover and materialize CTEs defined in nested subqueries, ensuring they are available regardless of nesting depth.
The physical planning phase bridges the gap between declarative query specification and efficient execution, making decisions that can impact query performance by orders of magnitude.
# Chapter 11: Graph Storage and Operations
Chapter 3 established the mathematical foundations for graph structures within Cognica's unified algebra, demonstrating that graph posting lists are isomorphic to document posting lists and that traversal, pattern matching, and path queries all produce sets amenable to Boolean composition. This chapter shifts from theory to implementation: how Cognica materializes the property graph model as document collections, how CRUD operations maintain graph invariants, how traversal algorithms navigate the adjacency structure, and how an in-memory adjacency cache accelerates multi-hop queries.
## 11.1 Property Graph Model in Cognica
### 11.1.1 From Algebra to Implementation
Chapter 3 defined a property graph as a tuple $G = (V, E, \rho, \lambda, \sigma)$ with vertices, edges, property assignments, labels, and edge types. The challenge of implementation is to map this abstract structure onto Cognica's concrete storage layer — document collections backed by LSM-trees — while preserving the algebraic properties that enable cross-paradigm query composition.
The central observation is that vertices and edges are both semi-structured entities carrying key-value properties, exactly like documents. A vertex with label "Person" and properties `{name: "Alice", age: 30}` differs from a document only in the additional graph metadata it carries (label, graph membership) and the structural relationships (edges) it participates in. This observation motivates a storage strategy where graph entities are stored as documents in specialized collections, leveraging all existing infrastructure for indexing, querying, and transaction management.
### 11.1.2 Property Graph Definition
Cognica implements the **labeled property graph** model, where:
- **Vertices** (nodes) carry a single label and a set of key-value properties
- **Edges** (relationships) carry a single type and a set of key-value properties
- Edges are **directed**: each edge has a source vertex and a target vertex
- Both vertices and edges have system-assigned unique identifiers
- Multiple edges of the same or different types may connect the same pair of vertices
This model is formalized as:
$$
G = (V, E, \text{id}, \text{label}, \text{type}, \text{props})
$$
where:
- $V$ is the set of vertices
- $E \subseteq V \times V \times \mathcal{T}$ is the set of typed, directed edges
- $\text{id}: V \cup E \to \text{String}$ assigns unique identifiers
- $\text{label}: V \to L$ assigns a label to each vertex
- $\text{type}: E \to \mathcal{T}$ assigns a type to each edge
- $\text{props}: V \cup E \to 2^{\mathcal{K} \times \mathcal{V}}$ assigns properties
### 11.1.3 Design Decisions: Why Property Graph
Several graph data models were evaluated during the design of Cognica's graph layer:
| Model | Strengths | Weaknesses |
|-------|-----------|------------|
| Property Graph | Rich properties, intuitive, industry standard | Multi-label support varies |
| RDF (Subject-Predicate-Object) | Standardized (SPARQL), reasoning | Verbose, no properties on edges natively |
| Hypergraph | Edges connect any number of vertices | Complexity, limited tooling |
| Bipartite Graph | Clean for specific domains | Too restrictive for general use |
The property graph model was selected for three reasons:
1. **Document-vertex isomorphism**: Properties on vertices and edges map directly to JSON document fields, enabling reuse of the document storage engine without an impedance mismatch.
2. **Industry adoption**: Neo4j, Amazon Neptune, Apache AGE, and TigerGraph all use the property graph model. Adopting it enables Apache AGE-compatible Cypher query support (described in the Cypher design document) and reduces the learning curve for users migrating from these systems.
3. **Expressive power**: The property graph model naturally represents the entities and relationships found in social networks, knowledge graphs, fraud detection, and recommendation systems — the dominant use cases for graph databases.
## 11.2 Storage Architecture
### 11.2.1 Dual-Collection Model
Each graph in Cognica is stored as a pair of document collections:
- `{graph}_nodes` — stores all vertices
- `{graph}_edges` — stores all edges
For example, a graph named `social` creates collections `social_nodes` and `social_edges`. This naming convention is enforced by constants in the implementation:
```cpp
constexpr auto kNodesSuffix = "_nodes";
constexpr auto kEdgesSuffix = "_edges";
```
(`graph_functions.cpp:34-35`)
The dual-collection model has several advantages over alternatives:
1. **Separation of concerns**: Node and edge schemas differ structurally, and separate collections allow independent indexing strategies.
2. **Query selectivity**: Queries that access only nodes (or only edges) scan a single collection, avoiding the overhead of filtering a mixed entity store.
3. **Existing infrastructure**: Each collection inherits the full capabilities of Cognica's document storage layer — primary and secondary indexes, cursor-based iteration, transactional operations — without requiring graph-specific storage code.
```mermaid
graph TB
subgraph "Graph: social"
NC[social_nodes Collection]
EC[social_edges Collection]
end
subgraph "Node Indexes"
PK1[Primary: _id]
SK1[Secondary: _label]
SK2[Secondary: _graph]
end
subgraph "Edge Indexes"
PK2[Primary: _id]
SK3[Secondary: _type]
SK4[Secondary: _graph]
SK5[Composite: _source + _type]
SK6[Composite: _target + _type]
end
NC --> PK1
NC --> SK1
NC --> SK2
EC --> PK2
EC --> SK3
EC --> SK4
EC --> SK5
EC --> SK6
```
### 11.2.2 Document-Vertex Correspondence
Chapter 3 proved that graph posting lists are isomorphic to document posting lists. The storage architecture realizes this isomorphism concretely: every vertex is a document, and every document in a `_nodes` collection is a vertex. The mapping preserves identity (the document `_id` is the vertex identifier) and properties (document fields are vertex properties).
This correspondence means that graph vertices are queryable through the standard SQL interface:
```sql
-- Query vertices directly as documents
SELECT name, age FROM social_nodes WHERE _label = 'Person' AND age > 25;
```
The dual nature — vertices as documents, documents as vertices — enables the cross-paradigm queries that Chapter 3 motivated algebraically.
### 11.2.3 Node Document Format
A node document stores graph metadata in system fields (prefixed with underscore) and user properties as top-level fields:
```json
{
"_id": "social:Person:1001",
"_graph": "social",
"_label": "Person",
"name": "Alice",
"age": 30,
"email": "alice@example.com"
}
```
The system fields are:
| Field | Type | Description |
|-------|------|-------------|
| `_id` | String | Unique vertex identifier |
| `_graph` | String | Graph namespace membership |
| `_label` | String | Vertex label (e.g., "Person", "Company") |
All other fields are user-defined properties, stored flat at the top level of the document.
### 11.2.4 Edge Document Format
An edge document follows a similar pattern, with additional fields recording the source and target vertices:
```json
{
"_id": "social:KNOWS:42",
"_graph": "social",
"_type": "KNOWS",
"_source": "social:Person:1001",
"_target": "social:Person:1002",
"since": 2020,
"strength": 0.85
}
```
The system fields are:
| Field | Type | Description |
|-------|------|-------------|
| `_id` | String | Unique edge identifier |
| `_graph` | String | Graph namespace membership |
| `_type` | String | Edge type (e.g., "KNOWS", "WORKS_AT") |
| `_source` | String | Source vertex `_id` |
| `_target` | String | Target vertex `_id` |
Edge properties (`since`, `strength` in the example) are stored flat alongside the system fields.
### 11.2.5 Identifier Scheme
Vertex and edge identifiers follow the format `{graph}:{label_or_type}:{sequence}`:
```
social:Person:1 -- first Person vertex in the social graph
social:Company:1 -- first Company vertex in the social graph
social:KNOWS:1 -- first KNOWS edge in the social graph
social:WORKS_AT:1 -- first WORKS_AT edge in the social graph
```
The identifier generation is implemented with per-key atomic sequence counters:
```cpp
auto make_node_id_(const std::string& graph, const std::string& label)
-> std::string {
auto seq = get_next_sequence_(graph + ":" + label);
return graph + ":" + label + ":" + std::to_string(seq);
}
```
(`graph_functions.cpp:163-167`)
This scheme provides several benefits:
1. **Namespacing**: The graph prefix prevents identifier collisions across graphs.
2. **Type encoding**: The label/type segment enables efficient prefix-based filtering.
3. **Monotonicity**: Sequence numbers ensure identifiers are monotonically increasing within each category, which provides good LSM-tree write locality.
4. **Human readability**: Unlike opaque integer or UUID identifiers, the structured format aids debugging and interactive querying.
### 11.2.6 Flat Property Storage
A key design decision is storing properties flat at the top level of the document rather than nested under a `properties` sub-object. Consider the two alternatives:
**Cognica (flat storage)**:
```json
{
"_id": "social:Person:1",
"_label": "Person",
"_graph": "social",
"name": "Alice",
"age": 30
}
```
**Alternative (nested storage, used by Apache AGE internally)**:
```json
{
"id": "social:Person:1",
"label": "Person",
"properties": {
"name": "Alice",
"age": 30
}
}
```
Flat storage was chosen for three reasons:
1. **Index utilization**: Cognica's secondary indexes operate on top-level document fields. Flat storage means `name` and `age` are directly indexable. Nested storage would require JSON path indexes or specialized index types.
2. **Query simplicity**: With flat storage, a Cypher expression `n.name` translates to a direct column reference `t0.name` in SQL. Nested storage would require `t0.properties->>'name'`, adding JSON extraction overhead.
3. **Document-vertex unification**: Flat storage means node documents are structurally identical to ordinary documents with a few extra system fields. This makes graph collections directly queryable with standard SQL, fulfilling the cross-paradigm promise.
The only cost is that system fields (prefixed with `_`) must be excluded when constructing a properties-only view for API compatibility. The implementation handles this with a simple prefix check:
```cpp
for (auto it = doc.MemberBegin(); it != doc.MemberEnd(); ++it) {
const auto* name = it->name.GetString();
if (name[0] != '_') {
// This is a user property, include in output
}
}
```
(`graph_functions.cpp:352-355`)
### 11.2.7 Index Strategy
The `graph_create` function creates both collections with carefully chosen indexes:
**Node collection indexes**:
| Index Name | Fields | Purpose |
|-----------|--------|---------|
| Primary | `_id` | Vertex lookup by identifier |
| `idx_label` | `_label` | Filter vertices by label |
| `idx_graph` | `_graph` | Filter vertices by graph namespace |
**Edge collection indexes**:
| Index Name | Fields | Purpose |
|-----------|--------|---------|
| Primary | `_id` | Edge lookup by identifier |
| `idx_type` | `_type` | Filter edges by type |
| `idx_graph` | `_graph` | Filter edges by graph namespace |
| `idx_src_type` | `_source, _type` | Outgoing edge traversal with type filter |
| `idx_tgt_type` | `_target, _type` | Incoming edge traversal with type filter |
The composite indexes `idx_src_type` and `idx_tgt_type` are particularly important for graph traversal performance. When expanding outgoing edges from vertex $v$ with a type filter, the query `{_source: v, _type: T}` can be resolved entirely through the `idx_src_type` composite index, avoiding a full collection scan.
## 11.3 Graph Management
### 11.3.1 Graph Creation
The `graph_create` function atomically creates both the node and edge collections with their respective schemas and indexes:
```sql
SELECT * FROM graph_create('social');
```
The implementation first checks whether a graph with the given name already exists by verifying the presence of both collections. If either already exists, the operation fails with an error rather than silently succeeding or partially creating the graph.
```cpp
auto nodes_schema
= db::document::SchemaBuilder {}
.set_collection_name(nodes_collection_name_(*graph_name))
.set_primary_key({kFieldId})
.add_secondary_key("idx_label", {kFieldLabel}, false)
.add_secondary_key("idx_graph", {kFieldGraph}, false)
.build();
```
(`graph_functions.cpp:479-485`)
The edge schema includes the composite indexes for efficient traversal:
```cpp
auto edges_schema
= db::document::SchemaBuilder {}
.set_collection_name(edges_collection_name_(*graph_name))
.set_primary_key({kFieldId})
.add_secondary_key("idx_type", {kFieldType}, false)
.add_secondary_key("idx_graph", {kFieldGraph}, false)
.add_secondary_key("idx_src_type",
{kFieldSource, kFieldType}, false)
.add_secondary_key("idx_tgt_type",
{kFieldTarget, kFieldType}, false)
.build();
```
(`graph_functions.cpp:492-502`)
If the edges collection creation fails, the already-created nodes collection is dropped to maintain atomicity.
### 11.3.2 Graph Deletion
Graph deletion removes both collections and invalidates all adjacency cache entries for the graph:
```sql
SELECT * FROM graph_drop('social');
```
The order of operations matters: the adjacency cache is invalidated first, then the edges collection is dropped, then the nodes collection. This ordering ensures that no traversal operation can read cached data pointing to dropped collections.
```cpp
GraphAdjacencyCache::instance().invalidate_graph(*graph_name);
auto edges_dropped
= doc_db->drop_collection(edges_collection_name_(*graph_name));
auto nodes_dropped
= doc_db->drop_collection(nodes_collection_name_(*graph_name));
```
(`graph_functions.cpp:538-543`)
### 11.3.3 Graph Discovery
Two functions support graph discovery:
**`graph_list()`** returns all graphs in the database by scanning collection names for the `_nodes` suffix and verifying that a corresponding `_edges` collection exists:
```sql
SELECT * FROM graph_list();
-- Returns: social, company, knowledge_base, ...
```
**`graph_exists(name)`** checks whether a specific graph exists:
```sql
SELECT * FROM graph_exists('social');
-- Returns: true or false
```
The existence check verifies both collections are present, not just one. This prevents a partially created or corrupted graph from appearing as valid.
## 11.4 Node and Edge Operations
### 11.4.1 Node Creation
Creating a node inserts a document into the `_nodes` collection with system fields and user properties:
```sql
SELECT * FROM graph_create_node('social', 'Person',
'{"name": "Alice", "age": 30}');
-- Returns: social:Person:1
```
The implementation generates a unique identifier, constructs a document with system fields, copies user properties to the top level, and inserts the document:
```cpp
auto node_id = make_node_id_(*graph_name, *label);
auto doc = db::document::Document {};
add_string_member_(doc, kFieldId, node_id);
add_string_member_(doc, kFieldGraph, *graph_name);
add_string_member_(doc, kFieldLabel, *label);
// Copy user properties to top level
for (auto it = properties->MemberBegin();
it != properties->MemberEnd(); ++it) {
// ... copy key-value pair to doc
}
```
(`graph_functions.cpp:634-650`)
The function validates that the graph exists before insertion and returns the generated node identifier on success.
### 11.4.2 Node Retrieval
A node can be retrieved by its identifier:
```sql
SELECT * FROM graph_get_node('social', 'social:Person:1');
```
The result includes three columns compatible with the Apache AGE format:
| Column | Type | Description |
|--------|------|-------------|
| `id` | String | Vertex identifier |
| `label` | String | Vertex label |
| `properties` | JSONB | User properties (system fields excluded) |
The properties column is constructed by iterating over the document and filtering out fields whose names begin with `_`.
### 11.4.3 Node Querying
The `graph_nodes` function queries vertices by label and optional property filters:
```sql
-- All Person nodes
SELECT * FROM graph_nodes('social', 'Person');
-- Person nodes with specific properties
SELECT * FROM graph_nodes('social', 'Person', '{"age": 30}');
-- All nodes regardless of label
SELECT * FROM graph_nodes('social', NULL);
```
The label and property filters are composed into a single query document and executed against the nodes collection, leveraging the `idx_label` secondary index for label filtering.
### 11.4.4 Node Update
Node properties can be updated in place:
```sql
SELECT * FROM graph_update_node('social', 'social:Person:1',
'{"age": 31, "title": "Engineer"}');
```
The update operation validates that no system fields (those prefixed with `_`) are being modified, with one exception: node updates allow changing the `_label` field to support vertex relabeling:
```sql
-- Relabel a Person node to Employee
SELECT * FROM graph_update_node('social', 'social:Person:1',
'{"_label": "Employee"}');
```
The `$delete` operator supports both single-property and multi-property deletion:
```sql
-- Delete a single property
SELECT * FROM graph_update_node('social', 'social:Person:1',
'{"$delete": "title"}');
-- Delete multiple properties at once
SELECT * FROM graph_update_node('social', 'social:Person:1',
'{"$delete": ["title", "email"]}');
```
The `$delete` operator also validates that reserved system fields cannot be removed. The validation logic checks both direct property keys and `$delete` targets against the reserved `_` prefix:
```cpp
if (key == "$delete") {
// Accepts a single string or an array of strings
// Each path is validated against reserved field names
}
if (key[0] == '_') {
if (allow_label_update && key == kFieldLabel) {
continue; // Node updates permit _label changes
}
return std::unexpected(/* system field error */);
}
```
(`graph_functions.cpp:296-335`)
After a successful update, the adjacency cache entry for the node is invalidated to ensure traversal operations see the current state.
### 11.4.5 Node Deletion
Deleting a node checks for connected edges before proceeding:
```sql
-- Fails if the node has any connected edges
SELECT * FROM graph_delete_node('social', 'social:Person:1');
-- Detach delete: removes the node and all connected edges
SELECT * FROM graph_delete_node('social', 'social:Person:1', true);
```
The detach delete operation collects all edges where the node appears as either source or target, removes them all, then removes the node. This is implemented as a two-phase process:
1. **Edge collection**: Query `{_source: node_id}` and `{_target: node_id}` to find all connected edge identifiers using a deduplication set.
2. **Edge removal**: Delete each connected edge by its identifier.
3. **Node removal**: Delete the node document.
```cpp
auto collect_connected_edge_ids = [&](auto* edges_view)
-> std::vector {
auto source_query = make_query_doc_(kFieldSource, *node_id);
auto target_query = make_query_doc_(kFieldTarget, *node_id);
auto edge_ids = std::vector {};
auto seen = absl::flat_hash_set {};
// ... collect from both source and target queries
};
```
(`graph_functions.cpp:937-963`)
After deletion, the adjacency cache is invalidated for the removed node.
### 11.4.6 Edge Creation
Creating an edge requires specifying the graph, edge type, source vertex, target vertex, and properties:
```sql
SELECT * FROM graph_create_edge('social', 'KNOWS',
'social:Person:1', 'social:Person:2',
'{"since": 2020}');
-- Returns: social:KNOWS:1
```
The implementation performs referential integrity checks — both source and target nodes must exist before the edge is created. This prevents dangling edges that reference non-existent vertices:
```cpp
if (auto err = verify_node_exists(*source_id, source_query);
err.has_value()) {
return std::unexpected("graph_create_edge: source node '"
+ *source_id + "' does not exist");
}
```
(`graph_functions.cpp:1106-1108`)
After edge insertion, the adjacency cache is invalidated for both the source and target nodes to ensure cache consistency.
### 11.4.7 Edge Retrieval and Querying
Individual edges are retrieved by identifier:
```sql
SELECT * FROM graph_get_edge('social', 'social:KNOWS:1');
```
The result follows the Apache AGE-compatible format:
| Column | Type | Description |
|--------|------|-------------|
| `id` | String | Edge identifier |
| `start_id` | String | Source vertex identifier |
| `end_id` | String | Target vertex identifier |
| `label` | String | Edge type |
| `properties` | JSONB | User properties |
Edges connected to a specific node can be queried with direction control:
```sql
-- Outgoing edges from Alice
SELECT * FROM graph_edges('social', 'social:Person:1', 'KNOWS', 'outgoing');
-- Incoming edges to Alice
SELECT * FROM graph_edges('social', 'social:Person:1', 'KNOWS', 'incoming');
-- All edges (both directions)
SELECT * FROM graph_edges('social', 'social:Person:1', 'KNOWS', 'both');
```
The direction parameter controls which index is queried:
| Direction | Index Used | Query Field |
|-----------|-----------|-------------|
| `outgoing` | `idx_src_type` | `_source` |
| `incoming` | `idx_tgt_type` | `_target` |
| `both` | Both indexes | `_source` and `_target` |
### 11.4.8 Edge Update and Deletion
Edge updates follow a similar pattern to node updates but with stricter validation. Unlike node updates, edge updates do not permit changes to any system field (including `_type`). The properties object must also be non-empty:
```sql
SELECT * FROM graph_update_edge('social', 'social:KNOWS:1',
'{"since": 2025}');
```
Before applying the update, the implementation loads the edge metadata to determine the source and target vertex identifiers. If the specified edge does not exist, the operation fails with an explicit error rather than silently succeeding:
```cpp
auto load_edge_metadata = [&](auto* edges_view) {
auto cursor = edges_view->find(query);
if (!cursor || !cursor->is_valid()) {
return;
}
found_edge = true;
// Extract source_id and target_id for cache invalidation
};
```
(`graph_functions.cpp:1385-1398`)
Edge updates also support the `$delete` operator for removing individual properties, with the same reserved-field protection as node updates. The `$delete` operator accepts either a single field name or an array of field names.
Edge deletion loads the edge metadata first (to determine source and target for cache invalidation), then removes the edge document:
```sql
SELECT * FROM graph_delete_edge('social', 'social:KNOWS:1');
```
Both operations invalidate cache entries for the source and target nodes of the affected edge.
### 11.4.9 Transaction-Aware Operations
All graph CRUD functions participate in the database transaction context when one is available. Each table function receives a `TableFunctionContext` that may carry an active transaction and workspace identifier. When a transaction context is present, all collection lookups and mutations are routed through the transaction rather than directly through the document database:
```cpp
auto find_collection_(const TableFunctionContext& ctx,
const std::string& collection_name)
-> std::shared_ptr {
auto* doc_db = resolve_document_db_(ctx);
if (doc_db == nullptr) {
return nullptr;
}
if (has_transaction_context_(ctx)) {
return ctx.transaction->find_collection(
ctx.workspace_id, collection_name);
}
return doc_db->find_collection(collection_name);
}
```
(`graph_functions.cpp:195-208`)
This design enables several capabilities:
1. **Atomic multi-step graph mutations**: A Cypher `CREATE` statement that creates multiple nodes and edges can execute all mutations within a single transaction. If any step fails, the entire operation rolls back.
2. **Read-your-own-writes**: Within a LATERAL join chain, a graph mutation in one step is visible to subsequent steps in the same query. For example, creating a node and then creating an edge to that node within the same query works correctly because both operations share the same transaction context.
3. **Isolation from concurrent queries**: Graph mutations in progress are not visible to other sessions until the transaction commits, preventing partial graph states from being observed.
The transaction routing is transparent to the function implementations. Each CRUD function uses `resolve_document_db_()` to obtain the database handle and `begin_collection_transaction_()` to obtain a collection-scoped transaction when available. When no transaction context is present (standalone queries outside an explicit transaction), the functions fall back to direct collection access, preserving backward compatibility.
### 11.4.10 Scalar JSON Helpers for Path Materialization
In addition to the table functions described above, four scalar functions provide JSON-formatted access to graph elements. These functions are used internally by the Cypher query compiler to materialize path expressions, where nodes and edges must be returned as structured JSON values rather than tabular rows:
| Function | Arguments | Returns | Description |
|----------|-----------|---------|-------------|
| `graph_get_node_json` | graph TEXT, id TEXT | JSONB | Single node as JSON object |
| `graph_get_nodes_json` | graph TEXT, ids TEXT[] | JSONB | Array of nodes as JSON array |
| `graph_get_edge_json` | graph TEXT, id TEXT | JSONB | Single edge as JSON object |
| `graph_get_edges_json` | graph TEXT, ids TEXT[] | JSONB | Array of edges as JSON array |
The JSON representation follows a structured format distinct from the flat document storage:
**Node JSON format**:
```json
{
"id": "social:Person:1",
"label": "Person",
"properties": {"name": "Alice", "age": 30}
}
```
**Edge JSON format**:
```json
{
"id": "social:KNOWS:1",
"start_id": "social:Person:1",
"end_id": "social:Person:2",
"label": "KNOWS",
"properties": {"since": 2020}
}
```
Unlike the flat storage format used in document collections, these JSON representations nest user properties under a `properties` key. This matches the Apache AGE convention for Cypher path output and enables clean separation of system metadata from user data in query results.
The scalar functions are registered with `kVolatile` volatility because they read from mutable graph collections. They are also transaction-aware, using the same `SessionContext` plumbing as the table functions to ensure read-your-own-writes consistency within multi-step Cypher queries.
## 11.5 Traversal Algorithms
Graph traversal is the distinguishing capability that separates graph databases from document databases. Cognica implements several traversal algorithms as SQL table functions, each optimized for specific access patterns.
### 11.5.1 Neighbor Discovery: graph_neighbors
The `graph_neighbors` function performs breadth-first neighbor discovery from a starting vertex:
```sql
SELECT * FROM graph_neighbors('social', 'social:Person:1',
'KNOWS', 'outgoing', 2);
```
**Parameters**:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `graph` | TEXT | required | Graph name |
| `start_node` | TEXT | required | Starting vertex ID |
| `edge_type` | TEXT | NULL | Edge type filter (NULL = all types) |
| `direction` | TEXT | 'outgoing' | 'outgoing', 'incoming', or 'both' |
| `max_depth` | INT | 1 | Maximum traversal depth |
**Algorithm**: The implementation uses BFS with a visited set to avoid cycles and a frontier deque to track vertices at each depth level:
```mermaid
graph LR
A[Alice] -->|depth 0| F1[Frontier]
F1 -->|expand| B[Bob]
F1 -->|expand| C[Charlie]
B -->|depth 1| F2[Frontier]
C -->|depth 1| F2
F2 -->|expand| D[David]
F2 -->|expand| E[Eve]
```
The algorithm operates as follows:
1. Initialize the visited set with the start node and push it onto the frontier with depth 0.
2. Pop the front of the frontier. If depth equals `max_depth`, skip expansion.
3. For each adjacent vertex not yet visited, add it to the visited set and the result set, then push it onto the back of the frontier with incremented depth.
4. Repeat until the frontier is empty.
```cpp
auto visited = absl::flat_hash_set {};
visited.insert(*start_node);
auto frontier = std::deque<
std::tuple>> {};
frontier.emplace_back(*start_node, 0,
std::vector {*start_node});
while (!frontier.empty()) {
auto [current_node, depth, path] = std::move(frontier.front());
frontier.pop_front();
if (depth >= max_depth) {
continue;
}
// ... expand neighbors
}
```
(`graph_functions.cpp:1593-1608`)
**Result columns**: Each discovered neighbor is returned with:
| Column | Type | Description |
|--------|------|-------------|
| `id` | String | Neighbor vertex ID |
| `label` | String | Neighbor vertex label |
| `properties` | JSONB | Neighbor properties |
| `depth` | INT | Distance from start node |
| `path` | ARRAY | Vertex IDs along the traversal path |
**Complexity**: For a graph with maximum degree $d$ and traversal depth $k$:
- Time: $O(d^k)$ — exponential in depth, as each level expands up to $d$ neighbors
- Space: $O(d^k)$ — the frontier and visited set grow proportionally
### 11.5.2 Configurable Traversal: graph_traverse
The `graph_traverse` function extends neighbor discovery with support for both BFS and DFS strategies, edge type filtering via arrays, and configurable depth limits:
```sql
-- BFS traversal following only KNOWS edges
SELECT * FROM graph_traverse('social', 'social:Person:1',
ARRAY['KNOWS'], 'outgoing', 3, 'bfs');
-- DFS traversal following KNOWS and FOLLOWS edges
SELECT * FROM graph_traverse('social', 'social:Person:1',
ARRAY['KNOWS', 'FOLLOWS'], 'outgoing', 5, 'dfs');
```
**Parameters**:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `graph` | TEXT | required | Graph name |
| `start_node` | TEXT | required | Starting vertex ID |
| `edge_types` | TEXT[] | NULL | Array of edge types (NULL = all) |
| `direction` | TEXT | 'outgoing' | 'outgoing', 'incoming', or 'both' |
| `max_depth` | INT | 10 | Maximum traversal depth |
| `strategy` | TEXT | 'bfs' | 'bfs' or 'dfs' |
**BFS vs DFS**: The implementation uses a single `std::deque` for both strategies. The only difference is where elements are removed:
```cpp
if (strategy == "bfs") {
state = std::move(frontier.front());
frontier.pop_front();
} else {
state = std::move(frontier.back());
frontier.pop_back();
}
```
(`graph_functions.cpp:1765-1771`)
BFS pops from the front (FIFO), guaranteeing that vertices are discovered in order of increasing depth. DFS pops from the back (LIFO), exploring each branch to its maximum depth before backtracking.
**Edge type filtering**: When the edge type array is non-empty, a hash set is constructed for $O(1)$ type membership checks during traversal:
```cpp
auto type_filter = absl::flat_hash_set {};
for (const auto& t : edge_types) {
type_filter.insert(t);
}
```
(`graph_functions.cpp:1738-1741`)
During expansion, each edge's type is checked against this filter set. Edges with types not in the set are skipped.
**Complexity**: Same as neighbor discovery — $O(d^k)$ time and space — but the effective degree $d$ may be reduced by edge type filtering.
### 11.5.3 Shortest Path: graph_shortest_path
The `graph_shortest_path` function finds the shortest path between two vertices using **bidirectional BFS**:
```sql
SELECT * FROM graph_shortest_path('path_graph',
'path_graph:Node:1', 'path_graph:Node:5',
NULL, 'outgoing', 10);
```
**Algorithm**: Bidirectional BFS simultaneously expands from both the start and end vertices, meeting in the middle. This reduces the search space from $O(d^k)$ to $O(2 \cdot d^{k/2})$ for a graph with degree $d$ and shortest path length $k$.
```mermaid
graph LR
subgraph "Forward Search"
S[Start] --> F1[Layer 1]
F1 --> F2[Layer 2]
end
subgraph "Backward Search"
E[End] --> B1[Layer 1]
B1 --> B2[Layer 2]
end
F2 -.->|Meeting Point| B2
```
The algorithm maintains two separate search frontiers and visited maps:
```cpp
auto forward = absl::flat_hash_map {};
auto backward = absl::flat_hash_map {};
forward[*start_node] = PathNode {*start_node, "", "", 0};
backward[*end_node] = PathNode {*end_node, "", "", 0};
auto forward_frontier = std::vector {*start_node};
auto backward_frontier = std::vector {*end_node};
```
(`graph_functions.cpp:1930-1937`)
Each `PathNode` stores the node ID, its parent in the search tree, the edge used to reach it, and its depth. This information enables path reconstruction once the two searches meet.
**Alternating expansion**: The main loop alternates between expanding the forward and backward frontiers. After each expansion, it checks whether any newly discovered vertex exists in the opposite frontier:
```cpp
for (int64_t depth = 0; depth < max_depth / 2 + 1; ++depth) {
// Expand forward frontier
expand_frontier(forward_frontier, forward,
kFieldSource, kFieldTarget);
// Check for meeting point
for (const auto& node : forward_frontier) {
if (backward.contains(node)) {
return reconstruct_path(node);
}
}
// Expand backward frontier
expand_frontier(backward_frontier, backward,
kFieldTarget, kFieldSource);
// Check for meeting point
for (const auto& node : backward_frontier) {
if (forward.contains(node)) {
return reconstruct_path(node);
}
}
}
```
(`graph_functions.cpp:2027-2061`)
**Path reconstruction**: When the two frontiers meet at vertex $m$, the path is reconstructed by tracing parents backward from $m$ to the start (forward path) and from $m$ to the end (backward path):
1. Trace the forward path: $m \to \text{parent}(m) \to \ldots \to s$ (start)
2. Reverse it to get: $s \to \ldots \to m$
3. Trace the backward path: $m \to \text{parent}(m) \to \ldots \to t$ (end)
4. Concatenate: $s \to \ldots \to m \to \ldots \to t$
**Trivial case**: When start equals end, the function immediately returns a path of length 0 containing just the single vertex.
**Result columns**:
| Column | Type | Description |
|--------|------|-------------|
| `nodes` | ARRAY | Ordered vertex IDs along the path |
| `edges` | ARRAY | Ordered edge IDs along the path |
| `length` | INT | Number of edges in the path |
| `total_weight` | FLOAT | Sum of edge weights (defaults to hop count) |
**Complexity**: For a graph with average degree $d$ and shortest path length $k$:
- Time: $O(d^{k/2})$ — the square root of unidirectional BFS
- Space: $O(d^{k/2})$ — for the two visited maps
The bidirectional approach provides substantial speedup for long paths in dense graphs. For a graph with $d = 100$ and $k = 6$:
- Unidirectional BFS: $O(100^6) = O(10^{12})$ vertex expansions
- Bidirectional BFS: $O(2 \times 100^3) = O(2 \times 10^6)$ vertex expansions
This represents a millionfold reduction.
### 11.5.4 All Paths: graph_all_paths
The `graph_all_paths` function enumerates all simple paths between two vertices within depth constraints:
```sql
SELECT * FROM graph_all_paths('path_graph',
'path_graph:Node:1', 'path_graph:Node:5',
NULL, 1, 5, 100);
```
**Parameters**:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `graph` | TEXT | required | Graph name |
| `start` | TEXT | required | Start vertex ID |
| `end` | TEXT | required | End vertex ID |
| `edge_types` | TEXT[] | NULL | Edge type filter |
| `min_depth` | INT | 1 | Minimum path length |
| `max_depth` | INT | 5 | Maximum path length |
| `limit` | INT | 100 | Maximum number of paths to return |
**Algorithm**: DFS with per-path cycle detection. Unlike the traversal functions which maintain a global visited set, `graph_all_paths` uses a per-path visited set to allow the same vertex to appear in different paths:
```cpp
struct DFSState {
std::string node;
std::vector path_nodes;
std::vector path_edges;
absl::flat_hash_set visited;
};
```
(`graph_functions.cpp:2114-2119`)
Each stack entry carries its own visited set. When a neighbor has already been visited on the current path, it is skipped — except when the neighbor is the target vertex, which is allowed to terminate the path:
```cpp
if (state.visited.contains(neighbor) && neighbor != *end_node) {
continue;
}
```
(`graph_functions.cpp:2167-2168`)
This cycle detection ensures that only simple paths (no repeated vertices except the terminal) are enumerated.
**Complexity**: In the worst case (complete graph), the number of simple paths between two vertices can be exponential: $O(n!)$ for $n$ vertices. The `limit` parameter bounds the output, but the search itself may explore exponentially many partial paths. The `max_depth` parameter provides a tighter bound by limiting path length.
### 11.5.5 Reachability: graph_reachable
The `graph_reachable` function checks whether a path exists between two vertices, optimized for early termination:
```sql
SELECT * FROM graph_reachable('social',
'social:Person:1', 'social:Person:5');
-- Returns: true or false
```
The implementation delegates to `graph_shortest_path` and checks whether any path was found:
```cpp
auto path_result = fn_graph_shortest_path(ctx, args);
if (!path_result) {
return std::unexpected(path_result.error());
}
auto& paths = std::get(*path_result);
auto result = JsonValueSeries {};
result.push_back(make_bool_value(!paths.empty()));
```
(`graph_functions.cpp:2199-2208`)
This leverages the bidirectional BFS for efficiency while providing a simpler Boolean interface.
**Complexity**: Same as `graph_shortest_path` — $O(d^{k/2})$ where $k$ is the shortest path length. The bidirectional BFS terminates as soon as any path is found, providing implicit early termination.
### 11.5.6 Complexity Summary
| Function | Algorithm | Time | Space |
|----------|-----------|------|-------|
| `graph_neighbors` | BFS | $O(d^k)$ | $O(d^k)$ |
| `graph_traverse` | BFS/DFS | $O(d^k)$ | $O(d^k)$ |
| `graph_shortest_path` | Bidirectional BFS | $O(d^{k/2})$ | $O(d^{k/2})$ |
| `graph_all_paths` | DFS + cycle detection | $O(n! / (n-k)!)$ | $O(k \cdot n)$ |
| `graph_reachable` | Bidirectional BFS | $O(d^{k/2})$ | $O(d^{k/2})$ |
Where $d$ = maximum degree, $k$ = traversal depth or path length, $n$ = total vertices.
## 11.6 Adjacency Cache
Multi-hop graph traversals require repeated edge lookups for each vertex in the frontier. Without caching, a 3-hop traversal expanding 100 neighbors at each level generates $100 + 100^2 + 100^3 = 1{,}010{,}100$ index lookups against the edge collection. The adjacency cache dramatically reduces this cost by storing edge adjacency lists in memory.
### 11.6.1 Cache Architecture
The `GraphAdjacencyCache` is a process-wide singleton that stores both outgoing and incoming edge lists for cached nodes:
```cpp
class GraphAdjacencyCache final {
// ...
private:
struct CacheEntry {
std::vector outgoing;
std::vector incoming;
uint64_t access_time;
};
std::unordered_map cache_;
mutable std::mutex mutex_;
size_t max_entries_ = 1000000; // 1M nodes
};
```
(`graph_functions.hpp:277-289`)
Each `EdgeInfo` stores the minimal information needed for traversal:
```cpp
struct EdgeInfo {
std::string edge_id;
std::string neighbor_id;
std::string edge_type;
};
```
(`graph_functions.hpp:216-220`)
The cache key is a composite string `"graph:node_id"`, ensuring that identically-named nodes in different graphs are cached independently.
### 11.6.2 LRU Eviction
When the cache reaches its capacity (default 1M entries), the eviction policy removes the least recently used entry. Each access updates an `access_time` counter:
```cpp
void GraphAdjacencyCache::evict_if_needed_() {
if (cache_.size() < max_entries_) {
return;
}
auto oldest_it = cache_.begin();
auto oldest_time = oldest_it->second.access_time;
for (auto it = cache_.begin(); it != cache_.end(); ++it) {
if (it->second.access_time < oldest_time) {
oldest_time = it->second.access_time;
oldest_it = it;
}
}
cache_.erase(oldest_it);
}
```
(`graph_functions.cpp:2530-2548`)
The eviction scan is $O(n)$ in the cache size. For production workloads that frequently trigger eviction, this could be improved with a doubly-linked list maintaining LRU order. However, for the current implementation, evictions are rare relative to cache hits because the 1M entry capacity accommodates most practical graph sizes.
### 11.6.3 Cache Warming
The `graph_warm_cache` function pre-populates the cache by iterating over all nodes in a graph and loading their adjacency lists:
```sql
SELECT * FROM graph_warm_cache('social', 100000);
-- Returns: cached = 100000, time_ms = 1500
```
The implementation scans the nodes collection, and for each node, queries its outgoing and incoming edges:
```cpp
for (const auto& node_id : node_ids) {
auto outgoing = std::vector {};
auto incoming = std::vector {};
auto out_query = make_query_doc_(kFieldSource, node_id);
auto out_cursor = edges_coll->find(out_query);
for (; out_cursor->is_valid(); out_cursor->next()) {
// ... collect EdgeInfo
}
auto in_query = make_query_doc_(kFieldTarget, node_id);
auto in_cursor = edges_coll->find(in_query);
for (; in_cursor->is_valid(); in_cursor->next()) {
// ... collect EdgeInfo
}
cache_node(graph, node_id, outgoing, incoming);
}
```
(`graph_functions.cpp:2468-2503`)
Cache warming is intended for use during low-traffic periods, such as application startup or after a graph bulk load. The `max_nodes` parameter limits the number of nodes cached to control memory usage and warming time.
### 11.6.4 Cache Invalidation
Cache consistency is maintained through eager invalidation on write operations. Every mutation that affects a node's adjacency triggers invalidation:
| Operation | Invalidation Scope |
|-----------|--------------------|
| `graph_create_edge` | Source node, Target node |
| `graph_update_edge` | Source node, Target node |
| `graph_delete_edge` | Source node, Target node |
| `graph_update_node` | The updated node |
| `graph_delete_node` | The deleted node |
| `graph_drop` | All nodes in the graph |
The invalidation is per-node, not per-edge. When an edge is created between vertices $A$ and $B$, both $A$'s and $B$'s cache entries are removed entirely (not surgically updated). This conservative approach avoids subtle consistency bugs at the cost of occasional additional cache misses:
```cpp
auto& cache = GraphAdjacencyCache::instance();
cache.invalidate_node(*graph_name, *source_id);
cache.invalidate_node(*graph_name, *target_id);
```
(`graph_functions.cpp:1152-1154`)
For `graph_drop`, all entries with the graph's prefix are removed:
```cpp
void GraphAdjacencyCache::invalidate_graph(const std::string& graph) {
auto lock = std::lock_guard {mutex_};
auto prefix = graph + ":";
for (auto it = cache_.begin(); it != cache_.end();) {
if (it->first.compare(0, prefix.size(), prefix) == 0) {
it = cache_.erase(it);
} else {
++it;
}
}
}
```
(`graph_functions.cpp:2513-2523`)
### 11.6.5 Thread Safety
All cache operations are protected by a mutex:
```cpp
auto GraphAdjacencyCache::get_outgoing(
const std::string& graph, const std::string& node_id)
-> std::optional> {
auto lock = std::lock_guard {mutex_};
auto key = make_key_(graph, node_id);
auto it = cache_.find(key);
if (it == cache_.end()) {
++miss_count_;
return std::nullopt;
}
++hit_count_;
it->second.access_time = ++access_counter_;
return it->second.outgoing;
}
```
(`graph_functions.cpp:2397-2410`)
The single-mutex design is simple and correct. For workloads with very high traversal concurrency, a sharded lock design (partitioning the cache by hash of the key) could reduce contention. However, cache lookups are fast relative to the I/O they avoid, so the mutex overhead is typically negligible.
### 11.6.6 Cache Statistics and Monitoring
The `graph_cache_stats` function exposes cache performance metrics:
```sql
SELECT * FROM graph_cache_stats();
```
| Column | Type | Description |
|--------|------|-------------|
| `entries` | INT | Current number of cached nodes |
| `hits` | INT | Total cache hit count |
| `misses` | INT | Total cache miss count |
| `hit_rate` | FLOAT | Hit rate as a decimal (0.0 to 1.0) |
These metrics enable operators to assess whether the cache is sized appropriately and whether cache warming is effective for their workload.
### 11.6.7 Performance Impact
The adjacency cache provides the greatest benefit for multi-hop traversals. Consider a 3-hop BFS traversal on a graph where each node has 50 outgoing edges:
**Without cache**: Each hop requires an index lookup against the edges collection.
- Total lookups: $50 + 50^2 + 50^3 = 127{,}550$
- Each lookup involves LSM-tree index traversal: multiple disk reads or memory-mapped page accesses
**With warm cache**: Each hop requires a hash table lookup.
- Total lookups: $50 + 50^2 + 50^3 = 127{,}550$ (same count)
- Each lookup: single hash table probe, $O(1)$ expected time, no disk I/O
The speedup is proportional to the ratio of index lookup time to hash table lookup time, which can be 100x or more depending on whether the index data is in the OS page cache.
## 11.7 Aggregation Operations
### 11.7.1 Degree Counting: graph_degree
The `graph_degree` function counts the number of edges connected to a vertex, optionally filtered by edge type and direction:
```sql
-- Count all edges (both directions)
SELECT * FROM graph_degree('social', 'social:Person:1', NULL, 'both');
-- Count only outgoing KNOWS edges
SELECT * FROM graph_degree('social', 'social:Person:1', 'KNOWS', 'outgoing');
```
The implementation iterates over matching edges and counts them:
```cpp
auto count_edges = [&](const char* field) {
auto query = db::document::Document {};
add_string_member_(query, field, *node_id);
if (edge_type) {
add_string_member_(query, kFieldType, *edge_type);
}
auto cursor = edges_coll->find(query);
for (; cursor->is_valid(); cursor->next()) {
++degree;
}
};
```
(`graph_functions.cpp:2248-2260`)
The degree computation leverages the composite indexes (`idx_src_type`, `idx_tgt_type`) when an edge type filter is provided, making the operation efficient even for high-degree nodes.
**Complexity**: $O(\text{deg}(v))$ where $\text{deg}(v)$ is the degree of vertex $v$ in the specified direction. The index limits the scan to edges matching the filter.
### 11.7.2 Common Neighbor Computation: graph_common_neighbors
The `graph_common_neighbors` function finds vertices that are outgoing neighbors of both input vertices:
```sql
SELECT * FROM graph_common_neighbors('social',
'social:Person:1', 'social:Person:2', 'KNOWS');
```
The algorithm is a straightforward set intersection:
1. Collect outgoing neighbors of vertex $A$ into set $N_A$.
2. Collect outgoing neighbors of vertex $B$ into set $N_B$.
3. Return $N_A \cap N_B$ — vertices present in both sets.
```cpp
auto neighbors_a = get_neighbors(*node_a);
auto neighbors_b = get_neighbors(*node_b);
for (const auto& neighbor : neighbors_a) {
if (neighbors_b.contains(neighbor)) {
// Fetch neighbor node and add to result
}
}
```
(`graph_functions.cpp:2340-2345`)
The implementation uses `absl::flat_hash_set` for $O(1)$ expected-time membership testing, making the intersection operation efficient.
**Complexity**: $O(\text{deg}(A) + \text{deg}(B) + |N_A \cap N_B| \cdot C_{\text{lookup}})$ where $C_{\text{lookup}}$ is the cost of fetching a node document.
**Applications**: Common neighbor counting is a fundamental building block for link prediction algorithms. The Jaccard coefficient between two vertices is computed as:
$$
J(A, B) = \frac{|N_A \cap N_B|}{|N_A \cup N_B|}
$$
The Adamic-Adar index weights each common neighbor by the inverse log of its degree:
$$
AA(A, B) = \sum_{u \in N_A \cap N_B} \frac{1}{\log(\text{deg}(u))}
$$
Both metrics can be computed efficiently using `graph_common_neighbors` combined with `graph_degree`.
## 11.8 Design Trade-offs
### 11.8.1 Flat vs Nested Property Storage
The choice to store properties flat alongside system fields (rather than nested under a `properties` key) is Cognica's most distinctive deviation from other graph databases.
**Advantages of flat storage**:
- Direct indexability of property fields
- Simpler SQL translation (`n.name` becomes `t.name`, not `t.properties->>'name'`)
- Unified document-graph model (graph vertices are ordinary documents)
- No JSON extraction overhead for property access
**Advantages of nested storage** (used by Apache AGE internally):
- Clean separation between system and user fields
- No naming conflicts between user properties and system fields
- Simpler introspection (all user data is under one key)
Cognica mitigates the disadvantage of flat storage by reserving the `_` prefix for system fields and validating that user properties cannot overwrite system fields during updates:
```cpp
if (key[0] == '_') {
return std::unexpected(std::string {fn_name}
+ ": cannot update reserved graph system fields");
}
```
(`graph_functions.cpp:328-330`)
### 11.8.2 Collection-Based vs Native Graph Storage
Cognica stores graphs as document collections rather than implementing a purpose-built graph storage engine. This is a fundamentally different approach from systems like Neo4j, which uses a native graph storage format with fixed-size records and pointer-based adjacency traversal.
**Collection-based storage** (Cognica):
| Aspect | Characteristic |
|--------|----------------|
| Adjacency traversal | Index lookup per hop |
| Vertex access | Index lookup by `_id` |
| Storage overhead | Standard document overhead |
| Implementation cost | Minimal (reuses existing engine) |
| Transaction support | Full (inherits from document layer) |
| Cross-paradigm queries | Native (vertices are documents) |
**Native graph storage** (Neo4j-style):
| Aspect | Characteristic |
|--------|----------------|
| Adjacency traversal | Pointer chase, O(1) per hop |
| Vertex access | Direct by internal ID |
| Storage overhead | Fixed-size records, compact |
| Implementation cost | Substantial (custom engine) |
| Transaction support | Custom implementation required |
| Cross-paradigm queries | Requires data duplication |
Cognica's approach trades per-hop traversal speed for implementation simplicity, full transaction support, and seamless cross-paradigm query composition. The adjacency cache closes much of the traversal performance gap by providing in-memory adjacency lookups that approach the speed of native pointer-based traversal.
### 11.8.3 Cache vs Disk Trade-offs
The adjacency cache introduces a classic space-time trade-off:
| Configuration | Memory | Traversal Speed | Write Overhead |
|---------------|--------|-----------------|----------------|
| No cache | Minimal | Slow (disk/index per hop) | None |
| Selective cache | Moderate | Fast for cached nodes | Per-write invalidation |
| Full cache | High (proportional to edge count) | Fastest | Per-write invalidation |
The 1M entry default limit assumes approximately 100 bytes per entry (edge lists for both directions), yielding roughly 100 MB of cache memory. This accommodates graphs with up to 1 million active vertices. For larger graphs, the LRU eviction policy ensures that frequently accessed vertices remain cached while infrequently accessed vertices are evicted.
### 11.8.4 Comparison with Existing Systems
| Feature | Cognica | Apache AGE | Neo4j |
|---------|---------|-----------|-------|
| Storage backend | Document collections (LSM-tree) | PostgreSQL heap tables | Native fixed-size records |
| Property storage | Flat (top-level fields) | Nested (agtype) | Native property store |
| Index type | B-tree secondary indexes | PostgreSQL indexes | Native index-free adjacency |
| Query language | SQL + table functions + Cypher | SQL + Cypher | Cypher |
| Transaction model | Document-level MVCC | PostgreSQL MVCC | Custom MVCC |
| Cross-paradigm | Native (same storage layer) | Via PostgreSQL tables | Requires connector |
| Adjacency cache | LRU in-memory cache | None (relies on PostgreSQL buffer) | Index-free adjacency |
| Traversal method | BFS/DFS via collection queries | Custom scan operators | Native traversal engine |
Apache AGE stores graph data in PostgreSQL heap tables with a custom `agtype` data type that wraps JSON-like values. Each vertex and edge is a row with an `agtype` properties column. Cognica's flat property storage avoids the `agtype` extraction overhead but requires system field naming discipline.
Neo4j uses a native storage format where each vertex record contains a pointer to its first relationship, and each relationship record contains pointers to the next relationships for both the source and target vertices. This forms a doubly-linked list that enables $O(1)$ adjacency traversal without index lookups. Cognica's adjacency cache provides similar $O(1)$ in-memory traversal when warmed, at the cost of cache memory.
## 11.9 Summary
This chapter presented Cognica's graph storage and operations layer, tracing the path from the mathematical property graph model defined in Chapter 3 to a concrete implementation built on the document storage engine described in Chapter 6.
The key architectural decisions are:
1. **Dual-collection model**: Each graph is stored as a pair of document collections (`_nodes` and `_edges`), inheriting all document storage capabilities including indexing, transactions, and SQL queryability.
2. **Flat property storage**: User properties are stored as top-level document fields rather than nested under a properties key. This enables direct indexing, simple SQL translation, and a unified document-graph data model.
3. **Structured identifiers**: The `{graph}:{label}:{sequence}` naming scheme provides namespace isolation, type encoding, and human readability while maintaining monotonic ordering for LSM-tree locality.
4. **Traversal algorithms**: BFS neighbor discovery, configurable BFS/DFS traversal, bidirectional BFS shortest path, DFS all-paths enumeration with per-path cycle detection, and BFS reachability checking cover the fundamental graph query patterns.
5. **Adjacency cache**: An LRU cache storing outgoing and incoming edge lists for up to 1 million nodes dramatically accelerates multi-hop traversals by replacing index lookups with hash table probes.
6. **Eager invalidation**: Write operations eagerly invalidate affected cache entries, maintaining consistency between the cache and the persistent storage layer.
7. **Transaction-aware CRUD**: All graph mutation functions participate in the database transaction context, enabling atomic multi-step operations and read-your-own-writes consistency within complex query pipelines.
8. **Scalar JSON helpers**: Dedicated scalar functions materialize graph elements as structured JSON values for Cypher path expressions, bridging the gap between the flat document storage model and the nested representation expected by graph query languages.
The graph layer is accessible through SQL table functions, enabling composition with standard SQL operations:
```sql
-- Cross-paradigm query: full-text search + graph traversal
SELECT n.id, n.properties->>'name' AS name, n.depth
FROM social_nodes docs
JOIN fts_search('social_nodes', 'database expert') s ON docs._id = s._id
CROSS JOIN LATERAL graph_neighbors('social', docs._id, 'KNOWS', 'both', 2) n
WHERE n.depth <= 2
ORDER BY s._score DESC;
```
Graph mutations can also be composed through LATERAL joins, enabling complex multi-step operations within a single SQL statement:
```sql
-- Create a node and an edge in a single query pipeline
WITH src AS (SELECT 'social:Person:1' AS source_id)
SELECT refreshed.since
FROM src
CROSS JOIN LATERAL graph_create_node(
'social', 'Person', '{"name": "Bob"}') AS cn(id)
CROSS JOIN LATERAL graph_create_edge(
'social', 'KNOWS', src.source_id, cn.id, '{}') AS ce(id)
CROSS JOIN LATERAL graph_update_edge(
'social', ce.id, '{"since": 2024}') AS ue(success)
JOIN LATERAL (
SELECT e.since FROM social_edges e WHERE e._id = ce.id
) refreshed ON true;
```
This pattern is the foundation for Cypher-to-SQL compilation: each Cypher clause (CREATE, MERGE, SET) is lowered to a LATERAL join step that can read results from previous steps and feed values to subsequent steps, all within a single transactional context.
This composability is the practical realization of the algebraic unification described in Chapter 3: graph posting lists, document posting lists, and full-text search posting lists all participate in the same query evaluation framework, enabling queries that span paradigms without data duplication or cross-system orchestration.
# Chapter 12: Cypher Query Language
Graph databases have long suffered from a fragmentation problem: users must choose between the expressiveness of graph query languages and the ubiquity of SQL. Cognica resolves this tension through an approach inspired by Apache AGE — Cypher queries are rewritten into SQL at parse time, allowing the full power of a declarative graph pattern language to flow through an existing SQL optimizer and executor. This chapter details the lexer, parser, abstract syntax tree, and the Cypher-to-SQL rewriting engine that makes this possible.
## 12.1 Architecture Overview
### 12.1.1 Why Cypher in a SQL Database
Graph traversal queries in SQL are notoriously verbose. Consider finding friends-of-friends who work at the same company as a given person. The pure SQL approach using graph table functions requires multiple `CROSS JOIN LATERAL` invocations, explicit endpoint tracking, and careful column aliasing:
```sql
SELECT DISTINCT f2_props->>'name' AS friend_name,
c_props->>'name' AS company
FROM graph_neighbors('social', 'social:Person:1001',
'KNOWS', 'outgoing', 1) f1
CROSS JOIN LATERAL graph_neighbors('social', f1.id,
'KNOWS', 'outgoing', 1) f2
CROSS JOIN LATERAL graph_edges('social', f2.id,
'WORKS_AT', 'outgoing') we
CROSS JOIN LATERAL graph_get_node('social', we.end_id) c
WHERE f2.id != 'social:Person:1001';
```
The same query in Cypher reads like a diagram of the graph pattern it describes:
```cypher
MATCH (a:Person {name: 'Alice'})-[:KNOWS]->()-[:KNOWS]->(fof:Person),
(fof)-[:WORKS_AT]->(c:Company)
WHERE fof <> a
RETURN DISTINCT fof.name AS friend_name, c.name AS company
```
The ASCII-art syntax of Cypher — parentheses for nodes, square brackets for relationships, arrows for direction — maps directly to the visual intuition that graph users carry. Variable-length paths, optional patterns, and mutating operations all benefit from this visual clarity. Rather than building an entirely separate graph query engine, Cognica translates Cypher into its existing SQL infrastructure at parse time, inheriting decades of query optimization research for free.
### 12.1.2 Parse-Time Rewriting vs Runtime Interpretation
Two fundamentally different strategies exist for integrating a second query language into a SQL engine:
**Runtime interpretation** treats the graph language as a black box. A table function receives the query string, executes it against the storage layer, and returns results. The SQL optimizer has no visibility into the internal structure of the graph query and cannot push predicates, reorder joins, or select indexes.
**Parse-time rewriting** translates the graph query into SQL AST nodes before any planning occurs. The SQL optimizer sees a standard subquery with joins, filters, and projections — it can apply predicate pushdown, join reordering, index selection, and all other optimization passes without modification.
Cognica adopts the parse-time rewriting approach, following the architecture established by Apache AGE (A Graph Extension for PostgreSQL). The `cypher()` function that appears in SQL is not a real function that executes at runtime. It is a *sentinel* that the AST builder intercepts during parse tree construction:
```sql
SELECT * FROM cypher('social', $$
MATCH (a:Person)-[:FOLLOWS*1..3]->(b:Person)
WHERE a.name = 'Alice'
RETURN b.name, b.role
$$) AS (name TEXT, role TEXT);
```
When the AST builder encounters `cypher()` in a FROM clause, it extracts the graph name and Cypher string, invokes the Cypher parser and translator, and replaces the function reference with the generated SQL subquery. From that point forward, every downstream component — plan builder, optimizer, physical planner, CVM compiler — operates on standard SQL AST nodes.
### 12.1.3 The cypher() Sentinel Function
The `cypher()` function follows Apache AGE conventions:
| Argument | Purpose | Example |
|----------|---------|---------|
| Graph name | Identifies the graph namespace | `'social'` |
| Cypher query | Dollar-quoted Cypher text | `$$ MATCH ... RETURN ... $$` |
| Parameters (optional) | JSON object for `$param` binding | `'{"name": "Alice"}'::jsonb` |
The `AS (column TYPE, ...)` clause following the function call defines the output schema. The number of columns in the AS clause must match the number of RETURN items in the Cypher query. This requirement is validated during rewriting and produces a clear error message on mismatch.
### 12.1.4 Overall Translation Flow
```mermaid
graph TB
SQL["SQL: SELECT * FROM cypher('g', $$ ... $$) AS (...)"]
PG["libpg_query SQL Parser"]
AST["AST Builder detects cypher()"]
LEX["Cypher Lexer"]
PARSE["Cypher Parser"]
CYAST["Cypher AST"]
REWRITE["Cypher-to-SQL Rewriter"]
SQLAST["SQL AST (subquery)"]
PLAN["SQL Planner + Optimizer"]
EXEC["CVM Execution"]
SQL --> PG
PG --> AST
AST --> LEX
LEX --> PARSE
PARSE --> CYAST
CYAST --> REWRITE
REWRITE --> SQLAST
SQLAST --> PLAN
PLAN --> EXEC
```
The flow consists of three major phases. First, the outer SQL is parsed by libpg_query, which treats `cypher()` as an ordinary function call. Second, the AST builder intercepts this function call and invokes the Cypher pipeline: lexical analysis, parsing into a Cypher AST, and rewriting into SQL AST nodes. Third, the generated SQL AST re-enters the standard planning pipeline, where it is indistinguishable from hand-written SQL.
## 12.2 Lexical Analysis
The Cypher lexer (`CypherLexer`) transforms raw query text into a stream of typed tokens. It is a hand-written scanner that handles Cypher's unique lexical requirements — pattern arrows, case-insensitive keywords, dollar-quoted parameters, and escaped identifiers.
### 12.2.1 Token Types
The lexer produces tokens of type `TokenType`, an enumeration with 29 variants:
```cpp
enum class TokenType {
// Delimiters
kLeftParen, kRightParen, // ( )
kLeftBracket, kRightBracket, // [ ]
kLeftBrace, kRightBrace, // { }
kComma, kColon, kDot, kDotDot, // , : . ..
kDollar, kPipe, // $ |
// Comparison and assignment
kEquals, kNotEquals, // = <>
kLessThan, kLessThanOrEqual, // < <=
kGreaterThan, kGreaterThanOrEqual, // > >=
kRegexMatch, // =~
// Arithmetic
kPlus, kMinus, kStar, kSlash, kPercent, // + - * / %
// Pattern arrows
kArrowLeft, kArrowRight, // <- ->
// Atoms
kIdentifier, kInteger, kFloat, kString,
// Control
kEof, kError,
};
```
Each `Token` carries its type, string value, and source location (line and column), enabling precise error reporting throughout the parsing and rewriting phases:
```cpp
struct Token {
TokenType type {TokenType::kError};
std::string value;
int32_t line {1};
int32_t column {1};
};
```
### 12.2.2 Multi-Character Operators and Pattern Arrows
Several Cypher operators consist of two characters. The lexer resolves these using ordered lookahead before falling through to single-character cases. The ordering matters: `<-` must be checked before `<` alone, and `..` before a single `.`:
```cpp
// cypher_lexer.cpp:103-137
if (c == '<' && peek_char_(1) == '-') {
advance_char_(); advance_char_();
return Token {TokenType::kArrowLeft, {}, start_line, start_column};
}
if (c == '-' && peek_char_(1) == '>') {
advance_char_(); advance_char_();
return Token {TokenType::kArrowRight, {}, start_line, start_column};
}
if (c == '.' && peek_char_(1) == '.') {
advance_char_(); advance_char_();
return Token {TokenType::kDotDot, {}, start_line, start_column};
}
```
The `<-` and `->` tokens are central to Cypher's visual pattern syntax. A relationship pattern like `(a)-[:KNOWS]->(b)` produces the token sequence: `(`, `a`, `)`, `-`, `[`, `:`, `KNOWS`, `]`, `->`, `(`, `b`, `)`. The `->` is a single token, not two separate tokens, which simplifies the parser's relationship direction logic.
The `..` token (range operator) appears exclusively in variable-length path specifications like `[:KNOWS*1..3]`, distinguishing it from the single `.` used for property access.
### 12.2.3 Case-Insensitive Keyword Recognition
Unlike SQL, which has a fixed set of reserved words handled by the grammar, Cypher keywords are recognized at the parser level through case-insensitive string comparison. The lexer emits all alphabetic sequences as `kIdentifier` tokens, and the parser checks whether an identifier matches a keyword using a helper function:
```cpp
// cypher_parser.cpp:19-31
auto iequals(std::string_view lhs, std::string_view rhs) -> bool {
if (lhs.size() != rhs.size()) {
return false;
}
for (size_t i = 0; i < lhs.size(); ++i) {
auto a = static_cast(lhs[i]);
auto b = static_cast(rhs[i]);
if (std::tolower(a) != std::tolower(b)) {
return false;
}
}
return true;
}
```
This design means `MATCH`, `match`, and `Match` are all valid. It also means that keywords like `MATCH` and `RETURN` are not reserved — they can be used as variable names in contexts where the parser does not expect a keyword. This permissiveness follows the openCypher specification's approach to context-sensitive keywords.
### 12.2.4 Multi-Word Keywords
Several Cypher constructs use multi-word keywords: `ORDER BY`, `STARTS WITH`, `ENDS WITH`, `IS NULL`, `IS NOT NULL`, and `OPTIONAL MATCH`. These are not tokenized as single tokens. Instead, the parser consumes them as sequences of identifiers. For example, `ORDER BY` is parsed as the keyword `ORDER` followed by the keyword `BY`:
```cpp
// cypher_parser.cpp:486-492
if (match_keyword_("ORDER")) {
expect_keyword_("BY");
body.order_by.push_back(parse_sort_item_());
while (match_(TokenType::kComma)) {
body.order_by.push_back(parse_sort_item_());
}
}
```
### 12.2.5 String Literals and Escape Sequences
Cypher strings can be delimited by either single or double quotes. The lexer supports standard escape sequences within strings: `\n` (newline), `\r` (carriage return), `\t` (tab), `\\` (backslash), `\'` (single quote), and `\"` (double quote). Unterminated strings produce an error token rather than silently consuming the rest of the input (`cypher_lexer.cpp:342-391`).
### 12.2.6 Escaped Identifiers and Parameters
Backtick-delimited identifiers (`` `my variable` ``) allow arbitrary characters in variable names. Doubled backticks within the delimiters represent a literal backtick. Parameters use the `$` prefix followed by an identifier (`$name`, `$personId`), tokenized as a `kDollar` token followed by a `kIdentifier`.
### 12.2.7 Position Tracking
The lexer maintains line and column counters that are updated on every character advance. Newline characters increment the line counter and reset the column to 1. Each token records its starting position, providing the foundation for error messages that include precise source locations (`cypher_lexer.cpp:426-433`).
## 12.3 Parsing
The Cypher parser (`CypherParser`) is a hand-written recursive descent parser that consumes the token stream and produces a typed abstract syntax tree. Recursive descent was chosen over parser generators (Bison/Flex, ANTLR) to avoid external dependencies and to enable precise error messages that reference Cypher-specific concepts.
### 12.3.1 Recursive Descent Design
The parser maintains a current token and a lexer reference. It advances through the token stream using a small set of primitives:
| Method | Behavior |
|--------|----------|
| `current_()` | Returns the current token without consuming it |
| `peek_()` | Returns the next token without consuming either |
| `advance_()` | Consumes the current token and loads the next |
| `expect_(type)` | Consumes the current token if it matches; throws on mismatch |
| `match_(type)` | Consumes and returns true if the current token matches; false otherwise |
| `check_keyword_(kw)` | Tests if the current token is the given keyword (case-insensitive) |
These primitives compose cleanly: `match_keyword_("OPTIONAL")` tests and consumes `OPTIONAL` in one call, while `check_keyword_("MATCH")` tests without consuming.
### 12.3.2 Grammar Structure
The parser implements the following grammar hierarchy:
```
Query -> ClauseSequence (UNION [ALL] ClauseSequence)*
ClauseSequence -> Clause+
Clause -> MatchClause
| CreateClause
| MergeClause
| DeleteClause
| SetClause
| RemoveClause
| ReturnClause
| WithClause
| UnwindClause
MatchClause -> [OPTIONAL] MATCH PatternPart (, PatternPart)*
[WHERE Expression]
PatternPart -> [variable =] PatternElement
PatternElement -> NodePattern (RelationshipChain)*
NodePattern -> ( [variable] [:Label]* [{properties}] )
RelationshipChain -> RelationshipPattern NodePattern
RelationshipPattern -> <-[details]- | -[details]-> | -[details]-
ProjectionBody -> [DISTINCT] ProjectionItem (, ProjectionItem)*
[ORDER BY SortItem (, SortItem)*]
[SKIP expr] [LIMIT expr]
```
### 12.3.3 Clause Parsing Dispatch
The `parse_clause_()` method inspects the current token and dispatches to the appropriate clause parser. It uses keyword checks rather than a token-type switch because Cypher keywords are context-sensitive identifiers:
```cpp
// cypher_parser.cpp:177-209
auto CypherParser::parse_clause_() -> ast::Clause {
if (check_keyword_("OPTIONAL") || check_keyword_("MATCH")) {
return ast::Clause {parse_match_clause_()};
}
if (check_keyword_("CREATE")) {
return ast::Clause {parse_create_clause_()};
}
if (check_keyword_("MERGE")) {
return ast::Clause {parse_merge_clause_()};
}
if (check_keyword_("DETACH") || check_keyword_("DELETE")) {
return ast::Clause {parse_delete_clause_()};
}
// ... remaining clauses ...
error_("Expected Cypher clause");
}
```
`OPTIONAL` must be checked alongside `MATCH` because the two-word keyword `OPTIONAL MATCH` begins with `OPTIONAL`. Similarly, `DETACH DELETE` begins with `DETACH`.
### 12.3.4 Pattern Parsing
Graph patterns are the syntactic heart of Cypher. A pattern consists of a chain of node-relationship-node segments:
```cypher
(a:Person {name: 'Alice'})-[:KNOWS]->(b:Person)-[:WORKS_AT]->(c:Company)
```
The parser handles this as a `PatternElement`: an initial `NodePattern` followed by zero or more `RelationshipChain` entries. Each chain contains a `RelationshipPattern` and a `NodePattern`.
Relationship direction detection works in two phases. First, the parser checks whether the relationship begins with `<-` (left arrow, indicating an incoming edge). If not, it expects a `-` (dash). After the optional bracket-enclosed details, the parser checks the trailing token: `->` sets direction to `kRight`, another `-` sets direction to `kBoth` (undirected):
```cpp
// cypher_parser.cpp:407-451
auto CypherParser::parse_relationship_pattern_()
-> ast::RelationshipPattern {
// ...
if (match_(TokenType::kArrowLeft)) {
pattern.direction = ast::RelationshipDirection::kLeft;
} else {
expect_(TokenType::kMinus);
}
// ... parse bracket contents ...
if (pattern.direction == ast::RelationshipDirection::kLeft) {
expect_(TokenType::kMinus);
} else if (match_(TokenType::kArrowRight)) {
pattern.direction = ast::RelationshipDirection::kRight;
} else {
expect_(TokenType::kMinus);
pattern.direction = ast::RelationshipDirection::kBoth;
}
return pattern;
}
```
Variable-length path specifications (`*1..3`, `*..5`, `*3`) are parsed when a `kStar` token appears inside the relationship brackets. The range parser handles four cases: both bounds present (`*1..3`), only minimum (`*3` is both min and max), only maximum (`*..5`), and unbounded (`*`):
```cpp
// cypher_parser.cpp:453-472
auto CypherParser::parse_relationship_range_()
-> ast::RelationshipRange {
auto range = ast::RelationshipRange {};
if (check_(TokenType::kInteger)) {
range.min_hops = std::stoll(advance_().value);
}
if (match_(TokenType::kDotDot)) {
if (check_(TokenType::kInteger)) {
range.max_hops = std::stoll(advance_().value);
}
return range;
}
if (range.min_hops.has_value()) {
range.max_hops = range.min_hops; // *3 means exactly 3 hops
}
return range;
}
```
### 12.3.5 Expression Parsing with Precedence Levels
Cypher expressions follow a precedence hierarchy implemented through a chain of parsing methods. Each level calls the next-higher-precedence level, and loops to handle left-associative operators at its own level:
| Precedence | Operators | Parser Method |
|------------|-----------|---------------|
| 1 (lowest) | OR | `parse_or_expression_()` |
| 2 | AND | `parse_and_expression_()` |
| 3 | NOT | `parse_not_expression_()` |
| 4 | = <> < <= > >= IN STARTS WITH ENDS WITH CONTAINS IS NULL IS NOT NULL =~ | `parse_comparison_expression_()` |
| 5 | + - | `parse_additive_expression_()` |
| 6 | * / % | `parse_multiplicative_expression_()` |
| 7 | Unary + - | `parse_unary_expression_()` |
| 8 | Property lookup, subscript, function call | `parse_postfix_expression_()` |
| 9 (highest) | Literals, variables, parameters, CASE, parenthesized | `parse_primary_expression_()` |
Negated predicates (`NOT IN`, `NOT STARTS WITH`, `NOT ENDS WITH`, `NOT CONTAINS`) are handled by lookahead in the comparison parser. When `NOT` is followed by one of these predicate keywords, the parser consumes `NOT`, parses the predicate, and wraps the result in a unary `NOT` node (`cypher_parser.cpp:600-643`).
### 12.3.6 Error Handling
Parse errors are reported through the `ParseError` structure, which carries a message and the source location (line and column). The `parse()` entry point wraps the parsing logic in a try-catch block and returns `std::expected`:
```cpp
// cypher_parser.cpp:127-139
auto CypherParser::parse()
-> std::expected {
try {
if (current_token_.type == TokenType::kError) {
error_(current_token_, current_token_.value);
}
return parse_query_();
} catch (const std::exception& e) {
if (error_info_.message.empty()) {
error_info_.message = e.what();
}
return std::unexpected(error_info_);
}
}
```
The `error_()` methods record the error location before throwing, ensuring that the caller receives both the message and the position in the Cypher source where parsing failed.
## 12.4 Abstract Syntax Tree
The Cypher AST (`cognica::sql::graph::cypher::ast`) represents the parsed structure of a Cypher query as a tree of typed nodes. It uses `std::variant` for sum types and `std::unique_ptr` for ownership, following the same conventions as the SQL AST elsewhere in the codebase.
### 12.4.1 Expression Nodes
Expressions are represented as a struct containing a `std::variant` of 19 expression types:
```cpp
using ExpressionData
= std::variant;
struct Expression {
ExpressionData data;
template
auto is() const -> bool {
return std::holds_alternative(data);
}
template
auto as() -> T& { return std::get(data); }
template
auto as() const -> const T& { return std::get(data); }
};
```
The `is()` and `as()` template methods provide a clean dispatch interface. The pattern `if (expr.is()) { const auto& lookup = expr.as(); ... }` appears throughout the translator.
Key expression types include:
| Type | Purpose | Example |
|------|---------|---------|
| `Variable` | Bound variable reference | `n`, `r` |
| `PropertyLookup` | Property access | `n.name` |
| `FunctionCall` | Function invocation | `id(n)`, `count(*)` |
| `Parameter` | Query parameter | `$name` |
| `BinaryExpression` | Binary operators | `a.age > 21` |
| `UnaryExpression` | Unary operators | `NOT x`, `n IS NULL` |
| `CaseExpression` | CASE/WHEN/THEN/ELSE | `CASE x WHEN 1 THEN 'a' END` |
| `ListComprehensionExpression` | List comprehension | `[x IN list WHERE x > 0 \| x * 2]` |
| `ExistsPatternExpression` | Pattern existence test | `exists((a)-->(b))` |
### 12.4.2 Clause Nodes
Clauses form the top-level structure of a Cypher query. Like expressions, they use a variant-based design:
```cpp
using ClauseData
= std::variant;
```
Each clause type carries the data specific to its syntax:
- **MatchClause**: An `optional` flag, a list of patterns, and an optional WHERE expression.
- **CreateClause**: A list of patterns describing nodes and edges to create.
- **MergeClause**: A single pattern and a list of `MergeAction` entries (ON MATCH SET, ON CREATE SET).
- **DeleteClause**: A `detach` flag and a list of variable names to delete.
- **SetClause**: A list of property assignment items (target expression = value expression).
- **RemoveClause**: A list of property removal items.
- **ReturnClause** and **WithClause**: A `ProjectionBody` containing projection items, ORDER BY, SKIP, and LIMIT.
- **UnwindClause**: An expression and a variable name for the unwound elements.
### 12.4.3 Pattern Nodes
Patterns represent the graph structure to match or create:
```cpp
struct NodePattern {
std::string variable; // Optional binding name
std::vector labels; // Zero or more labels
std::optional properties; // Inline property constraints
};
struct RelationshipPattern {
std::string variable;
std::vector types;
std::optional range;
std::optional properties;
RelationshipDirection direction; // kLeft, kRight, kBoth
};
struct RelationshipChain {
RelationshipPattern relationship;
NodePattern node;
};
struct PatternElement {
NodePattern node; // Starting node
std::vector chains; // Relationship-node pairs
};
struct PatternPart {
std::string variable; // Optional path variable
PatternElement element;
};
```
A `PatternElement` models the chain structure of Cypher patterns. The pattern `(a)-[r]->(b)-[s]->(c)` produces a `PatternElement` with starting node `a` and two chains: `(r, b)` and `(s, c)`. This representation directly mirrors the visual syntax and simplifies the translator's iteration over relationship hops.
### 12.4.4 Query Structure
At the top level, a `SingleQuery` contains a list of clauses and optional UNION parts:
```cpp
struct SingleQuery {
std::vector clauses;
std::vector unions;
};
struct UnionQueryPart {
bool all {false};
std::vector clauses;
};
```
This structure supports queries with UNION: `MATCH ... RETURN ... UNION ALL MATCH ... RETURN ...`.
## 12.5 Cypher-to-SQL Rewriting
The rewriter (`Translator` class in `cypher_rewriter.cpp`) is the most substantial component of the Cypher subsystem at over 5,000 lines. It transforms a Cypher AST into SQL AST nodes that the standard planner can process. The design follows the Apache AGE architecture: each clause contributes to a growing SQL `SelectStmt`, and scope boundaries (WITH clauses, MERGE) introduce subquery nesting.
### 12.5.1 Subquery Pipeline Architecture
The core insight of the Cypher-to-SQL translation is that each Cypher clause modifies or wraps the current SQL SELECT statement. The `translate_clause_sequence_()` method processes clauses in order:
```cpp
// cypher_rewriter.cpp:276-391
auto translate_clause_sequence_(
const std::vector& clauses)
-> std::expected {
auto select = sqlast::make_select_stmt();
for (size_t i = 0; i < clauses.size(); ++i) {
const auto& clause = clauses[i];
if (clause.is()) {
// Adds FROM/JOIN/WHERE to current select
translate_match_clause_(..., *select);
}
if (clause.is()) {
// Wraps current select in a subquery, creates new outer select
select = materialize_with_clause_(std::move(select), ...);
}
if (clause.is()) {
// Adds SELECT list, ORDER BY, LIMIT to current select
translate_projection_(..., *select, ...);
}
// ... other clauses ...
}
return sqlast::StmtPtr {std::move(select)};
}
```
**MATCH** clauses add FROM entries (for node/edge table scans) and JOIN clauses (for connecting nodes to edges) to the current SELECT. Multiple MATCH clauses in sequence add to the same SELECT, building up a multi-table join.
**WITH** clauses create a scope boundary. The current SELECT is wrapped inside a subquery, and a fresh outer SELECT is created. Variables projected by WITH become the only visible bindings in the new scope. This directly models Cypher's scoping semantics.
**RETURN** adds the SELECT list (target entries), ORDER BY, SKIP, and LIMIT to the final SELECT.
### 12.5.2 Variable Binding Tracking
The translator maintains a binding table (`std::unordered_map`) that maps Cypher variable names to their SQL representations. Each binding records:
```cpp
struct Binding {
BindingKind kind; // kNode, kEdge, or kScalar
std::string qualifier; // SQL table alias
std::string id_column; // Column containing _id
std::string label_column; // Column containing _label
std::string type_column; // Column containing _type (edges)
std::string source_column; // Column containing _source (edges)
std::string target_column; // Column containing _target (edges)
bool direct_properties; // Can access properties as direct columns?
bool is_path; // Is this a path variable?
};
```
When the translator encounters a variable reference like `n.name`, it looks up `n` in the binding table, determines whether `n` is a node or edge, and generates the appropriate SQL column reference. For nodes with direct property access, `n.name` becomes `t_n.name` (a direct column reference). After a scope boundary (WITH clause), properties may be carried through as explicit projected columns, and the binding's `property_columns` map tracks which properties are available.
### 12.5.3 Pattern Translation
#### Node Patterns
Each node variable in a MATCH pattern becomes a scan of the `{graph}_nodes` collection. For a pattern like `(n:Person {name: 'Alice'})`, the translator:
1. Acquires a binding for variable `n` with a fresh SQL alias (e.g., `__cy_n0`).
2. Adds a FROM or JOIN clause: `FROM social_nodes AS __cy_n0`.
3. Generates label constraints: `array_position(string_to_array(__cy_n0._label, ':'), 'Person') IS NOT NULL`.
4. Generates property constraints: `__cy_n0.name = 'Alice'`.
The label matching uses `string_to_array` and `array_position` rather than simple equality because a node can carry multiple labels stored as a colon-separated string (e.g., `Person:Employee`).
#### Edge Patterns
Relationships are translated to JOINs against the `{graph}_edges` collection. The join condition depends on the relationship direction:
| Direction | Edge Entry Condition | Node Entry Condition |
|-----------|---------------------|---------------------|
| Outgoing (`->`) | `node._id = edge._source` | `edge._target = next_node._id` |
| Incoming (`<-`) | `node._id = edge._target` | `edge._source = next_node._id` |
| Undirected (`-`) | `node._id = edge._source OR node._id = edge._target` | Complex bidirectional join |
For undirected edges, the translator generates an OR condition that accepts traversal in either direction:
```cpp
// cypher_rewriter.cpp:3988-4011
auto make_edge_entry_condition_(
const Binding& node_binding, const Binding& edge_binding,
cyast::RelationshipDirection direction)
-> std::expected {
if (direction == cyast::RelationshipDirection::kRight) {
return sqlast::make_binary_expr(
sqlast::BinaryOpType::kEqual, id_ref_(node_binding),
source_ref_(edge_binding));
}
if (direction == cyast::RelationshipDirection::kLeft) {
return sqlast::make_binary_expr(
sqlast::BinaryOpType::kEqual, id_ref_(node_binding),
target_ref_(edge_binding));
}
// Undirected: OR of both directions
auto left = sqlast::make_binary_expr(
sqlast::BinaryOpType::kEqual, id_ref_(node_binding),
source_ref_(edge_binding));
auto right = sqlast::make_binary_expr(
sqlast::BinaryOpType::kEqual, id_ref_(node_binding),
target_ref_(edge_binding));
return sqlast::make_binary_expr(
sqlast::BinaryOpType::kOr, std::move(left), std::move(right));
}
```
#### Multiple Relationship Types
When a relationship pattern specifies multiple types (`[:KNOWS|WORKS_WITH]`), each type becomes an equality predicate, and the predicates are combined with OR:
```cpp
// cypher_rewriter.cpp:3070-3078
if (!relationship.types.empty()) {
auto type_predicates = std::vector {};
for (const auto& type : relationship.types) {
type_predicates.push_back(sqlast::make_binary_expr(
sqlast::BinaryOpType::kEqual, type_ref_(binding),
sqlast::make_constant_string(type)));
}
predicates.push_back(combine_with_or_(type_predicates));
}
```
This generates SQL like: `__cy_r0._type = 'KNOWS' OR __cy_r0._type = 'WORKS_WITH'`.
### 12.5.4 Variable-Length Paths
Variable-length path patterns (e.g., `[:KNOWS*1..3]`) require recursive evaluation. The translator generates a `WITH RECURSIVE` common table expression (CTE) that iterates from the start node, following edges up to the maximum depth while tracking visited edges to prevent cycles.
#### Recursive CTE Structure
The generated CTE has five columns:
| Column | Purpose |
|--------|---------|
| `start_id` | ID of the path's starting node |
| `end_id` | ID of the current endpoint |
| `depth` | Number of hops traversed so far |
| `visited_edge_ids` | Array of edge IDs on the current path (cycle detection) |
| `visited_node_ids` | Array of node IDs on the current path |
The CTE consists of a base case (single-hop traversal from start nodes) and a recursive case (extending paths by one hop):
```mermaid
graph TB
subgraph "Recursive CTE: __cy_path0"
BASE["Base Case: depth = 1
FROM nodes JOIN edges JOIN nodes
WHERE label/type constraints"]
RECURSIVE["Recursive Case: depth + 1
FROM __cy_path0 JOIN edges JOIN nodes
WHERE depth < max_hops
AND edge NOT IN visited_edge_ids"]
BASE -->|"UNION ALL"| RECURSIVE
end
subgraph "Outer Query"
CTE_REF["FROM __cy_path0
JOIN nodes (start) JOIN nodes (end)
WHERE depth >= min_hops"]
end
RECURSIVE --> CTE_REF
```
The cycle detection mechanism uses `array_position` to check whether the current edge has already been visited:
```cpp
// cypher_rewriter.cpp:1998-2004
auto visited_lookup_args = make_binary_expr_list(
sqlast::make_column_ref(previous_alias, "visited_edge_ids"),
id_ref_(recursive_edge_scan));
recursive_conditions.push_back(sqlast::make_unary_expr(
sqlast::UnaryOpType::kIsNull,
sqlast::make_function_call("array_position",
std::move(visited_lookup_args))));
```
This translates to: `array_position(prev.visited_edge_ids, edge._id) IS NULL`, which is true only when the edge has not been visited.
#### Zero-Hop Paths
When the minimum hop count is 0 (e.g., `*0..3`), the translator generates a separate zero-hop SELECT that returns the start node as both the start and end of a zero-length path. This SELECT is UNION ALL'd with the recursive CTE.
#### Segmented Multi-Hop Patterns
When a pattern contains multiple relationship segments and at least one is variable-length, the translator splits the pattern into individual segments. Each segment is translated independently — fixed-length segments become standard JOINs, and variable-length segments become recursive CTEs. The segments are connected by shared node variables at their endpoints (`cypher_rewriter.cpp:686-774`).
### 12.5.5 Clause Translation
#### MATCH to FROM/JOIN/WHERE
A `MATCH` clause adds table scans and joins to the current SELECT. The translator iterates over the pattern's node-relationship chains, acquiring bindings for each variable and generating join conditions:
```cypher
MATCH (a:Person)-[:KNOWS]->(b:Person)
WHERE a.name = 'Alice'
```
Translates to:
```sql
SELECT ...
FROM social_nodes AS __cy_n0
JOIN social_edges AS __cy_r0
ON __cy_n0._id = __cy_r0._source
JOIN social_nodes AS __cy_n1
ON __cy_r0._target = __cy_n1._id
WHERE array_position(string_to_array(__cy_n0._label, ':'),
'Person') IS NOT NULL
AND __cy_n0.name = 'Alice'
AND array_position(string_to_array(__cy_n1._label, ':'),
'Person') IS NOT NULL
```
#### OPTIONAL MATCH to LEFT JOIN
`OPTIONAL MATCH` uses a LATERAL LEFT JOIN to ensure that the outer query's rows are preserved even when the pattern has no matches. The translator creates a sub-translator with a copy of the current bindings, translates the pattern in a subquery, and joins the subquery with a LEFT JOIN:
```cpp
// cypher_rewriter.cpp:2280-2289
auto join = std::make_unique();
join->join_type = sqlast::JoinType::kLeft;
join->join_condition = sqlast::make_constant_bool(true);
join->right_subquery = std::make_unique();
join->right_subquery->subquery = sqlast::StmtPtr {std::move(subselect)};
join->right_subquery->alias = "__cy_opt" + std::to_string(optional_index_++);
join->right_subquery->lateral = true;
```
The `lateral = true` flag ensures the subquery can reference columns from the outer query, which is necessary when the OPTIONAL MATCH pattern references variables bound by a preceding MATCH.
#### WITH as Scope Boundary
The `WITH` clause creates a subquery boundary. The current SELECT is materialized into a subquery, and the translator creates a fresh outer SELECT. Only the variables projected by WITH are visible in the subsequent scope:
```cypher
MATCH (a:Person)
WITH a.name AS name, count(*) AS cnt
WHERE cnt > 5
MATCH (b:Person {name: name})
RETURN b
```
The WITH clause produces a subquery `(SELECT a.name AS name, count(*) AS cnt FROM ...) AS __cy_scope0`, and the subsequent MATCH clause operates within a new SELECT that reads from `__cy_scope0`.
#### UNWIND to Table Function
`UNWIND` transforms a list into rows. The translator generates an `unnest()` table function call, joined laterally to the current query:
```cypher
UNWIND [1, 2, 3] AS x
RETURN x * 2
```
Becomes:
```sql
SELECT x * 2
FROM unnest(ARRAY[1, 2, 3]) AS __cy_unwind0(x)
```
When there is an existing FROM clause, the unnest is joined with a CROSS JOIN LATERAL to ensure it can reference outer columns (`cypher_rewriter.cpp:2173-2211`).
#### RETURN to SELECT
The `RETURN` clause translates to the SELECT list, with optional DISTINCT, ORDER BY, SKIP (OFFSET), and LIMIT. Graph variable references in RETURN are handled specially: bare node or edge variables are projected as JSON objects containing id, label, and properties, matching the Apache AGE output format (`cypher_rewriter.cpp:2625-2679`).
### 12.5.6 Expression Translation
#### Property Access
Property access on graph variables (`n.name`) is the most common expression in Cypher queries. The translator resolves it based on the binding's storage model:
- **Direct properties**: When the binding scans the node/edge collection directly, `n.name` becomes `__cy_n0.name` — a direct column reference on the collection.
- **Carried properties**: After a scope boundary, the property may be available as a projected column tracked in the binding's `property_columns` map.
- **Scalar bindings**: For variables bound by UNWIND or WITH to scalar values, property access becomes a JSON field access.
```cpp
// cypher_rewriter.cpp:3558-3574
auto translate_property_lookup_(
const cyast::PropertyLookup& lookup,
bool allow_identity_variables)
-> std::expected {
if (lookup.object->is()) {
const auto& variable = lookup.object->as();
auto binding = find_binding_(variable.name);
// ... path variable handling ...
return property_ref_(*binding, lookup.property);
}
// ... fallthrough to general expression translation ...
}
```
#### Function Mapping
Cypher functions are translated to their SQL equivalents. The translator recognizes function names case-insensitively and supports the `ag_catalog.` schema prefix for Apache AGE compatibility:
| Cypher Function | SQL Translation |
|----------------|-----------------|
| `id(n)` | `__cy_n0._id` |
| `labels(n)` | `string_to_array(__cy_n0._label, ':')` |
| `type(r)` | `__cy_r0._type` |
| `start_id(r)` | `__cy_r0._source` |
| `end_id(r)` | `__cy_r0._target` |
| `startNode(r)` | Lateral join to `graph_get_node` with `_source` |
| `endNode(r)` | Lateral join to `graph_get_node` with `_target` |
| `properties(n)` | Lateral join to `graph_get_node`, returns `properties` |
| `keys(n)` | `jsonb_object_keys_array(properties)` |
| `exists(n.prop)` | `jsonb_exists(properties, 'prop')` |
| `count(*)` | `count(*)` |
| `collect(x)` | Passed through as aggregate |
| `vertex_stats(n)` | Complex subquery with in/out degree counts |
The `count(*)` case requires special handling: the parser produces a `Wildcard` inside a `FunctionCall`, and the translator sets the `is_star` flag on the SQL function call node rather than translating arguments (`cypher_rewriter.cpp:3821-3826`).
General functions not in the mapping table are passed through with their name lowercased, allowing standard SQL aggregate and scalar functions to work transparently within Cypher queries.
#### String Predicates
Cypher's string predicates translate to SQL function calls:
| Cypher | SQL |
|--------|-----|
| `a STARTS WITH 'x'` | `starts_with(a, 'x')` |
| `a ENDS WITH 'x'` | `ends_with(a, 'x')` |
| `a CONTAINS 'x'` | `strpos(a, 'x') > 0` |
| `a =~ 'pattern'` | `a ~ 'pattern'` (regex match) |
| `a IN list` | `array_position(list, a) IS NOT NULL` |
The `IN` operator uses `array_position` rather than SQL's `IN` keyword because Cypher's `IN` operates on array values, not SQL row sets (`cypher_rewriter.cpp:3349-3365`).
#### NULL Handling
Cypher's `IS NULL` and `IS NOT NULL` map directly to their SQL counterparts. The `IS` keyword is parsed as part of the comparison expression handler, which checks for the `NOT` modifier between `IS` and `NULL` (`cypher_parser.cpp:588-598`).
#### CASE Expressions
Cypher CASE expressions support both simple form (`CASE expr WHEN val THEN ...`) and searched form (`CASE WHEN condition THEN ...`). Both translate directly to SQL CASE expressions with the same structure.
#### List Operations
List literals `[1, 2, 3]` translate to SQL `ARRAY[1, 2, 3]`. List subscript `list[0]` translates to array subscript with zero-based to one-based index conversion (Cypher uses 0-based indexing, SQL uses 1-based). Negative indices in Cypher index from the end, so the translator generates a CASE expression:
```cpp
// cypher_rewriter.cpp:3499-3511
// CASE WHEN index >= 0 THEN index + 1 ELSE index END
auto sql_case = std::make_unique();
sqlast::CaseWhen when_clause;
when_clause.condition = sqlast::make_binary_expr(
sqlast::BinaryOpType::kGreaterThanOrEqual,
clone(index), sqlast::make_constant_int(0));
when_clause.result = sqlast::make_binary_expr(
sqlast::BinaryOpType::kAdd, clone(index),
sqlast::make_constant_int(1));
sql_case->when_clauses.push_back(std::move(when_clause));
sql_case->else_result = std::move(*index);
```
List comprehensions `[x IN list WHERE x > 0 | x * 2]` translate to `ARRAY(SELECT x * 2 FROM unnest(list) AS t(x) WHERE x > 0)`, using a subquery with `unnest` and the array aggregate subquery type (`cypher_rewriter.cpp:3882-3968`).
### 12.5.7 Write Operation Handling
#### CREATE
The `CREATE` clause generates calls to `graph_create_node` and `graph_create_edge` table functions. These are not SQL INSERT statements — they invoke Cognica's graph storage API directly, which handles ID assignment, label indexing, and adjacency cache updates.
For each new node:
```cpp
// cypher_rewriter.cpp:986-1000
auto create_func = std::make_unique();
create_func->function_name = "graph_create_node";
create_func->arguments.push_back(
sqlast::make_constant_string(graph_name_));
create_func->arguments.push_back(
sqlast::make_constant_string(join_labels(node.labels)));
// ... properties as jsonb_build_object(...) ...
create_func->alias = create_alias;
create_func->output_columns.push_back("id");
```
After creation, the translator immediately looks up the created node to establish a binding with full metadata (id, label, properties), enabling subsequent clauses to reference the newly created entity.
For edges, the translator determines the actual source and target based on the relationship direction — a left-directed relationship `(a)<-[:KNOWS]-(b)` makes `b` the source and `a` the target (`cypher_rewriter.cpp:1066-1073`).
#### MERGE
MERGE implements find-or-create semantics. The translator builds two branches — a "matched" branch and a "created" branch — and combines them with UNION ALL:
```mermaid
graph TB
SCOPE["Scope CTE: current bindings"]
MATCHED["Matched Branch:
FROM scope
JOIN LATERAL (MATCH pattern) probe
WHERE probe found"]
CREATED["Created Branch:
FROM scope
LEFT JOIN LATERAL (MATCH pattern) probe
WHERE probe IS NULL
+ CREATE pattern"]
SCOPE --> MATCHED
SCOPE --> CREATED
UNION["UNION ALL"]
MATCHED --> UNION
CREATED --> UNION
RESULT["Result: all variables bound"]
UNION --> RESULT
```
The matched branch performs a LATERAL INNER JOIN against a subquery that attempts to match the pattern. If the pattern exists, the ON MATCH SET actions execute. The created branch performs a LATERAL LEFT JOIN against the same pattern subquery; rows where the probe returns NULL (pattern not found) pass through to the CREATE logic, and the ON CREATE SET actions execute.
This two-branch approach ensures that each input row produces exactly one output row, regardless of whether the pattern was matched or created.
#### DELETE and DETACH DELETE
The `DELETE` clause generates calls to `graph_delete_node` or `graph_delete_edge`. The translator validates that the target variable is bound to a node or edge (not a scalar), and orders deletions to process edges before nodes. `DETACH DELETE` passes a cascade flag to `graph_delete_node`, which removes all connected edges before removing the node:
```cpp
// cypher_rewriter.cpp:1170-1183
delete_func->function_name = target.binding.kind == BindingKind::kEdge
? "graph_delete_edge"
: "graph_delete_node";
delete_func->arguments.push_back(
sqlast::make_constant_string(graph_name_));
delete_func->arguments.push_back(id_ref_(target.binding));
if (target.binding.kind == BindingKind::kNode) {
delete_func->arguments.push_back(
sqlast::make_constant_bool(detach_nodes));
}
```
#### SET and REMOVE
The `SET` clause translates property assignments into `graph_update_node` or `graph_update_edge` calls. Multiple assignments to the same variable are grouped into a single update call with a `jsonb_build_object` argument containing all key-value pairs. Setting a property to NULL removes it. The `REMOVE` clause uses the same update mechanism, passing a `$delete` marker key with the property names to remove.
After each mutation (SET, REMOVE, CREATE), the translator re-establishes the affected binding by looking up the entity with its new state, ensuring that subsequent clauses see the updated values.
## 12.6 Supported Syntax Reference
The following table summarizes the Cypher syntax supported by Cognica's implementation:
| Category | Syntax | Supported |
|----------|--------|-----------|
| **Reading** | | |
| | `MATCH (n:Label)` | Yes |
| | `MATCH (n:Label {prop: value})` | Yes |
| | `MATCH (n:Label1:Label2)` — multi-label | Yes |
| | `MATCH (a)-[r:TYPE]->(b)` | Yes |
| | `MATCH (a)<-[r:TYPE]-(b)` | Yes |
| | `MATCH (a)-[r:TYPE]-(b)` — undirected | Yes |
| | `MATCH (a)-[r:T1\|T2]->(b)` — multiple types | Yes |
| | `MATCH (a)-[*1..3]->(b)` — variable-length | Yes |
| | `MATCH (a)-[*]->(b)` — unbounded | Yes |
| | `OPTIONAL MATCH` | Yes |
| | `WHERE` | Yes |
| **Projection** | | |
| | `RETURN expr AS alias` | Yes |
| | `RETURN DISTINCT` | Yes |
| | `RETURN *` | Yes |
| | `ORDER BY expr [ASC\|DESC]` | Yes |
| | `SKIP n` | Yes |
| | `LIMIT n` | Yes |
| **Composition** | | |
| | `WITH expr AS alias` | Yes |
| | `WITH ... WHERE` | Yes |
| | `UNWIND expr AS variable` | Yes |
| | `UNION [ALL]` | Yes |
| **Writing** | | |
| | `CREATE (n:Label {props})` | Yes |
| | `CREATE (a)-[:TYPE]->(b)` | Yes |
| | `MERGE (n:Label {props})` | Yes |
| | `MERGE ... ON MATCH SET` | Yes |
| | `MERGE ... ON CREATE SET` | Yes |
| | `SET n.prop = value` | Yes |
| | `SET n.prop = NULL` (property removal) | Yes |
| | `DELETE n` | Yes |
| | `DETACH DELETE n` | Yes |
| | `REMOVE n.prop` | Yes |
| **Expressions** | | |
| | Arithmetic: `+`, `-`, `*`, `/`, `%` | Yes |
| | Comparison: `=`, `<>`, `<`, `<=`, `>`, `>=` | Yes |
| | Boolean: `AND`, `OR`, `NOT` | Yes |
| | String: `STARTS WITH`, `ENDS WITH`, `CONTAINS` | Yes |
| | Regex: `=~` | Yes |
| | `IS NULL`, `IS NOT NULL` | Yes |
| | `IN` (list membership) | Yes |
| | `NOT IN`, `NOT STARTS WITH`, etc. | Yes |
| | CASE/WHEN/THEN/ELSE/END | Yes |
| | List literals: `[1, 2, 3]` | Yes |
| | Map literals: `{key: value}` | Yes |
| | List comprehension: `[x IN list WHERE p \| e]` | Yes |
| | Parameters: `$name` | Yes |
| **Functions** | | |
| | `id(n)`, `labels(n)`, `type(r)` | Yes |
| | `startNode(r)`, `endNode(r)` | Yes |
| | `start_id(r)`, `end_id(r)` | Yes |
| | `properties(n)`, `keys(n)`, `exists(n.p)` | Yes |
| | `count(*)`, `count(expr)` | Yes |
| | `collect(expr)` | Yes |
| | `sum`, `avg`, `min`, `max` | Yes |
| | `length(path)`, `nodes(path)`, `relationships(path)` | Yes |
| | `substring`, `left`, `right` | Yes |
| | `exists((a)-->(b))` — pattern exists | Yes |
| | `vertex_stats(n)` | Yes |
## 12.7 Translation Examples
This section presents complete translations from Cypher to SQL, illustrating the key rewriting patterns.
### 12.7.1 Simple MATCH with Property Filter
**Cypher:**
```cypher
MATCH (a:Person {name: 'Alice'})-[:KNOWS]->(b:Person)
RETURN b.name AS friend_name
```
**Generated SQL (conceptual):**
```sql
SELECT __cy_n1.name AS friend_name
FROM social_nodes AS __cy_n0
JOIN social_edges AS __cy_r0
ON __cy_n0._id = __cy_r0._source
JOIN social_nodes AS __cy_n1
ON __cy_r0._target = __cy_n1._id
WHERE array_position(
string_to_array(__cy_n0._label, ':'), 'Person'
) IS NOT NULL
AND __cy_n0.name = 'Alice'
AND __cy_r0._type = 'KNOWS'
AND array_position(
string_to_array(__cy_n1._label, ':'), 'Person'
) IS NOT NULL
```
The translation creates three table scans (two node collections, one edge collection) connected by join conditions that enforce the relationship direction. Label checks use `string_to_array` and `array_position` to support multi-label nodes. The property filter `{name: 'Alice'}` becomes a WHERE clause predicate.
### 12.7.2 Multi-Hop Traversal with JOIN Chain
**Cypher:**
```cypher
MATCH (a:Person)-[:KNOWS]->(b:Person)-[:WORKS_AT]->(c:Company)
WHERE a.name = 'Alice'
RETURN b.name, c.name AS company
```
**Generated SQL (conceptual):**
```sql
SELECT __cy_n1.name, __cy_n2.name AS company
FROM social_nodes AS __cy_n0
JOIN social_edges AS __cy_r0
ON __cy_n0._id = __cy_r0._source
JOIN social_nodes AS __cy_n1
ON __cy_r0._target = __cy_n1._id
JOIN social_edges AS __cy_r1
ON __cy_n1._id = __cy_r1._source
JOIN social_nodes AS __cy_n2
ON __cy_r1._target = __cy_n2._id
WHERE array_position(
string_to_array(__cy_n0._label, ':'), 'Person'
) IS NOT NULL
AND __cy_n0.name = 'Alice'
AND __cy_r0._type = 'KNOWS'
AND array_position(
string_to_array(__cy_n1._label, ':'), 'Person'
) IS NOT NULL
AND __cy_r1._type = 'WORKS_AT'
AND array_position(
string_to_array(__cy_n2._label, ':'), 'Company'
) IS NOT NULL
```
Each hop in the chain adds one edge scan and one node scan, connected by join conditions. The SQL optimizer can reorder these joins based on selectivity estimates, potentially starting from the most selective predicate (`a.name = 'Alice'`) and expanding outward.
### 12.7.3 Variable-Length Path with Recursive CTE
**Cypher:**
```cypher
MATCH (a:Person {name: 'Alice'})-[:KNOWS*1..3]->(b:Person)
RETURN b.name, b.age
```
**Generated SQL (conceptual):**
```sql
WITH RECURSIVE __cy_path0(start_id, end_id, depth,
visited_edge_ids, visited_node_ids) AS (
-- Base case: one hop
SELECT n0._id, n1._id, 1,
ARRAY[e0._id], ARRAY[n0._id, n1._id]
FROM social_nodes AS n0
JOIN social_edges AS e0 ON n0._id = e0._source
JOIN social_nodes AS n1 ON e0._target = n1._id
WHERE array_position(
string_to_array(n0._label, ':'), 'Person'
) IS NOT NULL
AND n0.name = 'Alice'
AND e0._type = 'KNOWS'
UNION ALL
-- Recursive case: extend by one hop
SELECT prev.start_id, n1._id, prev.depth + 1,
array_append(prev.visited_edge_ids, e1._id),
array_append(prev.visited_node_ids, n1._id)
FROM __cy_path0 AS prev
JOIN social_edges AS e1 ON prev.end_id = e1._source
JOIN social_nodes AS n1 ON e1._target = n1._id
WHERE prev.depth < 3
AND array_position(prev.visited_edge_ids, e1._id) IS NULL
AND e1._type = 'KNOWS'
)
SELECT __cy_n1.name, __cy_n1.age
FROM __cy_path0 AS __cy_path0
JOIN social_nodes AS __cy_n0
ON __cy_path0.start_id = __cy_n0._id
JOIN social_nodes AS __cy_n1
ON __cy_path0.end_id = __cy_n1._id
WHERE __cy_path0.depth >= 1
AND array_position(
string_to_array(__cy_n1._label, ':'), 'Person'
) IS NOT NULL
```
The recursive CTE maintains visited edge arrays for cycle detection, and the outer query applies the minimum depth and target label constraints.
### 12.7.4 WITH Aggregation and Scope Boundary
**Cypher:**
```cypher
MATCH (a:Person)-[:KNOWS]->(b:Person)
WITH a.name AS person, count(b) AS friend_count
WHERE friend_count > 5
RETURN person, friend_count
ORDER BY friend_count DESC
```
**Generated SQL (conceptual):**
```sql
SELECT __cy_scope0.person, __cy_scope0.friend_count
FROM (
SELECT __cy_n0.name AS person, count(*) AS friend_count
FROM social_nodes AS __cy_n0
JOIN social_edges AS __cy_r0
ON __cy_n0._id = __cy_r0._source
JOIN social_nodes AS __cy_n1
ON __cy_r0._target = __cy_n1._id
WHERE array_position(
string_to_array(__cy_n0._label, ':'), 'Person'
) IS NOT NULL
AND __cy_r0._type = 'KNOWS'
AND array_position(
string_to_array(__cy_n1._label, ':'), 'Person'
) IS NOT NULL
GROUP BY __cy_n0.name
) AS __cy_scope0
WHERE __cy_scope0.friend_count > 5
ORDER BY __cy_scope0.friend_count DESC
```
The WITH clause creates a subquery boundary. The inner query performs the join and aggregation, and the outer query applies the post-aggregation filter and final ordering.
### 12.7.5 MERGE Find-or-Create
**Cypher:**
```cypher
MATCH (a:Person {name: 'Alice'})
MERGE (a)-[:KNOWS]->(b:Person {name: 'Bob'})
ON CREATE SET b.created_at = '2024-01-01'
ON MATCH SET b.last_seen = '2024-06-15'
RETURN b.name
```
This generates a two-branch UNION ALL structure. The scope CTE captures the current state (the matched `a` variable). The matched branch performs a LATERAL INNER JOIN to find existing `(a)-[:KNOWS]->(b:Person {name: 'Bob'})` patterns and applies the ON MATCH SET. The created branch performs a LATERAL LEFT JOIN to the same pattern; where the probe returns NULL, it invokes `graph_create_node` and `graph_create_edge` and applies the ON CREATE SET. The UNION ALL of both branches ensures exactly one row per input, with the correct variable binding regardless of whether the pattern was matched or created.
## 12.8 Optimization Considerations
### 12.8.1 Predicate Pushdown Through Subquery Layers
Because the Cypher-to-SQL translation produces standard SQL AST nodes, the existing optimizer passes apply without modification. Predicate pushdown through subquery layers is handled by the plan optimizer's `pushdown_predicates_()` pass, which recognizes that filters on the outer query can often be pushed into subqueries.
For Cypher queries, this means that a WHERE clause in an outer SQL context wrapping a `cypher()` call can potentially be pushed down into the generated subquery:
```sql
SELECT * FROM cypher('social', $$
MATCH (n:Person) RETURN n.name, n.age
$$) AS (name TEXT, age INT)
WHERE age > 21;
```
The `age > 21` filter can be pushed below the subquery boundary, reducing the number of rows that flow through the pipeline.
### 12.8.2 Join Reordering for Graph Patterns
Multi-hop traversal patterns produce a chain of JOINs. The SQL optimizer's join reordering pass (described in Chapter 9) can evaluate different join orderings based on selectivity estimates. For a pattern like:
```cypher
MATCH (a:Person {name: 'Alice'})-[:KNOWS]->(b)-[:WORKS_AT]->(c:Company)
```
The optimizer might choose to start with the most selective scan (`a.name = 'Alice'`), then join edges, then join the next node — or it might start from a different point if statistics suggest a different ordering would be more efficient.
### 12.8.3 Index Utilization
The `_label` and `_type` columns on the node and edge collections are common filter targets. Indexes on these columns enable fast filtering, and the optimizer's index selection logic applies to the WHERE predicates generated by label and type constraints. Property constraints (e.g., `{name: 'Alice'}`) translate to equality predicates on collection columns, which benefit from column-level indexes.
### 12.8.4 Relationship Uniqueness
The openCypher specification requires that within a single MATCH clause, each relationship is traversed at most once per result row. Cognica's flat storage model and join-based translation naturally enforce this for fixed-length patterns, since each edge table scan produces distinct rows. For variable-length paths, the recursive CTE's `visited_edge_ids` array explicitly prevents revisiting edges, implementing cycle detection and relationship uniqueness simultaneously.
## 12.9 Summary
Cognica's Cypher implementation demonstrates that a graph query language need not require a separate execution engine. By rewriting Cypher into SQL at parse time, the system leverages the full power of an existing query optimizer and executor.
The architecture consists of three tightly integrated components:
1. **A hand-written lexer and parser** that transforms Cypher text into a typed abstract syntax tree with 19 expression variants and 9 clause types, supporting the core openCypher grammar including variable-length paths, list comprehensions, and pattern existence tests.
2. **A binding-aware translator** that converts the Cypher AST into SQL AST nodes through a clause-by-clause pipeline. Node patterns become table scans, relationship patterns become JOINs, variable-length paths become recursive CTEs, and write operations become graph storage function calls. The translator tracks variable bindings across scope boundaries, ensuring that property access, function calls, and aggregation all resolve correctly.
3. **Transparent integration with the SQL pipeline**, where the generated subquery passes through plan building, optimization (predicate pushdown, join reordering, index selection), physical planning, and CVM compilation without any Cypher-specific processing. The optimizer sees standard SQL constructs and applies its full repertoire of transformations.
The `cypher()` sentinel function — inspired by Apache AGE — provides a clean interface between the SQL and Cypher worlds. Users write graph patterns in Cypher's visual syntax, and the database executes them with the same efficiency as hand-written SQL joins. Chapter 3 provides the algebraic foundations that this translation preserves, and Chapter 11 describes the graph storage layer that the generated queries ultimately access.
# Chapter 13: CVM Architecture
## 13.1 Introduction to the Cognica Virtual Machine
The Cognica Virtual Machine (CVM) is a specialized bytecode interpreter designed for high-performance query execution. Unlike general-purpose virtual machines such as the JVM or CLR, CVM is purpose-built for database operations, with native support for document manipulation, aggregation, joins, sorting, and other query primitives.
CVM occupies a unique position in the database execution landscape. Traditional database engines use either interpreted expression evaluation (slow but flexible) or compile queries directly to native code (fast but complex). CVM provides a middle ground: a bytecode representation that is faster than interpretation, simpler than native compilation, and provides a clean target for optimization passes.
### 13.1.1 Design Philosophy
CVM's architecture reflects several key design principles:
1. **Query-Centric Instruction Set**: Rather than general-purpose operations, CVM provides opcodes specifically designed for database operations—field access, aggregation, sorting, and joins are first-class operations.
2. **Register-Based Architecture**: Like modern CPUs and high-performance VMs (Lua, Dalvik), CVM uses registers rather than a stack machine. This reduces instruction count and enables better optimization.
3. **Computed Goto Dispatch**: The interpreter uses computed goto (threaded code) for 10-20% faster dispatch compared to switch-based interpretation.
4. **Spillable Data Structures**: Memory-intensive operations (sort, hash join, aggregation) can automatically spill to disk when memory is exhausted.
5. **Zero-Copy Optimizations**: Composite rows enable multi-table joins without deep copying documents.
### 13.1.2 CVM in the Query Pipeline
CVM integrates into the query execution pipeline as follows:
```mermaid
flowchart LR
subgraph Frontend
SQL[SQL Query] --> Parser
Parser --> AST
end
subgraph Optimization
AST --> Analyzer[Semantic Analysis]
Analyzer --> LP[Logical Plan]
LP --> Optimizer
Optimizer --> PP[Physical Plan]
end
subgraph Compilation
PP --> Lowering[Plan Lowering]
Lowering --> IR[CVM IR]
IR --> CodeGen[Code Generator]
CodeGen --> BC[Bytecode Module]
end
subgraph Execution
BC --> Interpreter
Interpreter --> Results
end
style Frontend fill:#e3f2fd
style Optimization fill:#fff3e0
style Compilation fill:#e8f5e9
style Execution fill:#fce4ec
```
## 13.2 Architectural Overview
### 13.2.1 Core Components
The CVM system consists of several interconnected components:
| Component | Location | Purpose |
|-----------|----------|---------|
| Interpreter | `interpreter/interpreter.hpp` | Bytecode execution with computed-goto dispatch |
| VMContext | `interpreter/context.hpp` | Runtime state (registers, stacks, buffers) |
| Opcode System | `bytecode/opcode.hpp` | 512+ opcode definitions |
| Instruction Format | `bytecode/instruction.hpp` | 32-bit instruction encoding |
| BytecodeModule | `bytecode/bytecode_module.hpp` | Compiled bytecode unit |
| Value System | `types/value.hpp` | 24-byte tagged union |
| Type System | `types/type_system.hpp` | 16 CVM types |
### 13.2.2 Key Dimensions
| Dimension | Value |
|-----------|-------|
| Primary Opcodes | 256 (0x00-0xFF) |
| Extended Opcodes | 256 (via 0xFE prefix) |
| General Registers | 16 (R0-R15) |
| Float Registers | 8 (F0-F7) |
| CVM Types | 16 |
| VMValue Size | 24 bytes |
| Max Call Stack | 256 frames |
| Max Operand Stack | 1024 values |
| Max Cursors | 16 |
## 13.3 Type System
CVM implements a rich type system supporting SQL data types, JSON values, and internal execution types.
### 13.3.1 Type Enumeration
```cpp
enum class CVMType : uint8_t {
kNull = 0x00, // SQL NULL / JSON null
kBool = 0x01, // Boolean true/false
kInt64 = 0x02, // 64-bit signed integer
kDouble = 0x03, // IEEE 754 double precision
kString = 0x04, // UTF-8 string
kArray = 0x05, // Ordered collection
kDocument = 0x06, // Key-value object
kBinary = 0x07, // Raw byte array (BYTEA)
kTimestamp = 0x08, // TIMESTAMP WITHOUT TIME ZONE
kTimestampTZ = 0x09, // TIMESTAMP WITH TIME ZONE
kDate = 0x0A, // Date (year-month-day)
kTime = 0x0B, // Time of day
kInterval = 0x0C, // Time interval
kDecimal = 0x0D, // Arbitrary precision decimal
kUUID = 0x0E, // 128-bit UUID
kCompositeRow = 0x0F, // Zero-copy JOIN composite
kAggState = 0x10, // Aggregation state (internal)
kVoid = 0xFE, // No value (side-effect only)
kUnknown = 0xFF // Type unknown at compile time
};
```
### 13.3.2 Type Categories
Types are classified into categories for operation dispatch:
**Numeric Types** (support arithmetic):
- `kInt64`: 64-bit signed integer
- `kDouble`: IEEE 754 double-precision floating point
- `kDecimal`: Arbitrary precision decimal (128-bit coefficient)
**Temporal Types**:
- `kTimestamp`: Microseconds since Unix epoch (no timezone)
- `kTimestampTZ`: UTC microseconds with timezone awareness
- `kDate`: Days since Unix epoch
- `kTime`: Microseconds since midnight
- `kInterval`: PostgreSQL-style (months + days + microseconds)
**Container Types**:
- `kArray`: Ordered collection of heterogeneous values
- `kDocument`: Key-value map (JSON object)
**Special Types**:
- `kCompositeRow`: Zero-copy reference to multiple documents (for joins)
- `kAggState`: Internal aggregation state
### 13.3.3 VMValue Structure
All values in CVM are represented using a 24-byte tagged union:
```cpp
struct VMValue {
CVMType type; // 1 byte: type discriminant
uint8_t flags; // 1 byte: kFlagOwned, kFlagConst
uint16_t reserved; // 2 bytes: alignment padding
uint32_t padding; // 4 bytes: alignment padding
union { // 16 bytes: value storage
bool bool_val;
int64_t int64_val;
double double_val;
StringRef string_val; // (ptr, len)
ArrayRef array_val; // (ptr, len, cap)
Document* doc_val;
CompositeRow* composite_row_val;
TimestampVal timestamp_val;
DateVal date_val;
TimeVal time_val;
IntervalVal interval_val;
DecimalVal decimal_val;
UUIDVal uuid_val;
BinaryRef binary_val;
void* ptr_val;
uint8_t raw_bytes[16];
};
};
static_assert(sizeof(VMValue) == 24, "Cache-efficient size");
```
The 24-byte size is carefully chosen for cache efficiency—three VMValues fit exactly in a single 64-byte cache line on most modern processors.
### 13.3.4 Temporal Value Representations
**Timestamp**:
$$
timestamp = microseconds_{since\_epoch}
$$
Where epoch is 1970-01-01 00:00:00 UTC.
**Date**:
$$
date = days_{since\_epoch}
$$
**Time**:
$$
time = microseconds_{since\_midnight}
$$
**Interval** (PostgreSQL-compatible):
```cpp
struct IntervalVal {
int32_t months; // Total months
int32_t days; // Total days (separate for DST)
int64_t microseconds; // Remaining microseconds
};
```
The three-component interval representation follows PostgreSQL's design, allowing correct handling of month boundaries and daylight saving time transitions.
**Decimal** (128-bit):
```cpp
struct DecimalVal {
int64_t coefficient_high; // Upper 64 bits
int64_t coefficient_low; // Lower 64 bits
};
```
## 13.4 Instruction Encoding
CVM uses fixed-width 32-bit instructions with six primary formats, enabling efficient decoding and cache-friendly code layout.
### 13.4.1 Instruction Formats
**Format A: 3-Register (32 bits)**
```mermaid
packet-beta
0-7: "Opcode (8)"
8-11: "Dst (4)"
12-15: "Src1 (4)"
16-19: "Src2 (4)"
20-23: "Flags (4)"
24-31: "Reserved (8)"
```
**Format B: 2-Register + Imm16 (32 bits)**
```mermaid
packet-beta
0-7: "Opcode (8)"
8-11: "Dst (4)"
12-15: "Src (4)"
16-31: "Immediate (16)"
```
**Format C: Jump (32 bits)**
```mermaid
packet-beta
0-7: "Opcode (8)"
8-11: "Cond (4)"
12-15: "Reserved (4)"
16-31: "Offset (16, signed)"
```
**Format D: Extended Imm64 (96 bits)**
```mermaid
packet-beta
0-7: "Opcode (8)"
8-11: "Dst (4)"
12-31: "Reserved (20)"
32-95: "Immediate64 (64)"
```
**Format A** (3-operand register operations):
```
R[dst] = R[src1] OP R[src2]
```
Used for arithmetic, comparison, and logical operations.
**Format B** (2-operand with 16-bit immediate):
```
R[dst] = R[src] OP sign_extend(imm16)
```
Used for immediate operations and pool index references.
**Format C** (conditional jump):
```
if R[cond]: PC += offset * 4
```
Used for branches and loops. The 16-bit signed offset is measured in words (4 bytes), providing a range of approximately +/- 128KB.
**Format D** (64-bit immediate):
```
R[dst] = imm64
```
Used for loading large constants (8-byte instruction total).
**Format E** (unary operations):
```
R[dst] = OP R[src]
```
Used for type conversions, negation, and other single-operand operations.
**Format F** (no operands):
```
HALT, NOP, YIELD
```
Used for control operations with no data dependencies.
### 13.4.2 Extended Instruction Formats
For complex operations like composite rows, CVM uses extended formats:
**Format G** (register + pool index):
```
[Opcode:8][Dst:4][Reserved:4][PoolIdx:8][Reserved:8]
```
Used for batch constant instructions and pool references.
**Format H** (two-word extended):
```
Word 1: [0xFE:8][ExtOpcode:8][Dst:8][Src:8]
Word 2: [Operand1:16][Operand2:16]
```
Used for composite row operations requiring 16-bit operands.
### 13.4.3 Encoding/Decoding
Instructions are encoded and decoded using helper functions:
```cpp
// Encoding
auto encode_format_a(Opcode op, uint8_t dst, uint8_t src1,
uint8_t src2, uint8_t flags) -> uint32_t {
return (static_cast(op) << 24) |
(static_cast(dst) << 20) |
(static_cast(src1) << 16) |
(static_cast(src2) << 12) |
(static_cast(flags) << 8);
}
// Decoding
auto decode_opcode(uint32_t word) -> Opcode {
return static_cast(word >> 24);
}
auto decode_dst(uint32_t word) -> uint8_t {
return (word >> 20) & 0xF;
}
```
## 13.5 Opcode System
CVM defines over 512 opcodes organized into functional categories.
### 13.5.1 Opcode Organization
```
0x00-0x0F: Data Movement
0x10-0x1F: Integer Arithmetic
0x20-0x2F: Floating Point Arithmetic
0x30-0x3F: Bitwise Operations
0x40-0x4F: Integer Comparison
0x50-0x57: Float Comparison
0x58-0x5F: String Comparison
0x60-0x6F: Logical Operations & Debug
0x70-0x7F: Control Flow
0x80-0x8F: Type Operations
0x90-0x9F: Field Access
0xA0-0xAF: Array Operations
0xB0-0xBF: String Operations
0xC0-0xCF: Aggregation
0xD0-0xDF: Query Buffers (Window, Hash, Sort, Set)
0xE0-0xEF: Working Tables & Functions
0xF0-0xF9: Cursor Operations
0xFA-0xFD: External Calls
0xFE: Extended Opcode Prefix
0xFF: Undefined
```
### 13.5.2 Data Movement Operations (0x00-0x0F)
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0x00 | NOP | No operation |
| 0x01 | MOVE | R[dst] = R[src] |
| 0x02 | MOVE_INT64 | R[dst] = imm64 |
| 0x03 | MOVE_FLOAT64 | R[dst] = imm64 (as double) |
| 0x04 | MOVE_NULL | R[dst] = NULL |
| 0x05 | MOVE_TRUE | R[dst] = true |
| 0x06 | MOVE_FALSE | R[dst] = false |
| 0x07 | LOAD_CONST | R[dst] = constant_pool[imm16] |
| 0x08 | LOAD_STRING | R[dst] = string_pool[imm16] |
| 0x09 | LOAD_ARRAY | R[dst] = array_pool[imm16] |
| 0x0A | LOAD_DOC | R[dst] = doc_pool[imm16] |
| 0x0B | STORE_REG | temp_stack[imm16] = R[src] |
| 0x0C | LOAD_REG | R[dst] = temp_stack[imm16] |
| 0x0D | LOAD_PARAM | R[dst] = parameters[imm16] |
| 0x0E | LOAD_INPUT | R[dst] = input_document |
### 13.5.3 Arithmetic Operations (0x10-0x2F)
**Integer Arithmetic** (0x10-0x1F):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0x10 | ADD_I64 | R[dst] = R[src1] + R[src2] |
| 0x11 | SUB_I64 | R[dst] = R[src1] - R[src2] |
| 0x12 | MUL_I64 | R[dst] = R[src1] * R[src2] |
| 0x13 | DIV_I64 | R[dst] = R[src1] / R[src2] |
| 0x14 | MOD_I64 | R[dst] = R[src1] % R[src2] |
| 0x15 | NEG_I64 | R[dst] = -R[src] |
| 0x16 | ABS_I64 | R[dst] = abs(R[src]) |
| 0x17 | INC_I64 | R[dst] = R[src] + 1 |
| 0x18 | DEC_I64 | R[dst] = R[src] - 1 |
**Floating Point Arithmetic** (0x20-0x2F):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0x20 | ADD_F64 | R[dst] = R[src1] + R[src2] |
| 0x21 | SUB_F64 | R[dst] = R[src1] - R[src2] |
| 0x22 | MUL_F64 | R[dst] = R[src1] * R[src2] |
| 0x23 | DIV_F64 | R[dst] = R[src1] / R[src2] |
| 0x24 | NEG_F64 | R[dst] = -R[src] |
| 0x25 | ABS_F64 | R[dst] = abs(R[src]) |
| 0x26 | SQRT | R[dst] = sqrt(R[src]) |
| 0x27 | POW | R[dst] = pow(R[src1], R[src2]) |
| 0x28 | FLOOR | R[dst] = floor(R[src]) |
| 0x29 | CEIL | R[dst] = ceil(R[src]) |
| 0x2A | ROUND | R[dst] = round(R[src]) |
### 13.5.4 Comparison Operations (0x40-0x5F)
**Integer Comparison** (0x40-0x4F):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0x40 | CMP_EQ_I64 | R[dst] = (R[src1] == R[src2]) |
| 0x41 | CMP_NE_I64 | R[dst] = (R[src1] != R[src2]) |
| 0x42 | CMP_LT_I64 | R[dst] = (R[src1] < R[src2]) |
| 0x43 | CMP_LE_I64 | R[dst] = (R[src1] <= R[src2]) |
| 0x44 | CMP_GT_I64 | R[dst] = (R[src1] > R[src2]) |
| 0x45 | CMP_GE_I64 | R[dst] = (R[src1] >= R[src2]) |
**String Comparison** (0x58-0x5F):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0x58 | CMP_EQ_STR | R[dst] = (R[src1] == R[src2]) |
| 0x59 | CMP_NE_STR | R[dst] = (R[src1] != R[src2]) |
| 0x5A | CMP_LT_STR | R[dst] = (R[src1] < R[src2]) |
| 0x5B | CMP_LE_STR | R[dst] = (R[src1] <= R[src2]) |
### 13.5.5 Control Flow Operations (0x70-0x7F)
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0x70 | JUMP | PC += offset * 4 |
| 0x71 | JUMP_TRUE | if R[cond]: PC += offset * 4 |
| 0x72 | JUMP_FALSE | if !R[cond]: PC += offset * 4 |
| 0x73 | JUMP_NULL | if R[cond] is NULL: PC += offset * 4 |
| 0x74 | JUMP_NOT_NULL | if R[cond] is not NULL: PC += offset * 4 |
| 0x75 | CALL | Push frame, jump to function |
| 0x76 | RETURN | Pop frame, return value |
| 0x77 | HALT | Stop execution |
| 0x78 | YIELD | Yield for streaming |
### 13.5.6 Field Access Operations (0x90-0x9F)
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0x90 | GET_FIELD | R[dst] = R[doc].field[pool_idx] |
| 0x91 | SET_FIELD | R[doc].field[pool_idx] = R[src] |
| 0x92 | HAS_FIELD | R[dst] = R[doc].has(field[pool_idx]) |
| 0x93 | GET_NESTED | R[dst] = R[doc].path[pool_idx] |
| 0x94 | DELETE_FIELD | R[doc].delete(field[pool_idx]) |
In addition to the primary field access opcodes above, the extended opcode `GET_OUTER_FIELD` (ExtOp 0x85) reads a field from the outer query's current row rather than the current document. This opcode is emitted by the plan lowering pass when compiling correlated subqueries — column references whose table alias does not belong to the inner query's local alias set are lowered to `GET_OUTER_FIELD` instead of `GET_FIELD`. The interpreter resolves the field from the outer row context, supporting both Document and CompositeRow outer rows. This enables the CVM to execute correlated subqueries natively without falling back to the Volcano executor.
### 13.5.7 Aggregation Operations (0xC0-0xCF)
**Single Aggregation** (0xC0-0xC7):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0xC0 | AGG_INIT | Initialize aggregation state |
| 0xC1 | AGG_ACCUMULATE | Accumulate value into state |
| 0xC2 | AGG_FINALIZE | Compute final aggregate result |
| 0xC3 | AGG_MERGE | Merge two aggregate states |
**Aggregation Table** (0xC8-0xCF):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0xC8 | AGG_TABLE_NEW | Create aggregation table |
| 0xC9 | AGG_GET_OR_CREATE | Get/create group state |
| 0xCA | AGG_ITER_INIT | Initialize table iterator |
| 0xCB | AGG_ITER_NEXT | Get next group |
| 0xCC | AGG_TABLE_NEW_MULTI | Multi-function agg table |
| 0xCD | AGG_STATE_AT | Access state at index |
| 0xCE | AGG_ITER_HAS_NEXT | Check for more groups |
### 13.5.8 Query Buffer Operations (0xD0-0xDF)
**Window Buffer** (0xD0-0xD3):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0xD0 | WIN_BUF_NEW | Create window buffer |
| 0xD1 | WIN_BUF_ADD | Add row to window |
| 0xD2 | WIN_COMPUTE | Compute window function |
| 0xD3 | WIN_BUF_NEXT | Get next windowed row |
**Hash Table** (0xD4-0xD7):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0xD4 | HASH_TABLE_NEW | Create hash table |
| 0xD5 | HASH_TABLE_INSERT | Insert key-value pair |
| 0xD6 | HASH_TABLE_PROBE | Probe for key |
| 0xD7 | HASH_TABLE_DESTROY | Destroy hash table |
**Sort Buffer** (0xD8-0xDB):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0xD8 | SORT_BUF_NEW | Create sort buffer |
| 0xD9 | SORT_BUF_ADD | Add row to buffer |
| 0xDA | SORT_BUF_NEXT | Get next sorted row |
| 0xDB | SORT_BUF_DESTROY | Destroy sort buffer |
**Set Operations** (0xDC-0xDF):
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0xDC | SET_OP_NEW | Create set operation buffer |
| 0xDD | SET_OP_ADD | Add row from source |
| 0xDE | SET_OP_NEXT | Get next result row |
| 0xDF | SET_OP_DESTROY | Destroy set buffer |
### 13.5.9 Cursor Operations (0xF0-0xF7)
| Opcode | Mnemonic | Operation |
|--------|----------|-----------|
| 0xF0 | CURSOR_OPEN | Open collection cursor |
| 0xF1 | CURSOR_NEXT | Get next document |
| 0xF2 | CURSOR_CLOSE | Close cursor |
| 0xF3 | CURSOR_RESET | Reset to beginning |
| 0xF4 | CURSOR_IS_VALID | Check if more rows |
| 0xF5 | EMIT_ROW | Emit result row |
| 0xF6 | CURSOR_TAKE_DOC | Move document (no copy) |
### 13.5.10 Extended Opcodes (0xFE prefix)
Extended opcodes use a two-byte opcode sequence (0xFE + extended opcode):
**Array/Object Iteration** (0x01-0x0A):
- `ITER_ARRAY_BEGIN`, `ITER_ARRAY_NEXT`, `ITER_ARRAY_END`
- `ITER_OBJECT_BEGIN`, `ITER_OBJECT_NEXT_KEY`, `ITER_OBJECT_NEXT_VAL`, `ITER_OBJECT_END`
- `ITER_RANGE_BEGIN`, `ITER_RANGE_NEXT`, `ITER_RANGE_END`
**Document Construction** (0x0B-0x12):
- `DOC_NEW`, `DOC_FROM_JSON`, `DOC_TO_JSON`
- `DOC_MERGE`, `DOC_DEEP_COPY`
**Composite Row Operations** (0x13-0x1C):
- `COMPOSITE_NEW`, `COMPOSITE_ADD_SLOT`
- `COMPOSITE_GET_FIELD`, `COMPOSITE_GET_SLOT`
- `COMPOSITE_MATERIALIZE`, `COMPOSITE_EMIT`
**Batch/Vectorized Operations** (0x20-0x5F):
- `BATCH_SCAN_OPEN`, `BATCH_SCAN_NEXT`, `BATCH_EMIT`
- `BATCH_ADD_I64`, `BATCH_CMP_LT_I64`, etc.
- `BATCH_AND`, `BATCH_OR`, `BATCH_SELECT`
- `BATCH_HASH`, `BATCH_HASH_BUILD`, `BATCH_HASH_PROBE`
**Parallel Execution** (0x5C-0x67):
- `PARALLEL_SCAN_OPEN`, `PARALLEL_FILTER_EVAL`
- `PARALLEL_PARTITION`, `PARALLEL_MERGE`, `PARALLEL_BARRIER`
**Outer Context Operations** (0x85):
- `GET_OUTER_FIELD`: Read a field from the outer query's current row for correlated subquery support
## 13.6 Register Architecture
CVM employs a register-based architecture with 24 registers organized into two files.
### 13.6.1 Register Organization
```mermaid
flowchart TB
subgraph GPR["General Purpose Registers (16)"]
R0[R0] --- R1[R1] --- R2[R2] --- R3[R3]
R4[R4] --- R5[R5] --- R6[R6] --- R7[R7]
R8[R8] --- R9[R9] --- R10[R10] --- R11[R11]
R12[R12] --- R13[R13] --- R14[R14] --- R15[R15]
end
subgraph FPR["Float Registers (8)"]
F0[F0] --- F1[F1] --- F2[F2] --- F3[F3]
F4[F4] --- F5[F5] --- F6[F6] --- F7[F7]
end
subgraph Special["Special Registers"]
PC[PC: Program Counter]
end
```
**General Purpose Registers (R0-R15)**:
- 16 registers for scalar values
- Can hold any CVMType
- Used for integers, booleans, pointers
**Float Registers (F0-F7)**:
- 8 registers for floating-point values
- Optimized for double-precision arithmetic
- Separate to avoid type checks in hot loops
### 13.6.2 Register Storage
```cpp
class VMContext {
private:
std::array registers_; // R0-R15
std::array float_registers_; // F0-F7
uint32_t pc_; // Program counter
};
```
### 13.6.3 Register Access
```cpp
auto VMContext::get_register(uint8_t reg) const -> const VMValue& {
assert(reg < 16);
return registers_[reg];
}
void VMContext::set_register(uint8_t reg, VMValue value) {
assert(reg < 16);
registers_[reg] = std::move(value);
}
// Convenience accessors for common types
auto VMContext::get_int64(uint8_t reg) const -> int64_t {
assert(registers_[reg].type == CVMType::kInt64);
return registers_[reg].int64_val;
}
void VMContext::set_int64(uint8_t reg, int64_t value) {
registers_[reg].type = CVMType::kInt64;
registers_[reg].int64_val = value;
}
```
### 13.6.4 Register vs Stack Architecture
CVM's register-based design offers several advantages over stack machines:
| Property | Register Machine | Stack Machine |
|----------|-----------------|---------------|
| Instruction count | Lower | Higher |
| Operand encoding | Explicit registers | Implicit stack |
| Code size | Larger instructions | Smaller instructions |
| Optimization | Easier | Harder |
| Dispatch overhead | Lower | Higher |
For database workloads, the reduced instruction count and better optimization potential outweigh the slightly larger code size.
## 13.7 Execution Context
The `VMContext` maintains all runtime state during bytecode execution.
### 13.7.1 State Categories
```mermaid
flowchart TB
subgraph VMContext
subgraph Registers
GPR[General Purpose
R0-R15]
FPR[Float
F0-F7]
PC[Program Counter]
end
subgraph Stacks
OS[Operand Stack
1024 values]
CS[Call Stack
256 frames]
end
subgraph Buffers
SORT[Sort Buffers
x4]
HASH[Hash Tables
x4]
AGG[Agg Tables
x4]
WIN[Window Buffers
x4]
SET[Set Op Buffers
x4]
WORK[Working Tables
x8]
end
subgraph Cursors
CUR[Cursors
x16]
ITER[Iterators
x16]
end
subgraph IO
INPUT[Input Document]
EMIT[Row Emit Callback]
PARAMS[Parameters
x256]
end
end
```
### 13.7.2 Context Structure
```cpp
class VMContext {
private:
// === Registers ===
std::array registers_;
std::array float_registers_;
uint32_t pc_;
// === Control Flow ===
std::vector operand_stack_;
std::vector call_stack_;
// === Status ===
VMStatus status_;
VMFlags flags_;
std::optional error_;
// === Module ===
const BytecodeModule* module_;
// === I/O ===
VMValue input_document_;
RowEmitCallback row_emit_callback_;
uint64_t rows_emitted_;
uint64_t rows_scanned_;
// === Cursors ===
std::array, 16> cursors_;
std::array current_documents_;
std::array cursor_valid_;
// === Query Buffers (spillable) ===
std::array sort_buffers_;
std::array hash_tables_;
std::array agg_tables_;
std::array window_buffers_;
std::array set_op_buffers_;
std::array working_tables_;
// === Memory Management ===
std::vector> owned_documents_;
std::vector> owned_strings_;
std::vector>> owned_arrays_;
std::vector> owned_composite_rows_;
// === Parameters ===
std::array parameters_;
std::bitset<256> parameters_bound_;
// === Statistics ===
uint64_t instructions_executed_;
};
```
### 13.7.3 Status and Flags
**Execution Status**:
```cpp
enum class VMStatus : uint8_t {
kRunning = 0, // Normal execution
kHalted = 1, // Execution completed
kError = 2, // Runtime error
kBreakpoint = 3, // Hit debugger breakpoint
kYielded = 4 // Yielded for streaming
};
```
**CPU Flags** (set by comparisons):
```cpp
struct VMFlags {
bool zero = false; // Result was zero/equal
bool negative = false; // Result was negative
bool overflow = false; // Arithmetic overflow
bool carry = false; // Carry/borrow
};
```
### 13.7.4 Resource Limits
| Resource | Limit | Purpose |
|----------|-------|---------|
| Call Stack | 256 frames | Prevent stack overflow |
| Operand Stack | 1024 values | Function argument passing |
| Cursors | 16 | Concurrent table scans |
| Sort Buffers | 4 | Concurrent ORDER BY |
| Hash Tables | 4 | Concurrent joins |
| Agg Tables | 4 | Concurrent GROUP BY |
| Window Buffers | 4 | Window functions |
| Set Op Buffers | 4 | UNION/INTERSECT/EXCEPT |
| Working Tables | 8 | Recursive CTEs |
| Parameters | 256 | Prepared statement bindings |
## 13.8 Interpreter Implementation
The CVM interpreter uses computed goto (threaded code) for efficient dispatch.
### 13.8.1 Dispatch Table
```cpp
class Interpreter {
private:
// Dispatch table: 256 void* pointers to instruction handlers
static void* dispatch_table_[256];
public:
auto execute(VMContext& ctx) -> ExecutionResult;
};
```
### 13.8.2 Computed Goto Dispatch
The interpreter main loop uses computed goto for minimal dispatch overhead:
```cpp
auto Interpreter::execute(VMContext& ctx) -> ExecutionResult {
// Initialize dispatch table (once)
static void* dispatch_table[] = {
&&op_nop, // 0x00
&&op_move, // 0x01
&&op_move_int64, // 0x02
// ... 256 entries
};
// Fetch first instruction
auto word = ctx.fetch_instruction();
auto opcode = word >> 24;
goto *dispatch_table[opcode];
op_nop:
ctx.advance_pc(4);
word = ctx.fetch_instruction();
opcode = word >> 24;
goto *dispatch_table[opcode];
op_move:
{
auto dst = (word >> 20) & 0xF;
auto src = (word >> 16) & 0xF;
ctx.set_register(dst, ctx.get_register(src));
}
ctx.advance_pc(4);
word = ctx.fetch_instruction();
opcode = word >> 24;
goto *dispatch_table[opcode];
op_add_int64:
{
auto dst = (word >> 20) & 0xF;
auto src1 = (word >> 16) & 0xF;
auto src2 = (word >> 12) & 0xF;
auto a = ctx.get_int64(src1);
auto b = ctx.get_int64(src2);
ctx.set_int64(dst, a + b);
}
ctx.advance_pc(4);
word = ctx.fetch_instruction();
opcode = word >> 24;
goto *dispatch_table[opcode];
// ... handlers for all 256 opcodes
op_halt:
ctx.set_status(VMStatus::kHalted);
return ExecutionResult::kSuccess;
}
```
### 13.8.3 Why Computed Goto?
Computed goto provides significant performance benefits:
1. **No Bounds Check**: Direct table lookup, no range validation
2. **Direct Jumps**: Each handler jumps directly to the next
3. **Better Branch Prediction**: Dedicated indirect branch per handler
4. **10-20% Faster**: Measured improvement over switch-case
**Comparison with Switch**:
```cpp
// Switch-based dispatch (slower)
while (running) {
auto opcode = fetch_opcode();
switch (opcode) {
case OP_ADD: /* handler */ break;
case OP_SUB: /* handler */ break;
// ...
}
}
// Computed goto (faster)
goto *dispatch_table[opcode];
op_add:
/* handler */
goto *dispatch_table[next_opcode];
```
The switch version has a single indirect branch that must predict across all opcodes. Computed goto gives each handler its own branch, allowing the branch predictor to learn patterns specific to that instruction.
### 13.8.4 Execution Configuration
```cpp
struct InterpreterConfig {
uint64_t max_instructions = 100'000'000'000; // Anti-DoS limit
bool enable_tracing = false; // Debug tracing
bool enable_profiling = false; // Performance profiling
};
```
### 13.8.5 Execution Result
```cpp
enum class ExecutionResult : uint8_t {
kSuccess, // Completed normally
kError, // Runtime error occurred
kBreakpoint, // Hit debugger breakpoint
kMaxInstructions, // Exceeded instruction limit
kYield // Yielded for streaming
};
```
## 13.9 Specialized Execution Buffers
CVM provides specialized data structures for memory-intensive query operations, all capable of spilling to disk when memory is exhausted.
### 13.9.1 Sort Buffer
The sort buffer implements ORDER BY with automatic external sort fallback:
```cpp
class SortBuffer {
public:
// Field-based sorting
void configure(const std::vector& comparators);
// Value-based sorting (for expression keys)
void configure_with_values(size_t num_keys,
const std::vector& ascending,
const std::vector& nulls_first);
void add(Document* row);
void add_with_values(Document* row, std::vector keys);
auto next() -> Document*; // Sorts on first call
void destroy();
private:
std::vector rows_;
std::vector comparators_;
size_t memory_limit_;
bool sorted_ = false;
};
```
**Spill Strategy**:
When memory is exceeded, the buffer:
1. Sorts current in-memory data
2. Writes sorted run to temporary file
3. Clears memory
4. Continues accepting rows
5. On iteration, performs merge sort across runs
### 13.9.2 Hash Table
The hash table supports hash joins and hash aggregation:
```cpp
class HashTable {
public:
void insert(const VMValue& key, Document* value);
auto probe_first(const VMValue& key) -> Document*;
auto probe_next() -> Document*; // For multi-match
void destroy();
private:
std::unordered_multimap entries_;
size_t memory_limit_;
};
```
**Spill Strategy (Grace Hash Join)**:
When memory is exceeded:
1. Partition data by hash(key) into N partitions
2. Spill largest partition to disk
3. Continue with remaining partitions in memory
4. Process spilled partitions recursively
### 13.9.3 Aggregation Table
The aggregation table implements GROUP BY with multiple aggregation functions:
```cpp
class AggregationTable {
public:
void configure(const std::vector& functions);
auto get_or_create(const VMValue& key)
-> std::vector>*;
void iter_init();
auto iter_next() -> std::pair*>;
auto iter_has_next() const -> bool;
private:
std::unordered_map groups_;
std::vector