Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,8 @@
- [pass](super-sql/operators/pass.md)
- [put](super-sql/operators/put.md)
- [rename](super-sql/operators/rename.md)
- [shapes](super-sql/operators/shapes.md)
- [search](super-sql/operators/search.md)
- [shapes](super-sql/operators/shapes.md)
- [skip](super-sql/operators/skip.md)
- [sort](super-sql/operators/sort.md)
- [switch](super-sql/operators/switch.md)
Expand All @@ -115,6 +115,12 @@
- [JOIN](super-sql/sql/join.md)
- [Set Operators](super-sql/sql/set-ops.md)
- [Functions](super-sql/functions/intro.md)
- [Errors](super-sql/functions/errors/intro.md)
- [error](super-sql/functions/errors/error.md)
- [has_error](super-sql/functions/errors/has_error.md)
- [is_error](super-sql/functions/errors/is_error.md)
- [missing](super-sql/functions/errors/missing.md)
- [quiet](super-sql/functions/errors/quiet.md)
- [Generics](super-sql/functions/generics/intro.md)
- [coalesce](super-sql/functions/generics/coalesce.md)
- [compare](super-sql/functions/generics/compare.md)
Expand All @@ -123,12 +129,6 @@
- [map](super-sql/functions/generics/map.md)
- [nullif](super-sql/functions/generics/nullif.md)
- [under](super-sql/functions/generics/under.md)
- [Errors](super-sql/functions/errors/intro.md)
- [error](super-sql/functions/errors/error.md)
- [has_error](super-sql/functions/errors/has_error.md)
- [is_error](super-sql/functions/errors/is_error.md)
- [missing](super-sql/functions/errors/missing.md)
- [quiet](super-sql/functions/errors/quiet.md)
- [Math](super-sql/functions/math/intro.md)
- [abs](super-sql/functions/math/abs.md)
- [ceil](super-sql/functions/math/ceil.md)
Expand Down Expand Up @@ -174,8 +174,8 @@
- [is](super-sql/functions/types/is.md)
- [kind](super-sql/functions/types/kind.md)
- [nameof](super-sql/functions/types/nameof.md)
- [typeof](super-sql/functions/types/typeof.md)
- [typename](super-sql/functions/types/typename.md)
- [typeof](super-sql/functions/types/typeof.md)
- [Aggregate Functions](super-sql/aggregates/intro.md)
- [and](super-sql/aggregates/and.md)
- [any](super-sql/aggregates/any.md)
Expand Down
46 changes: 24 additions & 22 deletions book/src/database/api.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
## SuperDB API
# SuperDB API

### _Status_
## Status

>[!NOTE]
> This is a brief sketch of the functionality exposed in the
> SuperDB API. More detailed documentation of the API will be forthcoming.

### Endpoints
## Endpoints

#### Pools
### Pools

##### Create pool
#### Create pool

Create a new lake pool.

Expand Down Expand Up @@ -69,7 +69,7 @@ curl -X POST \

---

##### Rename pool
#### Rename pool

Change a pool's name.

Expand Down Expand Up @@ -98,7 +98,7 @@ On success, HTTP 204 is returned with no response payload.

---

##### Delete pool
#### Delete pool

Permanently delete a pool.

Expand All @@ -123,7 +123,7 @@ On success, HTTP 204 is returned with no response payload.

---

##### Vacuum pool
#### Vacuum pool

Free storage space by permanently removing underlying data objects that have
previously been subject to a delete operation.
Expand Down Expand Up @@ -156,9 +156,9 @@ curl -X POST \

---

#### Branches
### Branches

##### Load Data
#### Load Data

Add data to a pool and return a reference commit ID.

Expand Down Expand Up @@ -197,7 +197,7 @@ curl -X POST \

---

##### Get Branch
#### Get Branch

Get information about a branch.

Expand Down Expand Up @@ -229,7 +229,7 @@ curl -X GET \

---

##### Delete Branch
#### Delete Branch

Delete a branch.

Expand All @@ -255,7 +255,7 @@ On success, HTTP 204 is returned with no response payload.

---

##### Delete Data
#### Delete Data

Create a commit that reflects the deletion of some data in the branch. The data
to delete can be specified via a list of object IDs or
Expand Down Expand Up @@ -316,7 +316,7 @@ curl -X POST \

---

##### Merge Branches
#### Merge Branches

Create a commit with the difference of the child branch added to the selected
branch.
Expand Down Expand Up @@ -350,7 +350,7 @@ curl -X POST \

---

##### Revert
#### Revert

Create a revert commit of the specified commit.

Expand Down Expand Up @@ -383,7 +383,9 @@ curl -X POST \

---

### Query
### Queries

#### Query

Execute a Zed query against data in a data lake.

Expand Down Expand Up @@ -437,7 +439,7 @@ curl -X POST \
{"type":"QueryStats","value":{"start_time":{"sec":1658193276,"ns":964207000},"update_time":{"sec":1658193276,"ns":964592000},"bytes_read":55,"bytes_matched":55,"records_read":3,"records_matched":3}}
```

##### Query Status
#### Query Status

Retrieve any runtime errors from a specific query. This endpoint only responds
after the query has exited and is only available for a limited time afterwards.
Expand Down Expand Up @@ -468,7 +470,7 @@ curl -X GET \

---

#### Events
### Events

Subscribe to an events feed, which returns an event stream in the format of
[server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html).
Expand Down Expand Up @@ -509,12 +511,12 @@ data: {"pool_id": "1sMDXpVwqxm36Rc2vfrmgizc3jz"}

---

### Media Types
## Media Types

For both request and response payloads, the service supports a variety of
formats.

#### Request Payloads
### Request Payloads

When sending request payloads, include the MIME type of the format in the
request's Content-Type header. If the Content-Type header is not specified, the
Expand All @@ -526,7 +528,7 @@ determine the type automatically. The
[input formats](../command/super.md#supported-formats) table describes which
formats may be successfully auto-detected.

#### Response Payloads
### Response Payloads

To receive successful (2xx) responses in a preferred format, include the MIME
type of the format in the request's Accept HTTP header. If the Accept header is
Expand All @@ -537,7 +539,7 @@ different default response format can be specified by invoking the
For non-2xx responses, the content type of the response will be
`application/json` or `text/plain`.

#### MIME Types
### MIME Types

The following table shows the supported MIME types and where they can be used.

Expand Down
22 changes: 11 additions & 11 deletions book/src/database/format.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
## Database Format
# Database Format

>[!NOTE]
>This document is a rough draft and work in progress. We plan to
>soon bring it up to date with the current implementation and maintain it
>as we add new capabilities to the system.

### Introduction
## Introduction

To support the client-facing SuperDB access
implemented by the [`super db` command](../command/db.md), we are developing
Expand Down Expand Up @@ -33,13 +33,13 @@ via a table virtualization layer built on top of the SuperDB model.
All data and metadata in a database conforms to the super-structured data model, which materially
simplifies development, test, introspection, and so forth.

### Cloud Object Model
## Cloud Object Model

Every data element in a database is either of two fundamental object types:
* a single-writer _immutable object_, or
* a multi-writer _transaction journal_.

#### Immutable Objects
### Immutable Objects

All imported data in a data pool is composed of immutable objects, which are organized
around a primary data object. Each data object is composed of one or more immutable objects
Expand All @@ -63,7 +63,7 @@ such an assumption).
> local, small-scale deployments for test/debug workflows. Thus, for simple use cases,
> the complexity of running an object-store service may be avoided.

##### Data Objects
#### Data Objects

A data object is created by a single writer using a globally unique name
with an embedded KSUID.
Expand Down Expand Up @@ -104,7 +104,7 @@ processing only a subset of data.
> and the in-memory CSUP representation supports random access so there is
> no need to have a seek index for the vector object.

##### Commit History
#### Commit History

A branch's commit history is the definitive record of the evolution of data in
that pool in a transactionally consistent fashion.
Expand Down Expand Up @@ -136,7 +136,7 @@ This log represents the definitive record of a branch's present
and historical content, and accessing its complete detail can provide
insights about data layout, provenance, history, and so forth.

#### Transaction Journal
### Transaction Journal

State that is mutable is built upon a transaction journal of immutable
collections of entries. In this way, there are no objects in the
Expand Down Expand Up @@ -179,7 +179,7 @@ alongside the journal entry. This way, the snapshot at HEAD may be
efficiently computed by locating the most recent cached snapshot and scanning
forward to HEAD.

##### Journal Concurrency Control
#### Journal Concurrency Control

To provide for atomic commits, a writer must be able to atomically update
the HEAD of the log. There are three strategies for doing so.
Expand Down Expand Up @@ -208,7 +208,7 @@ system and such round trips can be tens of milliseconds, another approach
is to simply run a lock service as part of a cloud deployment that manages
a mutex lock for each pool's journal.

##### Configuration State
#### Configuration State

Configuration state describing a lake or pool is also stored in mutable objects.
The database uses a commit journal to store configuration like the
Expand All @@ -217,7 +217,7 @@ Here, a generic interface to a commit journal manages any configuration
state simply as a key-value store of snapshots providing time travel over
the configuration history.

#### Merge on Read
### Merge on Read

To support _sorted scans_,
data objects are store in a sorted order defined by the pool's sort key.
Expand Down Expand Up @@ -245,7 +245,7 @@ can be produced by executing a sorted scan and rewriting the results back to the
in a new commit. In addition, the objects comprising the total order
do not overlap. This is just the basic LSM algorithm at work.

#### Object Naming
### Object Naming

```
<lake-path>/
Expand Down
6 changes: 3 additions & 3 deletions book/src/dev/integrations/zeek/logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ with [`super`](../../../command/super.md).
The following example shows a TSV [`conn.log`](https://docs.zeek.org/en/master/logs/conn.html) being read via `super` and
output as [Super (SUP)](../../../formats/sup.md).

#### conn.log
### conn.log

```mdtest-input conn.log
#separator \x09
Expand All @@ -28,13 +28,13 @@ output as [Super (SUP)](../../../formats/sup.md).
1521911721.255387 C8Tful1TvM3Zf5x8fl 10.164.94.120 39681 10.47.3.155 3389 tcp - 0.004266 97 19 RSTR - - 0 ShADTdtr 10 730 6 342 -
```

#### Example
### Example

```mdtest-command
super -S -c 'head 1' conn.log
```

#### Output
### Output
```mdtest-output
{
_path: "conn",
Expand Down
2 changes: 1 addition & 1 deletion book/src/dev/libraries/go.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Go
# Go

SuperDB is developed in Go so support for Go clients is
fairly comprehensive. Documentation of exported
Expand Down
2 changes: 1 addition & 1 deletion book/src/dev/libraries/intro.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Libraries
# Libraries

SuperDB currently supports a small number of languages
with client libraries for manipulating super-structured data and interacting
Expand Down
Loading