Docs/website documentation

website/docs/api/core-api.md

@@ -0,0 +1,122 @@
+# Core API Reference
+
+This page documents the core API methods and configurations available 
+in Ingest.
+
+## createIngest()
+
+The main function to create an Ingest instance.
+
+```typescript
+function createIngest(config: IngestConfig): IngestInstance;
+```
+
+### Configuration Options
+
+```typescript
+interface IngestConfig {
+  name: string;                    // Name of your ingest pipeline
+  entries?: Record<string, Entry>; // Data source entries
+  plugins?: Plugin[];             // Array of plugins to use
+  options?: {
+    concurrent?: boolean;         // Run entries concurrently
+    maxRetries?: number;         // Maximum retry attempts
+    retryDelay?: number;         // Delay between retries (ms)
+  };
+}
+```
+
+### Entry Configuration
+
+```typescript
+interface Entry {
+  fetch: () => Promise<any>;      // Required: Function to fetch data
+  transform?: (data: any) => any; // Optional: Transform fetched data
+  validate?: (data: any) => boolean; // Optional: Validate data
+  options?: {
+    timeout?: number;            // Timeout in milliseconds
+    retries?: number;           // Entry-specific retry count
+  };
+}
+```
+
+## Instance Methods
+
+### start()
+
+Starts the ingest pipeline.
+
+```typescript
+async function start(): Promise<void>;
+```
+
+### stop()
+
+Stops the ingest pipeline.
+
+```typescript
+async function stop(): Promise<void>;
+```
+
+### addEntry()
+
+Adds a new entry to the pipeline.
+
+```typescript
+function addEntry(name: string, entry: Entry): void;
+```
+
+### removeEntry()
+
+Removes an entry from the pipeline.
+
+```typescript
+function removeEntry(name: string): void;
+```
+
+## Built-in Fetchers
+
+### httpFetcher()
+
+Creates a fetcher for HTTP requests.
+
+```typescript
+function httpFetcher(config: {
+  url: string;
+  method?: 'GET' | 'POST' | 'PUT' | 'DELETE';
+  headers?: Record<string, string>;
+  body?: any;
+}): () => Promise<any>;
+```
+
+### fileFetcher()
+
+Creates a fetcher for file operations.
+
+```typescript
+function fileFetcher(config: {
+  path: string;
+  encoding?: string;
+}): () => Promise<any>;
+```
+
+## Error Handling
+
+### IngestError
+
+Base error class for Ingest-specific errors.
+
+```typescript
+class IngestError extends Error {
+  constructor(message: string, options?: {
+    code?: string;
+    cause?: Error;
+  });
+}
+```
+
+## Next Steps
+
+- Check out [Plugin API](plugins)
+- View [Examples](../examples/with-entries)
+- Learn about [Error Handling](../core-concepts/error-handling)

website/docs/contributing.md

@@ -0,0 +1,65 @@
+# Contributing to Ingest
+
+We love your input! We want to make contributing to Ingest 
+as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+- Becoming a maintainer
+
+## Development Process
+
+We use GitHub to host code, to track issues and feature requests, 
+as well as accept pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Issue that pull request!
+
+## Pull Request Process
+
+1. Update the README.md with details of changes to the interface, if applicable.
+2. Update the docs with any new features or changes.
+3. The PR will be merged once you have the sign-off of two other developers.
+
+## Any Contributions You Make Will Be Under the MIT Software License
+
+In short, when you submit code changes, your submissions are understood 
+to be under the same [MIT License](http://choosealicense.com/licenses/mit/) 
+that covers the project. 
+Feel free to contact the maintainers if that's a concern.
+
+## Report Bugs Using GitHub's [Issue Tracker](https://github.com/your-org/ingest/issues)
+
+We use GitHub issues to track public bugs. 
+Report a bug by [opening a new issue]
+(https://github.com/your-org/ingest/issues/new); 
+it's that easy!
+
+## Write Bug Reports with Detail, Background, and Sample Code
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+  - Be specific!
+  - Give sample code if you can.
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, 
+or stuff you tried that didn't work)
+
+## Use a Consistent Coding Style
+
+* 2 spaces for indentation rather than tabs
+* You can try running `npm run lint` for style unification
+
+## License
+
+By contributing, 
+you agree that your contributions will be licensed under its MIT License.

website/docs/core-concepts/overview.md

@@ -0,0 +1,77 @@
+# Core Concepts
+
+Understanding the core concepts of Ingest will help you build better 
+data pipelines. Here's an overview of the key components and concepts.
+
+## Ingest Instance
+
+The ingest instance is the main entry point of your data pipeline. 
+It's created using the `createIngest` function and manages all aspects 
+of your data ingestion process.
+
+```javascript
+const ingest = createIngest({
+  name: 'my-pipeline',
+  // ... configuration options
+});
+```
+
+## Entries
+
+Entries are the building blocks of your data pipeline. Each entry 
+represents a data source and defines how to:
+- Fetch the data
+- Transform the data (optional)
+- Validate the data (optional)
+- Handle errors
+
+```javascript
+{
+  entries: {
+    myEntry: {
+      fetch: async () => { /* ... */ },
+      transform: (data) => { /* ... */ },
+      validate: (data) => { /* ... */ }
+    }
+  }
+}
+```
+
+## Fetchers
+
+Fetchers are responsible for retrieving data from various sources. 
+Ingest provides several built-in fetchers:
+- `httpFetcher`: For HTTP/HTTPS requests
+- `fileFetcher`: For reading files
+- Custom fetchers: Create your own fetcher for specific needs
+
+## Transformers
+
+Transformers modify the fetched data before it's passed to the next stage.
+They can:
+- Clean data
+- Format data
+- Enrich data with additional information
+- Filter unwanted data
+
+## Plugins
+
+Plugins extend Ingest's functionality. They can:
+- Add new features
+- Modify existing behavior
+- Add global transformers
+- Implement custom logging or monitoring
+
+## Error Handling
+
+Ingest provides robust error handling mechanisms:
+- Per-entry error handling
+- Global error handling
+- Retry mechanisms
+- Error reporting and logging
+
+## Next Steps
+
+- Learn about [Architecture](architecture)
+- Explore [Features](../features/entries)
+- Check out [API Reference](../api/core-api)

website/docs/examples/with-entries.md

@@ -0,0 +1,106 @@
+# Working with Entries
+
+This guide shows you how to work with entries in Ingest through practical examples.
+
+## Basic Entry
+
+Here's a basic entry that fetches data from an API:
+
+```javascript
+import { createIngest } from '@your-org/ingest';
+
+const ingest = createIngest({
+  entries: {
+    users: {
+      async fetch() {
+        const response = await fetch('https://api.example.com/users');
+        return response.json();
+      }
+    }
+  }
+});
+
+// Use the entry
+const users = await ingest.entries.users.fetch();
+```
+
+## Entry with Transform
+
+Transform the fetched data before using it:
+
+```javascript
+const ingest = createIngest({
+  entries: {
+    users: {
+      async fetch() {
+        const response = await fetch('https://api.example.com/users');
+        return response.json();
+      },
+      transform(users) {
+        return users.map(user => ({
+          id: user.id,
+          fullName: `${user.firstName} ${user.lastName}`,
+          email: user.email.toLowerCase()
+        }));
+      }
+    }
+  }
+});
+
+// Get transformed data
+const users = await ingest.entries.users.fetch();
+```
+
+## Entry with Validation
+
+Add validation to ensure data quality:
+
+```javascript
+const ingest = createIngest({
+  entries: {
+    users: {
+      async fetch() {
+        const response = await fetch('https://api.example.com/users');
+        return response.json();
+      },
+      validate(users) {
+        return users.every(user => 
+          user.id && 
+          typeof user.email === 'string' &&
+          user.email.includes('@')
+        );
+      }
+    }
+  }
+});
+```
+
+## Dependent Entries
+
+Create entries that depend on other entries:
+
+```javascript
+const ingest = createIngest({
+  entries: {
+    users: {
+      async fetch() {
+        const response = await fetch('https://api.example.com/users');
+        return response.json();
+      }
+    },
+    userPosts: {
+      async fetch(context) {
+        const users = await context.entries.users.fetch();
+        const posts = [];
+
+        for (const user of users) {
+          const response = await fetch(`https://api.example.com/users/${user.id}/posts`);
+          posts.push(...await response.json());
+        }
+
+        return posts;
+      }
+    }
+  }
+});
+```