> ## Documentation Index
> Fetch the complete documentation index at: https://docs.octavehq.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Load events from your data warehouse
> Backfill GTM history into Octave from S3, GCS, Azure Blob, Snowflake, BigQuery, Redshift or Databricks
The [generic events API](/generic-events/overview) assumes you already have an event in
hand — it is written for n8n, Clay, Make and app code that fires one activity at a time.
When your history lives in a warehouse or a bucket instead, the transport is the easy
part. The work is mapping: you have a `deals` table with forty columns named nothing like
Octave's fields, and you need to know that a row with `stage = 'Closed Won'` is a
`deal_won` event, that `owner` has to split into an email and a name, and that one deal
row is usually more than one event.
This section walks that mapping, then the pipeline around it.
## Where does your data live?
Parquet, CSV or JSONL in a bucket, usually partitioned by date.
Same file-drop shape, read through the GCS client.
Containers and blob prefixes, with or without a hierarchical namespace.
Query-based extract with a modified-at watermark.
Storage Read API, parameterised incremental queries.
Direct cursor for small pulls, UNLOAD for large backfills.
SQL warehouse queries or Delta change data feed.
## What kind of events do you have?
Octave takes four kinds of activity. Most warehouses have tables that map onto them
directly:
| Your table | Octave type | `eventType` values |
| ------------------------------------ | ----------- | ---------------------------------------------------------------------------- |
| Opportunities, deals, pipeline | `crm` | `opportunity_created`, `deal_won`, `deal_lost`, `meeting_booked` |
| Email log, sequence sends, replies | `email` | `sent`, `reply` |
| Call recordings, meeting transcripts | `call` | `transcript` |
| LinkedIn connections and messages | `social` | `connection_sent`, `connection_accepted`, `message_sent`, `message_received` |
Only `sent` and `reply` email events and `transcript` call events are processed for
analytics. All CRM and all social types are processed. Other email and call types are
accepted and stored, but produce no findings — sending them is not an error, it just
does not buy you anything.
## Backfill or continuous?
The answer decides which endpoint you POST to.
| | Endpoint | Why |
| ------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Historical backfill | `POST /api/v2/event/import` | Up to 1000 events per request, ingested asynchronously, mixed types allowed in one call, with a job you can poll |
| Ongoing sync | `POST /api/v2/analytics/webhook/receive/generic/{emails\|calls\|crm\|social}` | One event per request — fine at trickle volume, far too slow for a backfill |
Run the backfill first and confirm `counts.ingested` before you wire anything to a
schedule. Once the history is in, the same mapping code drives the incremental run: point
it at the webhook endpoints if you are reacting to individual rows, or keep using bulk
import if you are running batches on a timer. Bulk import is the better default for
anything scheduled — a nightly job pushing 4000 rows is four requests instead of four
thousand.
## Where this runs
This is not something to run off a laptop. The pipeline needs to fire on a trigger or a
schedule, hold a watermark between runs, and keep your credentials somewhere managed — so
it belongs in a serverless function next to the data.
Each source page names a concrete runtime, and the prompt on that page tells the agent to
build and deploy for it:
| Source | Runtime | What fires it |
| ------------------------------------------ | ----------------------- | -------------------------------------- |
| [S3](/warehouse-events/s3) | AWS Lambda | S3 event notification on object create |
| [GCS](/warehouse-events/gcs) | Cloud Run function | Eventarc on object finalized |
| [Azure Blob](/warehouse-events/azure-blob) | Azure Functions | Blob trigger |
| [Snowflake](/warehouse-events/snowflake) | Lambda or Cloud Run job | EventBridge or Cloud Scheduler |
| [BigQuery](/warehouse-events/bigquery) | Cloud Run job | Cloud Scheduler |
| [Redshift](/warehouse-events/redshift) | AWS Lambda | EventBridge schedule |
| [Databricks](/warehouse-events/databricks) | Databricks Job | The job's own schedule |
Keep the initial backfill separate from the incremental run. A function sized for "a file
landed, process it" will not survive several years of history — Lambda stops at 15
minutes, and consumption-plan Azure Functions sooner. Each source page names the right
vehicle for the backfill, and the prompt asks the agent to plan for both.
## The shape of the pipeline
Whatever the source and whatever the runtime, the flow is the same:
1. **Trigger** — an object lands, or a schedule fires.
2. **Read** incrementally, from a watermark held in a managed store.
3. **Map** your columns onto the Octave event structures.
4. **Batch** to 1000 events or fewer.
5. **POST** to `/api/v2/event/import`.
6. **Poll** the returned `jobOId` until the job reports `COMPLETED`.
7. **Advance** the watermark, once the batch is confirmed ingested.
Only steps 1 and 2 differ between sources. Everything from the mapping onward is
identical, which is why the prompt on each source page is almost entirely shared — and why
the mapping below is the part worth your attention.
## Mapping reference
Every event carries a `type` discriminator on bulk import. The per-type webhook endpoints
take it from the URL path instead, so it is the one field that differs between the two
transports.
Required fields, by type:
| Type | Required |
| -------- | ---------------------------------------------------------------------------- |
| `email` | `type`, `eventTimestamp`, `eventType`, `subject`, `body`, `from`, `to` |
| `call` | `type`, `eventTimestamp`, `eventType`, `title`, `transcript`, `participants` |
| `crm` | `type`, `eventTimestamp`, `eventType`, `opportunityId`, `opportunityName` |
| `social` | `type`, `eventTimestamp`, `eventType`, `body`, `from`, `to` |
A CRM event with the fields worth sourcing if your table has them:
```json theme={null}
{
"type": "crm",
"eventTimestamp": "2024-02-03T16:00:00.000Z",
"eventType": "opportunity_created",
"opportunityId": "opp_12345",
"opportunityName": "Acme Corp - Enterprise Plan",
"amount": "50000.00",
"currency": "USD",
"stage": "Qualification",
"stageCategory": "open",
"accountName": "Acme Corp",
"crmAccountId": "acct_67890",
"crmAccountDomain": "acme.com",
"contactEmail": "john@acme.com",
"contactEmails": ["john@acme.com", "priya@acme.com"],
"ownerEmail": "sarah@company.com",
"closeDate": "2024-03-31T00:00:00.000Z",
"crmLastModifiedAt": "2024-02-03T16:00:00.000Z",
"eventId": "deal_12345_created"
}
```
An email event. Calls and social touches follow the same shape — see
[the generic events reference](/generic-events/overview#call-events) for those:
```json theme={null}
{
"type": "email",
"eventTimestamp": "2024-02-03T10:30:00.000Z",
"eventType": "sent",
"subject": "Follow up on our conversation",
"body": { "text": "Hi John, just wanted to follow up..." },
"from": { "email": "sarah@company.com", "name": "Sarah Johnson" },
"to": [{ "email": "john@acme.com", "name": "John Smith" }],
"crmOpportunityId": "opp_12345",
"eventId": "email_98765"
}
```
### Conversion rules
The problems that actually come up when the input is a warehouse table rather than a
webhook payload:
| Your column looks like | Octave field | What to do |
| ----------------------------------------------------------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Epoch seconds, epoch millis, or an ISO 8601 string | `eventTimestamp` | All three are accepted as they are. A number is read as milliseconds when it is `>= 1e12`, otherwise as seconds. |
| `9999-12-31`, `1900-01-01`, or anything wildly out of range | `eventTimestamp` | Rejected. The timestamp must resolve within 50 years past and 5 years future. Drop those rows rather than sending them. |
| `owner_email` and `owner_name` in separate columns | `from`, `ownerEmail` | Combine into `{ "email": ..., "name": ... }`. Only `email` is required. |
| `recipients` as `"a@x.com,b@x.com"` | `to` | Split into an array of objects. At least one entry is required. |
| `amount` as `DECIMAL(18,2)` or `NUMBER` | `amount` | Send it as a **string**. A float will not preserve your precision exactly. |
| `stage` as free text, e.g. `"Closed Won"` | `eventType`, `stageCategory` | Derive `eventType` from it, and set `stageCategory` to `open`, `won` or `lost` explicitly. Left unset, the bucket is inferred from `eventType`, so a deal deep in procurement still reads as simply open. |
| `transcript` as one long string | `transcript` | Prefer an array of `{ speaker, text }` turns. A flat string has to be diarized with an LLM before anything can be attributed to a speaker. |
| No explicit internal/external flag on a speaker | `speaker.role` | Decide by email domain — your own domain is `internal`, everything else is `external`. |
| A `NULL` | any | Omit the key. Do not send an explicit `null`. |
| The row's primary key | `eventId` | Set it, up to 255 characters. This is what makes a re-run idempotent. |
| Rows that originally came from Salesforce or HubSpot | `sourceProviderName` | Sets the logo shown against the activity in the feed, instead of the generic Octave icon. |
### One row can be more than one event
This is the rule most mappings miss. A `deals` row with both `created_at` and `closed_at`
populated is two events, not one: an `opportunity_created` at the created timestamp, and a
`deal_won` or `deal_lost` at the closed timestamp. Each needs its own `eventTimestamp` and
its own `eventId` — suffix the row's primary key so they stay distinct.
Collapse them into a single event and the deal appears in Octave with no history. The
milestones Octave reports on are recorded per event, so a deal that only ever arrives as
`deal_won` has no creation date to measure a sales cycle against.
### Keeping deals current
Send `crmLastModifiedAt` on every CRM event. A deal is one object that changes over time,
so the same `opportunityId` comes back repeatedly as the amount moves and the stage
advances. Without a last-modified timestamp an updated deal can look identical to the
previous push and be discarded as a duplicate. See
[Keeping deals up to date](/generic-events/overview#crm-events) for the full explanation.
## Troubleshooting
**You get a 400 and nothing at all imports.** Validation is all-or-nothing per request —
one bad row rejects the entire batch, not just itself. The response message lists the
offending zero-based indices, so map them back to your batch and fix those rows.
**An invalid API key comes back as 404, not 401.** A "workspace not found" response
usually means the key is wrong or belongs to a different workspace, not that the endpoint
path is wrong.
**`counts.skipped` is high.** Skipped is not failed. Duplicates and event types that
aren't processed for analytics both land there. `counts.ingested` is the authoritative
success count, and `counts.failed` is the one to alert on.
**Old rows vanish without an error.** Check the timestamp window. Anything more than 50
years in the past or 5 years in the future is rejected, which catches sentinel dates and
any column where a null got coerced to epoch zero.
**Deals import but every pipeline chart is flat.** You are almost certainly emitting one
event per row instead of one per milestone. See
[One row can be more than one event](#one-row-can-be-more-than-one-event).