> ## Documentation Index
> Fetch the complete documentation index at: https://docs.octavehq.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Load events from your data warehouse
> Backfill GTM history into Octave from S3, GCS, Azure Blob, Snowflake, BigQuery, Redshift or Databricks
The [generic events API](/generic-events/overview) assumes you already have an event in
hand — it is written for n8n, Clay, Make and app code that fires one activity at a time.
When your history lives in a warehouse or a bucket instead, the transport is the easy
part. The work is mapping: you have a `deals` table with forty columns named nothing like
Octave's fields, and you need to know that a row with `stage = 'Closed Won'` is a
`deal_won` event, that `owner` has to split into an email and a name, and that one deal
row is usually more than one event.
This section walks that mapping, then the pipeline around it.
## Where does your data live?
Parquet, CSV or JSONL in a bucket, usually partitioned by date.
Same file-drop shape, read through the GCS client.
Containers and blob prefixes, with or without a hierarchical namespace.
Query-based extract with a modified-at watermark.
Storage Read API, parameterised incremental queries.
Direct cursor for small pulls, UNLOAD for large backfills.
SQL warehouse queries or Delta change data feed.
## What kind of events do you have?
Octave takes four kinds of activity. Most warehouses have tables that map onto them
directly:
| Your table | Octave type | `eventType` values |
| - | - | - |
| Opportunities, deals, pipeline | `crm` | `opportunity_created`, `deal_won`, `deal_lost`, `meeting_booked` |
| Email log, sequence sends, replies | `email` | `sent`, `reply` |
| Call recordings, meeting transcripts | `call` | `transcript` |
| LinkedIn connections and messages | `social` | `connection_sent`, `connection_accepted`, `message_sent`, `message_received` |
Only `sent` and `reply` email events and `transcript` call events are processed for
analytics. All CRM and all social types are processed. Other email and call types are
accepted and stored, but produce no findings — sending them is not an error, it just
does not buy you anything.
## Backfill or continuous?
The answer decides which endpoint you POST to.
| | Endpoint | Why |
| - | - | - |
| Historical backfill | [`POST /api/v2/event/import`](/v2-api-reference/events/bulk-import-events) | Up to 1000 events per request (30 MB body, 1 MB per event, 30 requests a minute), ingested asynchronously, mixed types allowed in one call, with a job you can poll |
| Ongoing sync | [`POST /api/v2/analytics/webhook/receive/generic/{emails\|calls\|crm\|social}`](/v2-api-reference/events/log-event) | One event per request (10 MB body) — fine at trickle volume, far too slow for a backfill |
Run the backfill first and confirm `counts.ingested` before you wire anything to a
schedule. Once the history is in, the same mapping code drives the incremental run: point
it at the webhook endpoints if you are reacting to individual rows, or keep using bulk
import if you are running batches on a timer. Bulk import is the better default for
anything scheduled — a nightly job pushing 4000 rows is four requests instead of four
thousand.
## Where this runs
This is not something to run off a laptop. The pipeline needs to fire on a trigger or a
schedule, hold a watermark between runs, and keep your credentials somewhere managed — so
it belongs in a serverless function next to the data.
Each source page names a concrete runtime, and the prompt on that page tells the agent to
build and deploy for it:
| Source | Runtime | What fires it |
| - | - | - |
| [S3](/warehouse-events/s3) | AWS Lambda | S3 event notification on object create |
| [GCS](/warehouse-events/gcs) | Cloud Run function | Eventarc on object finalized |
| [Azure Blob](/warehouse-events/azure-blob) | Azure Functions | Blob trigger |
| [Snowflake](/warehouse-events/snowflake) | Lambda or Cloud Run job | EventBridge or Cloud Scheduler |
| [BigQuery](/warehouse-events/bigquery) | Cloud Run job | Cloud Scheduler |
| [Redshift](/warehouse-events/redshift) | AWS Lambda | EventBridge schedule |
| [Databricks](/warehouse-events/databricks) | Databricks Job | The job's own schedule |
Keep the initial backfill separate from the incremental run. A function sized for "a file
landed, process it" will not survive several years of history — Lambda stops at 15
minutes, and consumption-plan Azure Functions sooner. Each source page names the right
vehicle for the backfill, and the prompt asks the agent to plan for both.
## The shape of the pipeline
Whatever the source and whatever the runtime, the flow is the same:
1. **Trigger** — an object lands, or a schedule fires.
2. **Read** incrementally, from a watermark held in a managed store.
3. **Map** your columns onto the Octave event structures.
4. **Batch** to 1000 events or fewer.
5. **POST** to `/api/v2/event/import`, with the workspace key in an `api_key` header.
6. **Poll** the returned `jobOId` on [`GET /api/v2/event/import/status`](/v2-api-reference/events/get-bulk-import-status)
until the job reports `COMPLETED` — or pass a `callbackUrl` and let Octave POST the
final counts to you. Polling remains the source of truth; the callback is best-effort.
7. **Advance** the watermark, once the batch is confirmed ingested.
Only steps 1 and 2 differ between sources. Everything from the mapping onward is
identical, which is why the prompt on each source page is almost entirely shared — and why
the mapping below is the part worth your attention.
## Mapping reference
Every event carries a `type` discriminator on bulk import. The per-type webhook endpoints
take it from the URL path instead, so it is the one field that differs between the two
transports.
The field-by-field reference — every optional field, its description and an example — is
generated from the API's own schema, so it cannot drift from what the endpoint accepts:
[Bulk import events](/v2-api-reference/events/bulk-import-events) for the batch endpoint,
[Log event](/v2-api-reference/events/log-event) for the per-event one. This page covers
the mapping rules those references cannot: which rows become which events, and what to do
with the columns that do not map one-to-one.
Required fields, by type:
| Type | Required |
| - | - |
| `email` | `type`, `eventTimestamp`, `eventType`, `subject`, `body`, `from`, `to` |
| `call` | `type`, `eventTimestamp`, `eventType`, `title`, `transcript`, `participants` |
| `crm` | `type`, `eventTimestamp`, `eventType`, `opportunityId`, `opportunityName` |
| `social` | `type`, `eventTimestamp`, `eventType`, `body`, `from`, `to` |
A CRM event with the fields worth sourcing if your table has them:
```json theme={null}
{
"type": "crm",
"eventTimestamp": "2024-02-03T16:00:00.000Z",
"eventType": "opportunity_created",
"opportunityId": "opp_12345",
"opportunityName": "Acme Corp - Enterprise Plan",
"amount": "50000.00",
"currency": "USD",
"stage": "Qualification",
"stageCategory": "open",
"accountName": "Acme Corp",
"crmAccountId": "acct_67890",
"crmAccountDomain": "acme.com",
"contactEmail": "john@acme.com",
"contactEmails": ["john@acme.com", "priya@acme.com"],
"ownerEmail": "sarah@company.com",
"closeDate": "2024-03-31T00:00:00.000Z",
"crmLastModifiedAt": "2024-02-03T16:00:00.000Z",
"eventId": "deal_12345_created"
}
```
An email event. Calls and social touches follow the same shape — see
[the generic events reference](/generic-events/overview#call-events) for those:
```json theme={null}
{
"type": "email",
"eventTimestamp": "2024-02-03T10:30:00.000Z",
"eventType": "sent",
"subject": "Follow up on our conversation",
"body": { "text": "Hi John, just wanted to follow up..." },
"from": { "email": "sarah@company.com", "name": "Sarah Johnson" },
"to": [{ "email": "john@acme.com", "name": "John Smith" }],
"crmOpportunityId": "opp_12345",
"eventId": "email_98765"
}
```
### Conversion rules
The problems that actually come up when the input is a warehouse table rather than a
webhook payload:
| Your column looks like | Octave field | What to do |
| - | - | - |
| Epoch seconds, epoch millis, or an ISO 8601 string | `eventTimestamp` | All three are accepted as they are. A number is read as milliseconds when it is `>= 1e12`, otherwise as seconds. |
| `9999-12-31`, `1900-01-01`, or anything wildly out of range | `eventTimestamp` | Rejected. The timestamp must resolve within 50 years past and 5 years future. Drop those rows rather than sending them. |
| `owner_email` and `owner_name` in separate columns | `from`, `ownerEmail` | Combine into `{ "email": ..., "name": ... }`. Only `email` is required. |
| `recipients` as `"a@x.com,b@x.com"` | `to` | Split into an array of objects. At least one entry is required. |
| `amount` as `DECIMAL(18,2)` or `NUMBER` | `amount` | Send it as a **string**. A float will not preserve your precision exactly. |
| `stage` as free text, e.g. `"Closed Won"` | `eventType`, `stageCategory` | Derive `eventType` from it, and set `stageCategory` to `open`, `closed_won` or `closed_lost` explicitly (`other` for stages that fit none). Left unset, the bucket is inferred from `eventType`, so a deal deep in procurement still reads as simply open. |
| `transcript` as one long string | `transcript` | Prefer an array of `{ speaker, text }` turns. A flat string has to be diarized with an LLM before anything can be attributed to a speaker. |
| No explicit internal/external flag on a speaker | `speaker.role` | Decide by email domain — your own domain is `internal`, everything else is `external`. |
| A `NULL` | any | Omit the key. Do not send an explicit `null`. |
| The row's primary key | `eventId` | Set it, up to 255 characters. This is what makes a re-run idempotent. |
| Rows that originally came from Salesforce or HubSpot | `sourceProviderName` | Sets the logo shown against the activity in the feed, instead of the generic Octave icon. |
### One row can be more than one event
This is the rule most mappings miss. A `deals` row with both `created_at` and `closed_at`
populated is two events, not one: an `opportunity_created` at the created timestamp, and a
`deal_won` or `deal_lost` at the closed timestamp. Each needs its own `eventTimestamp` and
its own `eventId` — suffix the row's primary key so they stay distinct.
Collapse them into a single event and the deal appears in Octave with no history. The
milestones Octave reports on are recorded per event, so a deal that only ever arrives as
`deal_won` has no creation date to measure a sales cycle against.
### Activity that is not about a deal
`crmOpportunityId` is optional on email, call and social events, and so is
`crmAccountId`. A support escalation, a quarterly check-in or a renewal conversation that
has an account but no open opportunity is still worth sending: set `crmAccountId` or
`crmAccountDomain` so it attaches at the account level, and leave `crmOpportunityId` off.
Octave still tries to place it against a deal by participants and domain inside a time
window, and an event that matches nothing is kept and analysed — it just is not counted
toward any pipeline.
Where you do know the deal, say so. An explicit `crmOpportunityId` that matches a deal in
the workspace — however that deal arrived, including through the Salesforce or HubSpot
connector — skips the inference entirely.
### Keeping deals current
Send `crmLastModifiedAt` on every CRM event. A deal is one object that changes over time,
so the same `opportunityId` comes back repeatedly as the amount moves and the stage
advances. Without a last-modified timestamp an updated deal can look identical to the
previous push and be discarded as a duplicate. See
[Keeping deals up to date](/generic-events/overview#crm-events) for the full explanation.
## Troubleshooting
**You get a 400 and nothing at all imports.** Validation is all-or-nothing per request —
one bad row rejects the entire batch, not just itself. The response message lists the
offending zero-based indices, so map them back to your batch and fix those rows.
**401 `Invalid API key`.** The key is wrong, revoked or belongs to a different
workspace. Send it as an `api_key` header — the query-string form only works on the
per-event webhook endpoints, not on bulk import.
**429.** The bulk endpoint allows 30 requests a minute per key. Back off and keep only a
few requests in flight; 1000 events per request means 30 a minute is 30,000 events a
minute, which is rarely the bottleneck.
**`counts.skipped` is high.** Skipped is not failed. Duplicates and event types that
aren't processed for analytics both land there. `counts.ingested` is the authoritative
success count, and `counts.failed` is the one to alert on.
**`counts.failed` is non-zero.** The status response carries an `errors` sample — up to 20
rows, failed first — with each event's dedupe ID and the reason. That is usually enough to
find the shape of row that is breaking; fix the mapping and re-send those rows with the
same `eventId`.
**Old rows vanish without an error.** Check the timestamp window. Anything more than 50
years in the past or 5 years in the future is rejected, which catches sentinel dates and
any column where a null got coerced to epoch zero.
**Deals import but every pipeline chart is flat.** You are almost certainly emitting one
event per row instead of one per milestone. See
[One row can be more than one event](#one-row-can-be-more-than-one-event).
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.