schema_version: 0.1.0 kind: source id: source.nexuscrm.lead_s3_snapshot name: nexuscrm Lead S3 snapshot export version: 0.1.0 owner: type: team name: data-platform domain: name: crm purpose: Source-faithful per-snapshot Parquet export of nexuscrm.Lead from TiDB Cloud Essential to S3, consumed by the lakehouse load stage. usage: intended: - Bulk load into the raw lakehouse layer - Audit and provenance of exported snapshots prohibited: - Direct analytical querying (load into a contracted product first). - Treating a single snapshot as an intra-day event stream. source: kind: database system: tidbcloud description: TiDB Cloud Essential nexuscrm.Lead identifiers: database: nexuscrm table: Lead cluster_id: '10996361222721502359' assets: - id: lead_s3_snapshot kind: file ref: s3://pingcap-datalake-nexuscrm-export-247906201378-us-west-2/tidb-essential/snapshots/source=nexuscrm/ materialization: external produced_by: kind: python ref: ingestion.extract.tidbcloud.engine.export schema: kind: relational columns: - name: id type: varchar(255) nullable: false - name: ownerid type: varchar(255) nullable: true - name: createdbyid type: varchar(255) nullable: true - name: lastmodifiedbyid type: varchar(255) nullable: true - name: createddate type: datetime nullable: false - name: lastmodifieddate type: datetime nullable: false - name: systemmodstamp type: datetime nullable: false - name: isdeleted type: tinyint(1) nullable: false - name: salutation type: varchar(255) nullable: true - name: firstname type: varchar(255) nullable: true - name: lastname type: varchar(255) nullable: false - name: company type: varchar(255) nullable: false - name: title type: varchar(255) nullable: true - name: leadsource type: varchar(255) nullable: true - name: status type: varchar(255) nullable: false - name: phone type: varchar(50) nullable: true - name: mobilephone type: varchar(50) nullable: true - name: email type: varchar(255) nullable: true - name: fax type: varchar(50) nullable: true - name: hasoptedoutofemail type: tinyint(1) nullable: true - name: donotcall type: tinyint(1) nullable: true - name: emailbounceddate type: datetime nullable: true - name: emailbouncedreason type: varchar(255) nullable: true - name: companydunsnumber type: varchar(255) nullable: true - name: website type: varchar(255) nullable: true - name: industry type: varchar(255) nullable: true - name: street type: text nullable: true - name: city type: varchar(255) nullable: true - name: state type: varchar(255) nullable: true - name: postalcode type: varchar(255) nullable: true - name: country type: varchar(255) nullable: true - name: statecode type: varchar(255) nullable: true - name: countrycode type: varchar(255) nullable: true - name: latitude type: decimal(65,30) nullable: true - name: longitude type: decimal(65,30) nullable: true - name: geocodeaccuracy type: varchar(255) nullable: true - name: annualrevenue type: decimal(65,30) nullable: true - name: numberofemployees type: decimal(65,30) nullable: true - name: rating type: varchar(255) nullable: true - name: description type: text nullable: true - name: jigsaw type: varchar(255) nullable: true - name: jigsawcontactid type: varchar(255) nullable: true - name: isconverted type: tinyint(1) nullable: true - name: converteddate type: date nullable: true - name: convertedaccountid type: varchar(255) nullable: true - name: convertedcontactid type: varchar(255) nullable: true - name: convertedopportunityid type: varchar(255) nullable: true - name: name type: varchar(255) nullable: true - name: middlename type: varchar(255) nullable: true - name: suffix type: varchar(255) nullable: true - name: aemeetingcompleteddwelltime__c type: decimal(65,30) nullable: true - name: aemeetingcompletedtime__c type: datetime nullable: true - name: ae_feedback_notes__c type: longtext nullable: true - name: ae_feedback_score__c type: varchar(255) nullable: true - name: ae__c type: varchar(255) nullable: true - name: ai_processed__c type: tinyint(1) nullable: true - name: ai_vertical__c type: tinyint(1) nullable: true - name: aws_campaign_name__c type: varchar(255) nullable: true - name: aws_lead_stage__c type: varchar(255) nullable: true - name: aws_project_description__c type: text nullable: true - name: aws_status__c type: varchar(255) nullable: true - name: accept_service__c type: tinyint(1) nullable: true - name: account_record_type__c type: varchar(255) nullable: true - name: account_status__c type: varchar(255) nullable: true - name: age_days_since_converted__c type: decimal(65,30) nullable: true - name: alicloud_channel__c type: tinyint(1) nullable: true - name: annualrevenue__c type: varchar(255) nullable: true - name: approval_poc_number__c type: decimal(65,30) nullable: true - name: approval_reason__c type: text nullable: true - name: approval_status__c type: varchar(255) nullable: true - name: approvalstatus__c type: varchar(255) nullable: true - name: associateaccountemail__c type: tinyint(1) nullable: true - name: associateaccount__c type: varchar(255) nullable: true - name: associateleademail__c type: tinyint(1) nullable: true - name: associatelead__c type: varchar(255) nullable: true - name: associate_lead_company__c type: varchar(255) nullable: true - name: authority_notes__c type: text nullable: true - name: authority__c type: varchar(255) nullable: true - name: auto_converted__c type: tinyint(1) nullable: true - name: bn_account__c type: varchar(255) nullable: true - name: bn_email_domain__c type: varchar(255) nullable: true - name: billing_email__c type: varchar(255) nullable: true - name: brief_description_of_end_user_s_business__c type: varchar(255) nullable: true - name: budget_notes__c type: text nullable: true - name: budget__c type: varchar(255) nullable: true - name: cldr__active_leads__c type: decimal(65,30) nullable: true - name: campaignid type: varchar(255) nullable: true - name: cloudservices__c type: json nullable: true - name: cloudd_vendor__c type: varchar(255) nullable: true - name: comments__c type: text nullable: true - name: connection_type__c type: varchar(255) nullable: true - name: convertaccount__c type: varchar(255) nullable: true - name: convertcontact__c type: varchar(255) nullable: true - name: convertopportunity__c type: varchar(255) nullable: true - name: converted_from_leads__c type: tinyint(1) nullable: true - name: country_rule_use__c type: varchar(255) nullable: true - name: country_for_aws__c type: varchar(255) nullable: true - name: currect_mql_owner__c type: varchar(255) nullable: true - name: currencyisocode type: varchar(255) nullable: true - name: customer_stage_datev2__c type: datetime nullable: true - name: customer_stage_source_details__c type: varchar(255) nullable: true - name: customer_stage_source__c type: varchar(255) nullable: true - name: dozisf__zoominfo_company_id__c type: varchar(255) nullable: true - name: dozisf__zoominfo_enrich_status__c type: varchar(255) nullable: true - name: dozisf__zoominfo_first_updated__c type: datetime nullable: true - name: dozisf__zoominfo_id__c type: varchar(255) nullable: true - name: dozisf__zoominfo_inboxai_id__c type: varchar(255) nullable: true - name: dozisf__zoominfo_last_updated__c type: datetime nullable: true - name: dozisf__zoominfo_non_matched_reason__c type: varchar(255) nullable: true - name: date_status_changed__c type: datetime nullable: true - name: deal_registration__c type: varchar(255) nullable: true - name: department__c type: varchar(255) nullable: true - name: developer_segment_name_ori__c type: text nullable: true - name: developer_segment_name__c type: json nullable: true - name: developer_summary__c type: longtext nullable: true - name: developer_tags__c type: longtext nullable: true - name: disqualifiedrasonisnulldwelltime__c type: decimal(65,30) nullable: true - name: disqualifiedrasonisnulltime__c type: datetime nullable: true - name: disqualified_date__c type: datetime nullable: true - name: disqualified_reason__c type: varchar(255) nullable: true - name: disqualified_stage_datev2__c type: datetime nullable: true - name: disqualified_stage_source_details__c type: varchar(255) nullable: true - name: disqualified_stage_source__c type: varchar(255) nullable: true - name: domestic_hq_state_province__c type: varchar(255) nullable: true - name: duration__c type: decimal(65,30) nullable: true - name: employee_numbers__c type: varchar(255) nullable: true - name: employee_range__c type: text nullable: true - name: estimated_opportunity_close_date__c type: date nullable: true - name: estimated_product_usage_scenario__c type: varchar(255) nullable: true - name: estimated_quantity__c type: varchar(255) nullable: true - name: existing_database__c type: json nullable: true - name: first_campaign__c type: varchar(255) nullable: true - name: first_conversion__c type: varchar(255) nullable: true - name: gclid__c type: varchar(255) nullable: true - name: googleformid__c type: varchar(255) nullable: true - name: hq_city__c type: varchar(255) nullable: true - name: hq_country__c type: varchar(255) nullable: true - name: hq_phone__c type: varchar(50) nullable: true - name: hq_state_province__c type: varchar(255) nullable: true - name: hq_street_address__c type: text nullable: true - name: hq_zip_postal_code__c type: text nullable: true - name: heavy_user_type__c type: varchar(255) nullable: true - name: heavy_user__c type: tinyint(1) nullable: true - name: hubspot_hubsoriginal_source_drill_down_1__c type: varchar(255) nullable: true - name: hubspot_original_source_drill_down_1__c type: varchar(255) nullable: true - name: hubspot_original_source__c type: varchar(255) nullable: true - name: hubspot_score__c type: decimal(65,30) nullable: true - name: ip_address__c type: varchar(255) nullable: true - name: ip_city__c type: varchar(255) nullable: true - name: ip_country__c type: varchar(255) nullable: true - name: iqmnoteisnulldwelltime__c type: decimal(65,30) nullable: true - name: iqmnoteisnulltime__c type: datetime nullable: true - name: iqm_completed_date__c type: date nullable: true - name: iqm_completed__c type: tinyint(1) nullable: true - name: iqm_completion__c type: tinyint(1) nullable: true - name: iqm_date__c type: date nullable: true - name: iqm_note_na_region_use_only__c type: text nullable: true - name: iqm_quality_score__c type: decimal(65,30) nullable: true - name: iqm_stage_datev2__c type: datetime nullable: true - name: iqm__c type: tinyint(1) nullable: true - name: individualid type: varchar(255) nullable: true - name: industry_legacy__c type: varchar(255) nullable: true - name: industry_for_aws__c type: varchar(255) nullable: true - name: initial_ai_scoring__c type: decimal(65,30) nullable: true - name: initial_activity__c type: datetime nullable: true - name: interest_level__c type: varchar(255) nullable: true - name: internal_user_bg__c type: varchar(255) nullable: true - name: internal_user__c type: tinyint(1) nullable: true - name: isflowchangeowner__c type: tinyint(1) nullable: true - name: job_function__c type: varchar(255) nullable: true - name: job_role__c type: varchar(255) nullable: true - name: key_messages__c type: longtext nullable: true - name: last_activity_date__c type: varchar(255) nullable: true - name: last_campaign__c type: varchar(255) nullable: true - name: leadvalidation__c type: tinyint(1) nullable: true - name: lead_lifecycle_stage__c type: varchar(255) nullable: true - name: lead_link__c type: varchar(255) nullable: true - name: lead_owner_assignment__c type: varchar(255) nullable: true - name: lead_owner_name__c type: varchar(255) nullable: true - name: lead_pool__c type: varchar(255) nullable: true - name: lead_rule_country__c type: varchar(255) nullable: true - name: lead_set_owner_time__c type: datetime nullable: true - name: lead_source_details__c type: varchar(255) nullable: true - name: lead_stage_datev2__c type: datetime nullable: true - name: lead_stage_source_details__c type: varchar(255) nullable: true - name: lead_stage_source__c type: varchar(255) nullable: true - name: lead_status_formula_hidden__c type: varchar(255) nullable: true - name: lifecycle_stage__c type: varchar(255) nullable: true - name: linkedin_company_profile_url__c type: varchar(255) nullable: true - name: linkedin_profile__c type: text nullable: true - name: linkedin_url__c type: varchar(255) nullable: true - name: mql_aging_14_days__c type: decimal(65,30) nullable: true - name: mql_aging_30_days__c type: decimal(65,30) nullable: true - name: mql_aging_3_days__c type: decimal(65,30) nullable: true - name: mql_aging__c type: decimal(65,30) nullable: true - name: mql_date__c type: date nullable: true - name: mql_priority__c type: varchar(255) nullable: true - name: mql_reason__c type: varchar(255) nullable: true - name: mql_sla_violation_date__c type: datetime nullable: true - name: mql_sla_violation__c type: tinyint(1) nullable: true - name: mql_stage_datev2__c type: datetime nullable: true - name: mql_stage_source_details__c type: varchar(255) nullable: true - name: mql_stage_source__c type: varchar(255) nullable: true - name: mql__c type: decimal(65,30) nullable: true - name: meeting_completed_datetime__c type: datetime nullable: true - name: most_recent_referrer_url__c type: text nullable: true - name: most_recent_working_status_date__c type: datetime nullable: true - name: mysql_family_capture__c type: varchar(255) nullable: true - name: need_create_opportunity__c type: tinyint(1) nullable: true - name: need_notes__c type: text nullable: true - name: next_steps_notes__c type: text nullable: true - name: next_steps__c type: varchar(255) nullable: true - name: no_of_times_mql__c type: decimal(65,30) nullable: true - name: nurturerasonisnulldwelltime__c type: decimal(65,30) nullable: true - name: nurturerasonisnulltime__c type: datetime nullable: true - name: nurturesalesdwelltime__c type: decimal(65,30) nullable: true - name: nurturesalestime__c type: datetime nullable: true - name: nurture_marketing_date__c type: datetime nullable: true - name: nurture_reason_other__c type: varchar(255) nullable: true - name: nurture_reason__c type: varchar(255) nullable: true - name: nurture_sales_date__c type: datetime nullable: true - name: opportunity_name_new__c type: varchar(255) nullable: true - name: opportunity_name__c type: varchar(255) nullable: true - name: opportunity_number__c type: varchar(255) nullable: true - name: opportunity_record_type__c type: varchar(255) nullable: true - name: opportunity_stage_datev2__c type: datetime nullable: true - name: opportunity_title__c type: varchar(255) nullable: true - name: opportunity_type_lead__c type: varchar(255) nullable: true - name: original_referrer_url__c type: text nullable: true - name: original_utm_campaign__c type: varchar(255) nullable: true - name: original_utm_content__c type: varchar(255) nullable: true - name: original_utm_medium__c type: varchar(255) nullable: true - name: original_utm_source__c type: varchar(255) nullable: true - name: original_utm_term__c type: varchar(255) nullable: true - name: other_emails__c type: varchar(255) nullable: true - name: owner_id__c type: varchar(255) nullable: true - name: owner_isactive__c type: tinyint(1) nullable: true - name: owner_profile__c type: varchar(255) nullable: true - name: plg_bd_pool__c type: tinyint(1) nullable: true - name: product_type__c type: varchar(255) nullable: true - name: parnter_status__c type: varchar(255) nullable: true - name: partner_regional_ops__c type: varchar(255) nullable: true - name: partner_sales_email__c type: varchar(255) nullable: true - name: partner_sales_owner__c type: varchar(255) nullable: true - name: person_lifecycle_stage__c type: varchar(255) nullable: true - name: person_owner_profile__c type: varchar(255) nullable: true - name: person_source_detail__c type: varchar(255) nullable: true - name: person_source__c type: varchar(255) nullable: true - name: person_status__c type: varchar(255) nullable: true - name: persona__c type: varchar(255) nullable: true - name: personal_email__c type: tinyint(1) nullable: true - name: phone_mobile__c type: varchar(255) nullable: true - name: portal_id__c type: varchar(255) nullable: true - name: post_iqm_stage_datev2__c type: datetime nullable: true - name: primary_need_pain__c type: varchar(255) nullable: true - name: priority__c type: varchar(255) nullable: true - name: product__c type: varchar(255) nullable: true - name: recent_source_detail__c type: varchar(255) nullable: true - name: recent_source__c type: varchar(255) nullable: true - name: recent_utm_campaign__c type: varchar(255) nullable: true - name: recent_utm_content__c type: varchar(255) nullable: true - name: recent_utm_medium__c type: varchar(255) nullable: true - name: recent_utm_source__c type: varchar(255) nullable: true - name: recent_utm_term__c type: varchar(255) nullable: true - name: recent_working_source_details__c type: varchar(255) nullable: true - name: recent_working_source__c type: varchar(255) nullable: true - name: referral_is_valid_until__c type: date nullable: true - name: referral_valid_period_is_extended__c type: tinyint(1) nullable: true - name: region_rule_use__c type: varchar(255) nullable: true - name: region__c type: varchar(255) nullable: true - name: registrant_address__c type: varchar(255) nullable: true - name: registrant_company_name__c type: varchar(255) nullable: true - name: registrant_email__c type: varchar(255) nullable: true - name: registrant_name__c type: varchar(255) nullable: true - name: registrant_phone__c type: varchar(255) nullable: true - name: registration_status__c type: varchar(255) nullable: true - name: reject_comment__c type: text nullable: true - name: replacing_database__c type: json nullable: true - name: sal_stage_source_details__c type: varchar(255) nullable: true - name: sal_stage_source__c type: varchar(255) nullable: true - name: sdr_for_na_only__c type: varchar(255) nullable: true - name: sdr_note_na_emea_only__c type: text nullable: true - name: signup_source__c type: varchar(255) nullable: true - name: sql_date__c type: date nullable: true - name: sql_stage_source_details__c type: varchar(255) nullable: true - name: sql_stage_source__c type: varchar(255) nullable: true - name: sql__c type: decimal(65,30) nullable: true - name: secondary_email__c type: varchar(255) nullable: true - name: segmentation_for_na_emea_reference__c type: varchar(255) nullable: true - name: skip_assign_flow__c type: tinyint(1) nullable: true - name: state_for_aws__c type: varchar(255) nullable: true - name: state_from_hubspot__c type: varchar(255) nullable: true - name: submitted_by_partner__c type: tinyint(1) nullable: true - name: suspect_stage_datev2__c type: datetime nullable: true - name: suspect_stage_source_details__c type: varchar(255) nullable: true - name: suspect_stage_source__c type: varchar(255) nullable: true - name: tcp_na__c type: varchar(255) nullable: true - name: team_notes__c type: text nullable: true - name: team__c type: varchar(255) nullable: true - name: technical_fit_details__c type: varchar(255) nullable: true - name: tidb_cloud_user__c type: tinyint(1) nullable: true - name: tied_card_time__c type: datetime nullable: true - name: time_zone__c type: varchar(255) nullable: true - name: timing_notes__c type: text nullable: true - name: timing__c type: varchar(255) nullable: true - name: unify_created_at__c type: date nullable: true - name: unify_first_written_at__c type: date nullable: true - name: unify_initial_play_at__c type: date nullable: true - name: unify_initial_play__c type: varchar(255) nullable: true - name: unify_initial_reply_classification__c type: varchar(255) nullable: true - name: unify_initial_reply_tags__c type: varchar(255) nullable: true - name: unify_initial_sequence_at__c type: date nullable: true - name: unify_initial_sequence_step_at__c type: date nullable: true - name: unify_initial_sequence__c type: varchar(255) nullable: true - name: unify_metadata__c type: varchar(255) nullable: true - name: unify_most_recent_play_at__c type: date nullable: true - name: unify_most_recent_play__c type: varchar(255) nullable: true - name: unify_most_recent_reply_classification__c type: varchar(255) nullable: true - name: unify_most_recent_reply_tags__c type: varchar(255) nullable: true - name: unify_most_recent_sequence_at__c type: date nullable: true - name: unify_most_recent_sequence_step_at__c type: date nullable: true - name: unify_most_recent_sequence__c type: varchar(255) nullable: true - name: unify_updated_at__c type: date nullable: true - name: vertical__c type: varchar(255) nullable: true - name: website_referrer_url__c type: text nullable: true - name: x_team_na_only__c type: tinyint(1) nullable: true - name: zoominfo_company_id__c type: text nullable: true - name: zoominfo_contact_id__c type: varchar(255) nullable: true - name: affectlayer__affectlayer_notes__c type: varchar(255) nullable: true - name: agreement_time__c type: datetime nullable: true - name: awsapn__apn_crm_id__c type: varchar(255) nullable: true - name: awsapn__eligible_to_sync_with_apn__c type: tinyint(1) nullable: true - name: awsapn__last_apn_sync_date__c type: datetime nullable: true - name: awsapn__sync_with_partner_central__c type: tinyint(1) nullable: true - name: created_at__c type: datetime nullable: true - name: remaining_points__c type: decimal(65,30) nullable: true - name: source__c type: varchar(255) nullable: true - name: utm_campaign__c type: varchar(255) nullable: true - name: utm_content__c type: varchar(255) nullable: true - name: utm_medium__c type: varchar(255) nullable: true - name: utm_source__c type: varchar(255) nullable: true - name: utm_term__c type: varchar(255) nullable: true - name: zoom_app__iscreatedbyzoomapp__c type: tinyint(1) nullable: true - name: genderidentity type: varchar(255) nullable: true - name: lastactivitydate type: date nullable: true - name: pronouns type: varchar(255) nullable: true freshness: max_age: PT48H degrade_after: PT30H tests: - id: lead_source_smoke type: sql severity: informational description: Placeholder informational smoke check for the source contract of lead. Replace with concrete row-count, freshness, or key-uniqueness assertions. sql: select 1 governance: classifications: - customer_data - financial - operational_sensitive - pii - restricted access: default: restricted retention: period: P7D audit: required: true events: - export - agent_action agent_permissions: default: deny compatibility: change_policy: semver extensions: primary_key: - Id watermark_columns: - SystemModstamp - LastModifiedDate row_estimate: 375311 known_blind_spots: - Intra-snapshot changes are collapsed. - Insert-then-delete between snapshots is invisible. - Schema drift requires review before promotion.