Ssis-776

| Scenario | Source Type | Volume (records) | Velocity (records/s) | |----------|-------------|------------------|----------------------| | Financial Transactions | SQL Server + Kafka | 150 M | 10 k | | IoT Telemetry | MQTT broker (JSON) | 300 M | 25 k | | Health‑Care Records | HL7 over REST | 80 M | 2 k | | E‑Commerce Click‑Streams | Azure Event Hubs | 200 M | 15 k |

All datasets contain a mixture of PII, PCI‑DSS, and non‑sensitive fields. SSIS-776

| Property | Recommended Value | Rationale | |----------|-------------------|-----------| | DefaultBufferMaxRows | 5,000 | Reduces rows per buffer, limits per‑row memory blow‑up | | DefaultBufferSize | 104,857,600 (100 MB) | Gives SSIS more breathing room while staying under the 2 GB limit | | EngineThreads | -1 (auto) or CPU Count | Lets the engine allocate optimal threads per core | | Scenario | Source Type | Volume (records)

Tip: You can set these globally in Project → Properties → Configuration → Buffer Settings, or per‑data‑flow in Advanced tab. Tip: You can set these globally in Project

| Dataset | Size | Partitioning | Original Run Time | SSIS‑776 Run Time | % Reduction | |---------|------|--------------|-------------------|-------------------|-------------| | FactSales (Daily) | 1.2 TB | 365 daily partitions | 18 min | 7 min | 61 % | | EventLog (Hourly) | 850 GB | 8 704 hourly partitions | 12 min | 4.8 min | 60 % | | DimCustomer (Monthly) | 120 GB | 48 monthly partitions | 1 min | 0.7 min | 30 % | | Staging_Orders (no partitioning) | 200 GB | – | 3 min | 2.9 min | 3 % (baseline) |

Observations