Overview
High-volume operations and large data integrations face bottlenecks when automation engines attempt to load massive files all at once or when visibility over execution queues is limited. Version v88.4 of the Pipefy Integrations (iPaaS) engine delivers improvements focused on stability at scale, efficient handling of large files and workloads, faster troubleshooting, enhanced security, and a smoother flow-building experience.
What's new?
- Incremental CSV batch processing: CSV files can now be processed incrementally in batches of up to 10k rows, allowing large imports to run in parallel without loading the full file into memory.
- Efficient large file handling: File processing no longer requires loading the entire file into memory when reading or writing across Amazon S3, Google Drive, Dropbox, OneDrive, SharePoint, Azure, and SFTP.
- Execution queue visibility: Metrics showing queue wait time versus active execution time, alongside a snapshot of the current queue state.
- Temporary AWS credentials support: Connections using Amazon S3, AWS Bedrock, and AWS Secrets Manager can now use temporary AWS credentials instead of storing permanent access keys.
- Faster data mapping experience: Field searches from previous steps run significantly faster within the workflow builder.
- Expanded connectors and actions: New native apps available, including Microsoft SQL Server, RingCentral, QuickBooks Online, Microsoft Teams, Workday, PolyDoc, LetMePost, DataForB2B, LinkupAPI, and Sofya, alongside new actions across existing connectors.
- Smoother builder experience: Enhancements to long lists, formulas, keyboard shortcuts, tables, result downloads, HTML/image previews, connection validation, and missing-app warnings.
- Engine stability fixes: Structural fixes addressing stuck automations, duplicate scheduled runs, and generic memory error reporting.
How does this help?
- Delivers scale and reliability: Processing large files and CSVs in batches prevents worker memory exhaustion during heavy data runs.
- Speeds up incident troubleshooting: Identify queue wait times and exact execution behavior clearly without guesswork.
- Strengthens governance and security: Temporary AWS credentials eliminate the need to store long-lived access keys without breaking existing setups.
- Optimizes the builder workflow: Faster field mapping and UI shortcuts streamline daily automation setup and maintenance.
- Increases operational resilience: Fixes to scheduled triggers and automatic recovery mechanisms minimize operational disruptions.
Use cases
Nightly sales report import for Commercial Operations
The Commercial Operations team runs a daily import of an 80k-row CSV report to update order records. Using incremental batching of up to 10k rows in Pipefy Integrations, the automation reads and processes data in parallel directly into the Fulfillment Pipe without exceeding worker memory limits. The import finishes predictably without stalling.
Secure audit file exchange via AWS S3 in Finance
A Finance Analyst manages an integration flow in Integrations Hub that reads large bank statement files from an Amazon S3 bucket to generate entries in an Accounts Payable Pipe. By utilizing temporary AWS credentials and memory-efficient file reading, file transfer complies with corporate governance guidelines without exposing permanent access keys in logs.
Service desk triage with Microsoft Teams and SQL Server
The IT team manages critical tickets using a Service Desk Pipe connected to Microsoft SQL Server for asset lookups and Microsoft Teams for notifications. During high-volume request periods, the Pipe Admin checks execution queue metrics to track wait time before alerts trigger, ensuring fast incident response SLAs.
Most relevant for
- Pipe Admins
- Builders
- IT Leaders
- Operations Teams
- Integration Engineers
Plans and availability
Version v88.4 is available across upgraded Multi-Tenant (MT) environments and is rolling out to Single-Tenant (ST) environments according to each environment's maintenance window.
