Five Days to Validate a ClickHouse Migration at Scale 

Faced with a dramatic increase in data volume, a leading cybersecurity company needed to upgrade its vulnerability-scanning analytics platform. The existing PostgreSQL database could not support the anticipated write load for our client, prompting a move to ClickHouse. Working with Lineate, an official ClickHouse partner, the client validated its new architecture, uncovered critical issues in schema design and infrastructure provisioning, and developed a concrete roadmap for a successful launch and rapid scale-up. The engagement delivered schema fixes, query optimizations, capacity planning guidance and a phased growth strategy.

Our client is a US-based AI and quantum technology company whose platform helps enterprise clients identify and manage cryptographic vulnerabilities across their IT infrastructure. A core component of their platform indexes cryptographic inventory and powers real-time security dashboards, giving clients immediate visibility into the state of their cryptographic posture.

• Scaling pressure: As their data volumes grew, the team moved away from PostgreSQL — which was too slow for the volume and complexity of queries their dashboards required.

• Migration to ClickHouse: To address the performance gap, the team migrated their analytics layer from PostgreSQL to a single-node ClickHouse instance inside Kubernetes and redesigned the schema for analytical workloads.

• Experience gap: Because ClickHouse and OLAP systems were new to the team, they needed expert validation before deploying into a locked-down environment where changes would be difficult to implement after the fact.

Lineate is a certified ClickHouse partner with extensive experience in building and scaling data platforms. The client engaged Lineate to perform a rapid readiness assessment and provide architectural guidance. The collaboration included an initial one-hour workshop followed by daily iterations over five days, ensuring that high-priority issues were surfaced quickly and recommendations were grounded in the client's operational realities.

To validate the ClickHouse deployment, the client engaged Lineate, an official ClickHouse partner, for a five-day technical review. Lineate's certified engineers worked closely with the client's team, holding daily sync meetings to review database schema design, materialized views, query patterns and Kubernetes deployment settings. The performance goal was to maintain sub-five-second dashboard loads while supporting write volumes that would climb by three orders of magnitude.

During the assessment, Lineate dissected the system's architecture and uncovered several issues that would limit performance under heavy use:

• Data updates: The materialized-view design performed destructive partial updates; if the application updated only a specific field, the database replaced untouched fields with empty values, causing data loss.

• Data cleanup: The main data table lacked time-based partitions. Without partitions, routine cleanup operations would require rewriting massive portions of the table, slowing down ingestion and causing maintenance headaches.

• Dashboard loading: Dashboard filters such as severity and tags had no supporting indexes, so each page load forced a full table scan, threatening the five-second read-latency goal.

• Server resources: The initial Kubernetes deployment allocated only 2 CPU cores and 8 GB of RAM, far below the 32-core, 128-GB baseline needed to support the anticipated Phase-1 workload.

Lineate delivered actionable guidance framed as Specific, Measurable, Achievable, Relevant and Time-bound recommendations:

• Specific fixes: Rewrite per-record-type materialized views to use NULL-safe pivots, eliminating the data-erasure problem; add monthly time-based partitioning to raw data and some upstream tables to simplify cleanup.

• Measurable and relevant scale: Upgrade server resources to at least 32 CPU cores and 128 GB of RAM with 10–20 TB NVMe storage to support the expected 1 billion rows at launch.

• Time-bound roadmap: Prioritise pre-launch blockers (schema changes, partitions and indexes) for completion before the June 2026 launch; schedule query optimisations and replication enhancements for the first 90 days; plan sharding and further compression work for the 6- to 12-month horizon.

At the end of the review, Lineate issued a "Conditional Go" verdict. The client could proceed with launch once the identified blockers were addressed and the recommended infrastructure upgrades deployed. Beyond diagnostics, Lineate delivered specific SQL code to implement the NULL-safe pivots and partitioning schemes, along with example index definitions and deployment manifests. The engagement provided both immediate performance improvements and a clear roadmap for scaling ClickHouse from 10 million to 10 billion rows while maintaining stringent service-level objectives.

To make the launch achievable under a demanding timeline, every discovered issue was categorised into one of three clear buckets — so the team knew immediately what had to be addressed before go-live, what could follow in the first 90 days, and what belonged on the longer-term horizon: