Performance Benchmarking of PostgreSQL on ScaleGrid vs. AWS RDS Using Sysbench
This article evaluates PostgreSQL’s performance on ScaleGrid and AWS RDS, focusing on versions 13, 14, and 15. We use Sysbench to benchmark key performance metrics under different workloads and thread configurations, including Transactions Per Second (TPS) and Queries Per Second (QPS). The results will help database administrators and decision-makers choose the right platform for their performance, scalability, and cost-efficiency needs.
Introduction
Purpose and Scope
Cloud-hosted PostgreSQL solutions are increasingly popular among organizations seeking scalable, high-performance databases. This study benchmarks PostgreSQL performance across two leading managed database platforms—ScaleGrid and AWS RDS—using versions 13, 14, and 15. The goal is to provide data-driven insights into how these platforms handle high-concurrency workloads.
Benchmark Objective
The primary objective of this benchmark is to measure and compare PostgreSQL’s performance on ScaleGrid and AWS RDS, analyzing how each platform scales under different PostgreSQL versions. Key metrics include TPS and QPS.
Audience
This article is designed for database administrators (DBAs), DevOps teams, system architects, CTOs, and any stakeholders responsible for selecting and optimizing database solutions in cloud environments.
Overview of PostgreSQL Versions
PostgreSQL 13, 14, and 15 Features
Each PostgreSQL version introduces key performance improvements and new features:
- PostgreSQL 13: Introduced optimizations for B-tree indexes, partitioning, and parallel vacuum operations, improving query performance and maintenance efficiency.
- PostgreSQL 14: Focused on improving parallel query performance and increased functionality for JSON data types. It also enhanced the management of large tables and indexes.
- PostgreSQL 15: Built on these improvements with faster sorts, extended statistics, and better performance for workloads involving large datasets.
For a more detailed comparison of performance features between different versions, refer to: https://www.postgresql.org/about/featurematrix/#performance
Benchmarking Methodology
Sysbench Overview
Sysbench is a versatile, open-source benchmarking tool ideal for testing OLTP (Online Transaction Processing) database workloads. It simulates high-concurrency environments, making it a go-to for performance testing of PostgreSQL across cloud platforms.
You can access the benchmark here: https://github.com/akopytov/sysbench.
Test Environment Setup
- Instance Types: We used similar cloud instances for AWS RDS and ScaleGrid to ensure a fair comparison. AWS RDS ran on db.m5.large instances, while ScaleGrid was configured on m5.large instances. Each ran with the following specs:
- 8GB RAM
- 2 vCPU
- 120GB SSD
- Configuration Details: Each PostgreSQL instance for Scalegrid and Amazon RDS was set up with default tuning parameters for PostgreSQL versions 13, 14, and 15. However, to ensure a level playing field regarding connection handling, we tuned ScaleGrid’s instances to allow 830 connections.
This adjustment was significant for Scalegrid, as its default PostgreSQL setup is highly configurable. By tuning Scalegrid, we aimed to match Amazon RDS’s connection capacity, ensuring that any performance differences observed were due to inherent platform efficiencies rather than default configurations. - Network Latency: We ran both machines in the same region and conducted the tests from within the same box in that region.
- Instance Configurations: The same AWS instance type was used for ScaleGrid and RDS.
- Storage I/O: Both ScaleGrid and RDS use GP3.
Sysbench Parameters
The following command was used to run sysbench:
sysbench --db-driver=pgsql --pgsql-user=postgres --pgsql_password="password" \ --pgsql-db=sbtest --pgsql-host=<hostname> --pgsql-port=5432 --tables=10 --table-size=1000000 \ --threads=500 --time=60 --report-interval=10 run
- Tables: 10
- Table Size: 1,000,000 rows per table
- Threads: 500 and 600
- Duration: 300 seconds per test
Metrics Collected
- Transactions Per Second (TPS): Number of transactions completed per second.
- Queries Per Second (QPS): Total number of queries executed each second.
Performance Test Results
PostgreSQL 13: AWS RDS vs. ScaleGrid
Higher is better
| Metric | RDS 500 Threads | ScaleGrid 500 Threads | RDS 600 Threads | ScaleGrid 600 Threads |
| TPS | 88.45 | 95.59 | 93.65 | 118.9 |
| QPS | 1840.85 | 1918.97 | 2020.81 | 2388.93 |
PostgreSQL 14: AWS RDS vs. ScaleGrid
Higher is better
| Metric | RDS 500 Threads | ScaleGrid 500 Threads | RDS 600 Threads | ScaleGrid 600 Threads |
| TPS | 85.74 | 97.31 | 106.91 | 116.66 |
| QPS | 1788.14 | 1954.32 | 2295.76 | 2342.98 |
PostgreSQL 15: AWS RDS vs. ScaleGrid
Higher is better
| Metric | RDS 500 Threads | ScaleGrid 500 Threads | RDS 600 Threads | ScaleGrid 600 Threads |
| TPS | 88.03 | 88.84 | 98.75 | 144.45 |
| QPS | 1832.02 | 1782.99 | 2124.84 | 2908.12 |
Analysis and Interpretation
Performance Trends
In all PostgreSQL versions tested, ScaleGrid almost consistently outperformed AWS RDS in both TPS and QPS.
Version-Specific Performance
- PostgreSQL 13: ScaleGrid showed an 8.07% improvement over AWS RDS in TPS and 4.24% in QPS. With 500 threads, this difference became even more pronounced, with a 26.96% improvement over RDS in TPS and an 18.22% improvement in QPS.
- PostgreSQL 14: With 500 threads, ScaleGrid’s lead widened to over 13.49% for TPS and 9.29% for QPS. The lead was less pronounced with 600 threads, with a 9.12% improvement in TPS and a 2.06% improvement in QPS.
- PostgreSQL 15: The latest version of PostgreSQL exhibited only marginal improvements on ScaleGrid compared to RDS with 500 threads but brought out the concurrency benefits with ScaleGrid at 600 threads. ScaleGris performed TPS better by 46.28% and QPS by 36.86%.
Cost Comparison
While performance metrics are crucial, the cost-to-performance ratio is just as important when choosing a cloud provider. Below is an estimated pricing comparison for both platforms during this benchmark.
Monthly Costs
- AWS RDS: The db.m5.large instance used for the benchmark costs $287.48 per month. This is calculated using AWS cost estimation tool.
- ScaleGrid: With similar specifications, ScaleGrid offers more flexibility in pricing, starting at $222 per month for equivalent configurations. This is calculated using ScaleGrid’s pricing plan.
Cost-Performance Analysis
In order to calculate cost per transaction, we first convert Transaction per Second (TPS) into Transaction per Minute (TPM) and then divide the cost per minute by the resulting number.
PostgreSQL 13: AWS RDS vs. ScaleGrid
Lower is better
| Metric | RDS 500 Threads | ScaleGrid 500 Threads | RDS 600 Threads | ScaleGrid 600 Threads |
| Cost per TPM | $1.25E-06 | $8.92E-07 | $1.18E-06 | $7.17E-07 |
Example Scenario: 25 Transactions per Second
To illustrate the cost-effectiveness and performance differences between AWS RDS and ScaleGrid for PostgreSQL, let’s consider a scenario with a steady load of 25 transactions per second, translating to approximately 64.8 million monthly transactions.
Examining the monthly costs for handling this load on both platforms under two thread configurations—500 and 600 threads—reveals that ScaleGrid consistently delivers more transactions per second (TPS) at a lower cost, providing a compelling case for choosing it over AWS RDS.
Monthly Cost=Cost per TPM × Transactions per minute × Minutes per month
- Transactions per minute (TPM) for 25 TPS = 1,500.
- Minutes per month = 60 × 24 × 30 = 43,200
- RDS 500 Threads: 1.25 × 10^(−6) × 1,500 × 43,200 = 81.00$
- ScaleGrid 500 Threads: 8.92 × 10^(−7) × 1,500 × 43,200 = 57.80$
- RDS 600 Threads: 1.18 × 10^(−6) × 1,500 × 43,200 = 76.46$
- ScaleGrid 600 Threads: 7.17 × 10^(−7) × 1,500 × 43,200 = 46.46$
ScaleGrid is more cost-effective than RDS in both scenarios (500 and 600 threads).
At 500 threads, ScaleGrid is approximately 29% cheaper than RDS.
At 600 threads, ScaleGrid is approximately 39% cheaper than RDS.
The combined graph shows monthly cost and transactions per second (TPS) for AWS RDS and ScaleGrid configurations at 500 and 600 threads. The bars represent the monthly costs, while the line plot with markers represents the TPS performance.
This visualization highlights that ScaleGrid provides higher performance at a lower monthly cost than AWS RDS in both configurations, illustrating the potential for cost savings with improved performance.
PostgreSQL 14: AWS RDS vs. ScaleGrid
Lower is better
| Metric | RDS 500 Threads | ScaleGrid 500 Threads | RDS 600 Threads | ScaleGrid 600 Threads |
| Cost per txn | 1.29E-06 | 8.76E-07 | 1.04E-06 | 7.31E-07 |
The monthly costs for handling 25 transactions per second on PostgreSQL 14 across the different configurations are:
- RDS 500 Threads: $83.59
- ScaleGrid 500 Threads: $56.76
- RDS 600 Threads: $67.39
- ScaleGrid 600 Threads: $47.37
Here is the combined graph showing both Monthly Cost and Transactions per Second (TPS) performance for PostgreSQL 14 (PG14) across AWS RDS and ScaleGrid configurations with 500 and 600 threads. The bars represent the monthly cost, while the line plot represents TPS performance:
This analysis demonstrates that ScaleGrid offers significant cost savings at 500 and 600 thread configurations compared to AWS RDS. For example, at 500 threads, ScaleGrid is approximately 32% cheaper than RDS, while at 600 threads, it provides a 30% cost reduction.
PostgreSQL 15: AWS RDS vs. ScaleGrid
Lower is better
| Metric | RDS 500 Threads | ScaleGrid 500 Threads | RDS 600 Threads | ScaleGrid 600 Threads |
| Cost per txn | 1.26E-06 | 9.60E-07 | 1.12E-06 | 5.90E-07 |
The monthly costs for handling 25 transactions per second on PostgreSQL 15 across the different configurations are:
- RDS 500 Threads: $81.65
- ScaleGrid 500 Threads: $62.21
- RDS 600 Threads: $72.58
- ScaleGrid 600 Threads: $38.23
The combined graph shows Monthly Cost and transaction per Second (TPS) performance for PostgreSQL 15 (PG15) across AWS RDS and ScaleGrid configurations with 500 and 600 threads. The bars represent the monthly cost, while the line plot represents TPS performance.
This analysis shows that ScaleGrid provides considerable cost savings over AWS RDS at both 500 and 600 thread configurations. For example, at 500 threads, ScaleGrid is approximately 24% cheaper than RDS, while at 600 threads, it delivers a 47% cost reduction compared to AWS RDS, all while achieving higher TPS performance.
Conclusion
ScaleGrid consistently provides better performance for PostgreSQL across all versions tested, particularly for high-concurrency workloads. For organizations prioritizing performance and scalability, ScaleGrid demonstrates superior results in both Transactions Per Second (TPS) and Queries Per Second (QPS), especially under heavy load.
Additionally, ScaleGrid offers a more flexible cost structure, making it a highly competitive option for businesses looking to optimize their PostgreSQL deployments without compromising on performance.
Key Findings
- Superior Performance: ScaleGrid outperformed AWS RDS across all PostgreSQL versions (13, 14, and 15), handling high-concurrency environments more efficiently.
- Cost-Effectiveness: ScaleGrid provides higher performance per dollar, making it the better choice for businesses with significant database workloads, especially those running complex OLTP applications.
- Platform Flexibility: ScaleGrid’s flexibility in PostgreSQL tuning and configuration offers more control for database administrators, enabling them to fine-tune their setup for specific use cases.
