How to Decide Storage Solution in Google Cloud Platform

Problem

The application needs a cost-effective, reliable, and highly scalable solution for key-value storage.

Workload pattern: the most common (> 80%) operation is a key write. The application needs a fast lookup (< 10 seconds).

Solutions

The storage solutions in GCP are painful to understand. Every one of them offers almost similar features. There are the following solutions available to store key-value data:

1. Big Query: An non-index storage, suitable for batch processing
2. BigTable: A low latency NoSQL storage, with a primary index
3. Firestore: A document store (like MongoDB), supports primary & secondary indexes
4. Cloud Storage: A S3 like objects store
5. CloudSQL: A relational DB (MySQL/Postgres)

How to Decide

The following dimensions can help find the most suitable fit:

1. Latency for operations
2. Type of workload
3. Cost

1. Operations Latency

The application’s latency requirement is < 10 seconds.

• BigTable: 10s of milliseconds
• Big Query: a few minutes to hours
• FireStore: few seconds (1–5s)
• Cloud Storage: few seconds (1 -5 s)