# SQLite is not a toy database

![rw-book-cover](https://antonz.org/assets/2021/sqlite-is-not-a-toy-database.png)

## Metadata
- Author: [[Anton Zhiyanov]]
- Full Title: SQLite is not a toy database
- Category: #articles
- Document Tags: [[dev]] [[dev/sql]] 
- URL: https://antonz.org/sqlite-is-not-a-toy-database/

> [!note]
> **Background:** Utilize SQLite for a wider array of applications beyond merely functioning as a basic data storage solution.
> ### Key Takeaways:
> 📊 **Leverage Advanced Indexing Capabilities:** 
> SQLite supports partial indexes and indexes on expressions, enabling the creation of efficient indexes on generated columns. This functionality allows users to transform SQLite into a document database by storing raw JSON and indexing on extracted JSON fields.
> 📝 **Utilize Built-in Statistical Functions:** 
> SQLite makes it easy to perform descriptive statistics, such as calculating mean, median, and percentiles, with minimal setup. By loading the sqlite3-stats extension, users can execute complex statistical queries in a straightforward manner, enhancing data analysis capabilities.
> 📁 **Seamless JSON Data Handling:** 
> SQLite excels at analyzing and transforming JSON data, allowing users to directly query JSON files as if they were regular tables. This feature simplifies data manipulation, making it convenient for users to extract and analyze JSON information efficiently.
> [!tldr]
> SQLite is highlighted as a versatile tool suitable for developers, data analysts, and various professionals due to its widespread use, serverless nature, and ease of integration. The database console feature is particularly praised for its data analysis capabilities and simplicity, offering powerful functions like CSV import, SQL query support, and various export options. Additionally, SQLite's compatibility with data exploration tools and its efficiency in processing large datasets make it a robust choice for those working with JSON, Common Table Expressions, set operations, generated columns, and mathematical statistics. The document also touches on SQLite's performance capabilities, handling hundreds of millions of records efficiently, with insert speeds improving when connecting CSV files as virtual tables.

## Highlights
The console is a killer SQLite feature for data analysis: more powerful than Excel and more simple than `pandas`. One can import CSV data with a single command, the table is created automatically:
> .import --csv city.csv city
> select count(*) from city;
1117 [View Highlight](https://read.readwise.io/read/01j6hehc1ap6yxya4135j8z7d1))

Data could be exported as SQL, CSV, JSON, even Markdown and HTML. Takes just a couple of commands:
.mode json
.output city.json
select city, foundation_year, timezone from city limit 10;
.shell cat city.json [View Highlight](https://read.readwise.io/read/01j6hekgzt1yg4dse4xt6kqec4))

There is nothing more convenient than SQLite for analyzing and transforming JSON. You can select data directly from a file as if it were a regular table. Or import data into the table and select from there.
select
json_extract(value, '$.iso.code') as code,
json_extract(value, '$.iso.number') as num,
json_extract(value, '$.name') as name,
json_extract(value, '$.units.major.name') as unit
from
json_each(readfile('currency.sample.json'))
; [View Highlight](https://read.readwise.io/read/01j6henb91d2vd1d43x4yj6e5w))

Descriptive statistics? Easy: mean, median, percentiles, standard deviation, you name it. You’ll have to load an extension, but it’s also a single command (and a single file).
.load sqlite3-stats
select
count(*) as book_count,
cast(avg(num_pages) as integer) as mean,
cast(median(num_pages) as integer) as median,
mode(num_pages) as mode,
percentile_90(num_pages) as p90,
percentile_95(num_pages) as p95,
percentile_99(num_pages) as p99
from books; [View Highlight](https://read.readwise.io/read/01j6hev7qj7qyszxhwft53r1w9))

There is a popular opinion among developers that SQLite is not suitable for the web, because it doesn’t support concurrent access. ... the write-ahead log mode ... there can be as many concurrent readers as you want. There can be only one concurrent writer, but often one is enough.

SQLite supports partial indexes and indexes on expressions, as ‘big’ DBMSs do. You can build indexes on generated columns and even turn SQLite into a document database. Just store raw JSON and build indexes on `json_extract()`-ed columns [View Highlight](https://read.readwise.io/read/01j6hf4c5250yyf6871sphxenm))