23 Flat File Database Open-source, JavaScript, Rust, Python, Java, PHP, and Go

23 Flat File Database Open-source, JavaScript, Rust, Python, Java, PHP, and Go
Photo by Leone Venter / Unsplash

What is a Flat File Database?

Flat-file databases, well, they're a kind of database that keep data in a plain text file, right? Every line of that text file holds a record, with fields split by delimiters, like commas or tabs.

Some of them don't have the fancy relational structure of other database types, but they're pretty simple and straightforward to use.

Where can you use these flat-file databases?

Well, there's a lot of places. They're a good fit for smaller applications, where you just need to read or write records with a simple structure. Like, they're great for storing stuff like configuration data, user accounts, or any other data that's pretty simple in structure.

Well, there's a bunch of reasons. They're really portable and don't need a lot of setup, which makes them perfect for prototyping and small projects. They don't need a separate database server, which can make deploying them a lot easier. Plus, because the data is just plain text, you can easily look at it and edit it with a text editor.

One cool thing you can do with flat-file databases is caching. By keeping frequently used data in a cache, the database doesn't have to read from the file system every time that data is needed. This can speed things up a lot, especially for workloads where there's a lot of reading going on.

Flat-file DB limitations

But, you gotta remember, flat-file databases have their limits. They don't have the fancy querying stuff that SQL databases do and they can't handle a ton of data. They also can't support a bunch of writers at the same time. So, while they're a great choice for some things, they're not gonna be the best fit for every application.


1. LowDB

LowDB is a simple flat-file database library that resembles MongoDB queries. It stores and retrieves information directly from a JSON formatted flat-file.

I've utilized it in various web and desktop projects based on Electron for some clients. It has proved to be dependable and speedy. However, I wouldn't recommend it for applications handling large amounts of data.

It supports Typescript and Lodash library.

Features

  • Compact and Efficient: This ensures the database does not take up unnecessary resources.
  • Minimalist Design: The simple design makes it user-friendly and easy to navigate.
  • TypeScript Support: This allows developers who use TypeScript to easily work with the database.
  • Plain JavaScript: The database can be used with simple JavaScript, making it accessible to a wide range of developers.
  • Safe Atomic Writes: This feature ensures that all data operations are completed safely without any data loss or corruption.
  • Customizable: Users have the option to change the storage, file format (such as JSON, YAML, etc.), or even add encryption through adapters. This flexibility allows the database to be tailored to specific project needs.
  • Extended Functionality: The database can be extended with lodash, ramda, and more, giving it super powers and making it incredibly versatile.
  • Testing Efficiency: For more efficient testing, it automatically switches to a high-speed in-memory mode.

Install

npm install lowdb

Usage

import { JSONFilePreset } from 'lowdb/node'

// Read or create db.json
const defaultData = { posts: [] }
const db = await JSONFilePreset('db.json', defaultData)

// Update db.json
await db.update(({ posts }) => posts.push('hello world'))

// Alternatively you can call db.write() explicitely later
// to write to db.json
db.data.posts.push('hello world')
await db.write()
GitHub - typicode/lowdb: Simple and fast JSON database
Simple and fast JSON database. Contribute to typicode/lowdb development by creating an account on GitHub.

2. NeDB (JavaScript)

NeDB is a powerful and lightweight JavaScript database, drawing inspiration from MongoDB. It seamlessly supports a MongoDB-like API. With its versatile use in Node.js and the browser, it stands as the ideal choice for apps requiring a database solution without the need for a full MongoDB instance.

It is an ideal solution for small web apps, desktop applications, and educational purposes, it's not just a good choice, it's the best choice.

GitHub - louischatriot/nedb: The JavaScript Database, for Node.js, nw.js, electron and the browser
The JavaScript Database, for Node.js, nw.js, electron and the browser - louischatriot/nedb

3. Teensy (JavaScript)

Teensy is a compact, highly efficient flat file database, adept at storing data in memory and periodically writing it to a file. Its performance metrics showcase its proficiency in swiftly seeking and placing items, maintaining an impressive speed even as the number of items increases.

Install

npm install --save teensy

Use

const Teensy = require('teensy');
// create database
let DB = new Teensy('teensy.db', 1000).store();
// put data
DB.put({_id: 'Ringo', _rev: 2, color: 'Orange', fangs: true});
GitHub - InfamousVague/teensy: At just 1.5k and 62 lines of JS, Teensy is a very small flat file database.
At just 1.5k and 62 lines of JS, Teensy is a very small flat file database. - InfamousVague/teensy

4. TSDB (JavaScript)

TSDB is a fast and reliable database port of the original sdb database written in PHP. It is my first TypeScript package, and I welcome any comments or feedback on my website or social media.

Install

npm install ts-sdb

Usage

import { TSDB } from 'tsdb';

// Create a new instance of TSDB
const db = new TSDB();

// Load the database
db.load('mydatabase');

// Perform operations on the database
db.create('key1', 'value1');
db.create('key2', 'value2');
db.update('key1', { value: 'newvalue' });
db.delete('key2');

// Save changes to the database
db.save();
GitHub - devsimsek/tsdb: TSDB - Fast, Reliable database port of the original sdb database (php)
Fast, Reliable database port of the original sdb database (php) - devsimsek/tsdb

5. JsonStreamDb (Node.js)

This is a simple flat-file database that uses JSON files to store their data.

Usage

var JsonStreamDb = require('jsonstreamdb');

var db = new JsonStreamDb('test.jsonstreamdb');
db.pipe(process.stdout, {history: true});

db.create('users', '47c0479c-2083-4797-8d3c-419de31d45a7', {userName: 'geon'});
db.update('users', '47c0479c-2083-4797-8d3c-419de31d45a7', {favoriteAnimal: 'kittens'});
db.delete('users', '47c0479c-2083-4797-8d3c-419de31d45a7');
GitHub - geon/jsonstreamdb: just dicking around
just dicking around. Contribute to geon/jsonstreamdb development by creating an account on GitHub.

6. Flat (PHP)

This is another open-source is a flat-file database with a mongo-like API, written in PHP. It is ideal for smart projects and examples that require CRUD operations.

Use it with caution, as it did not receive updates for years.

GitHub - mattbit/flat: A flat file database for PHP
A flat file database for PHP. Contribute to mattbit/flat development by creating an account on GitHub.

7. Daybreak (Ruby)

Daybreak confidently stands as a rapid, user-friendly key value store for Ruby. It boasts user-defined persistence and in-memory data storage, outstripping pstore and dbm in speed.

It expertly stores data in an append-only file and supports marshalled Ruby objects with finesse. Inclusive of Enumerable for functional methods, Daybreak effortlessly emulates a simple Ruby hash interface.

Install

$ gem install daybreak

Example

  require 'daybreak'

  db = Daybreak::DB.new "example.db"

  # set the value of a key
  db['foo'] = 2

  # set the value of a key and flush the change to disk
  db.set! 'bar', 2

  # You can also use atomic batch updates
  db.update :alpha => 1, :beta => 2
  db.update! :alpha => 1, :beta => 2

  # all keys are cast to strings via #to_s
  db[1] = 2
  db.keys.include? 1 # => false
  db.keys.include? '1' # => true

  # ensure changes are sent to disk
  db.flush

  # open up another db client
  db2 = Daybreak::DB.new "example2.db"
  db2['foo'] = 3

  # Ruby objects work too
  db2['baz'] = {:one => 1}
  db2.flush

  # Reread the changed file in the first db
  db.load
  p db['foo'] #=> 3
  p db['baz'] #=> {:one => 1}

  # Enumerable works too!
  1000.times {|i| db[i] = i }
  p db.reduce(0) {|m, k, v| m + k.last } # => 499500

  # Compaction is always a good idea. It will cut down on the size of the Database
  db.compact
  p db['foo'] #=> 1
  db2.load
  p db2['foo'] #=> 1

  # DBs can accessed from multiple processes at the same
  # time. You can use #lock to make an operation atomic.
  db.lock do
    db['counter'] += 1
  end

  # If you want to synchronize only between threads, prefer synchronize over lock!
  db.synchronize do
    db['counter'] += 1
  end

  # DBs can have default values
  db3 = Daybreak::DB.new "example3.db", :default => 'hello!'
  db3['bingo'] #=> hello!

  # If you don't like Marshal as serializer, you can write your own
  # serializer. Inherit Daybreak::Serializer::Default
  db4 = Daybreak::DB.new "example4.db", :serializer => MyJsonSerializer

  # close the databases
  db.close
  db2.close
  db3.close
  db4.close
Daybreak

8. Rustbreak (Rust)

Rustbreak is a robust, standalone file database, drawing inspiration from Daybreak and finely tuned for maximum speed and simplicity. It positions itself as the prime choice for swift application development in Rust, providing unparalleled persistence and the ability to write diverse data.

Though it's necessary for the entire database to fit into memory and not all backends extend support for atomic saves, these are minor constraints against the backdrop of its immense advantages.

Importantly, Rustbreak is thread-safe and confidently facilitates Serde compatible storage.

GitHub - TheNeikos/rustbreak: A simple, fast and easy to use self-contained single file storage for Rust
A simple, fast and easy to use self-contained single file storage for Rust - TheNeikos/rustbreak

9. Yellow-Table (PHP)

Yellow Table is a simple flat-file database management tool. To use it, download the ZIP file and copy it into your system/extensions folder.

Tables can be displayed using a [table] shortcut, with optional arguments for name, caption, and style. Files are stored in the media/tables folder, but additional folders can be created for organization.

GitHub - GiovanniSalmeri/yellow-table: Simple flat-file database management.
Simple flat-file database management. Contribute to GiovanniSalmeri/yellow-table development by creating an account on GitHub.

10. QuartzDB (PHP)

QuartzDB is a superb flat-file database expertly optimized for time-series data. Perfect for storing GPS logs, chat logs, and sensor logs from IoT devices, it's the go-to choice for those who appreciate the ease of backing up text files and have confidence in their long-term durability over binary database files.

GitHub - aaronpk/QuartzDB: A flat-file database optimized to hold time-series data.
A flat-file database optimized to hold time-series data. - aaronpk/QuartzDB

11. Lazer (PHP)

Lazer is a commanding PHP library that cleverly leverages JSON files as a database, equipped with capabilities akin to ORM's. It assertively requires PHP 7.1+ and Composer.

It stands as a top-tier choice for developers: swift, meticulously documented, offering relations, multiple select and supporting CRUD operations.

GitHub - Lazer-Database/Lazer-Database: PHP flat file database to store data with JSON
PHP flat file database to store data with JSON. Contribute to Lazer-Database/Lazer-Database development by creating an account on GitHub.

12. SleekDB (PHP)

SleekDB is a NoSQL-like database system implemented in PHP that stores data in plain JSON files without the need for any third-party dependencies. Unlike relational databases such as SQLite, MySQL, PostgreSQL, and MariaDB, SleekDB is more akin to MongoDB in terms of functionality.

It is specifically designed to manage a few gigabytes of data, making it ideal for low to medium operation loads. It is not intended for handling high-load IO operations.

SleekDB is particularly effective as a database engine for websites with low to medium traffic, simplifying database management. Another key feature of SleekDB is its ability to cache all query data by default, making dynamic sites as fast as static ones.

While it is a flat file database, it diverges from the norm by storing data in multiple JSON files rather than a single one. This approach enhances concurrency support compared to fully flat file database systems. The final query output is cached and later reused from a single file, eliminating the need to traverse all available files.

Features

  • Lightweight and Fast: SleekDB stores data in plain-text using JSON format, eliminating binary conversions. It also features a default query cache layer.
  • Schema-Free: No schema is required, enabling the insertion of any data types.
  • Query on Nested Properties: Schema-free data allows filtering and conditioning on nested properties of JSON documents.
  • Dependency-Free: Only requires PHP to run and supports PHP 7+. No need for third-party plugins or software.
  • Caching: SleekDB employs a default caching layer, serving data from cache by default and regenerating cache automatically.
  • Rich Conditions and Filters: Allows for multiple conditional comparisons, text searches, and sorting on multiple and nested properties.
  • On-Demand Data Processing: No background process or network protocol is required. All data for a query is fetched at runtime within the same PHP process.
  • Universal Compatibility: Runs perfectly on shared servers or VPS.
  • User-Friendly: SleekDB has an elegant and easy-to-understand API for data handling.
  • Easy Data Management: Data stored in files makes backup, import, and export tasks easier.
  • Actively Maintained: Created by @rakibtg and actively maintained by Timucin, SleekDB is continually improved in terms of code quality and new features.
SleekDB - PHP NoSQL Flat File Database
SleekDB is a simple flat file NoSQL like database implemented in PHP without any third-party dependencies that store data in plain JSON files.

13. FlatFiles (C#)

FlatFiles is an authoritative library that proficiently reads and writes CSV, fixed-length, and other flat file formats. This library prioritizes schema definition and proudly offers direct mapping between files and classes.

The schema, intelligently defined by data columns in your file, expertly guides the reader or writer in extracting or writing out values. FlatFiles commands comprehensive control over parsing/formatting behavior, handling unusual edge cases with ease. For those working with ADO.NET, it also confidently supports DataTables and IDataReader.

GitHub - jehugaleahsa/FlatFiles: Reads and writes CSV, fixed-length and other flat file formats with a focus on schema definition, configuration and speed.
Reads and writes CSV, fixed-length and other flat file formats with a focus on schema definition, configuration and speed. - jehugaleahsa/FlatFiles

14. FlatDB (Java)

FlatDB is an incredibly intuitive library for flat-file databases. It empowers you to create tables using Java classes, proactively updates tables when new fields are introduced, and ensures a seamless connection to your database file.

Please note, it necessitates Java 8 or higher.

GitHub - CyR1en/FlatDB: An easy to use library for flat-file databases.
An easy to use library for flat-file databases. Contribute to CyR1en/FlatDB development by creating an account on GitHub.

15. Flatbase (PHP)

Flatbase is a PHP-based flat file database that is lightweight, easy to install with minimal configuration, has a simple intuitive API, and is suitable for small data sets, low-load applications, and testing or prototyping.

Installation

composer require flatbase/flatbase

Example

<?php

$storage = new Flatbase\Storage\Filesystem('/path/to/storage/dir');
$flatbase = new Flatbase\Flatbase($storage);

$flatbase->insert()->in('users')
    ->set(['name' => 'Adam', 'height' => "6'4"])
    ->execute();

$flatbase->read()->in('users')
    ->where('name', '=', 'Adam')
    ->first();
// (array) ['name' => 'Adam', 'height' => "6'4"]

GitHub - adamnicholson/flatbase: Flat file database for PHP
Flat file database for PHP. Contribute to adamnicholson/flatbase development by creating an account on GitHub.

16. Filebase (PHP)

Despite the name, this is a different database than the last one.

Filebase is an uncompromisingly robust flat file database storage solution. There's no need for MySQL or an overpriced SQL server. All database entries are securely stored in files, accessible for direct modification or through our sleek API.

It supports version control, facilitating effortless sharing and syncing of data among teams. It's designed to work seamlessly with PHP 5.6 and PHP 7+.

Features

While Filebase is simple in design, it offers a range of advanced features:

  • Storage of Key/Value and Array-based Data
  • Data Querying
  • Customizable filters
  • Query Caching
  • Database Backup capabilities
  • Data Formatting (encode/decode)
  • Data Validation upon saving
  • CRUD (Create, Read, Update, Delete) APIs
  • File locking when saving
  • User-friendly Method Naming
GitHub - tmarois/Filebase: A Simple but Powerful Flat File Database Storage.
A Simple but Powerful Flat File Database Storage. - GitHub - tmarois/Filebase: A Simple but Powerful Flat File Database Storage.

17. DBJson (Python)

DBJson is a robust flat file database system that confidently safeguards its data within the filesystem, leveraging a blend of folders and JSON files. It's essential to underscore that DBJson is NOT DESIGNED FOR PRODUCTION USE.

Yet, it stands as the optimal solution for situations where the establishment of a formal schema and the execution of additional tasks common to Object-Relational Mapping (ORM) frameworks, like SQLAlchemy, appear excessive or superfluous.

Install

pip install dbjson

Example

from dbjson import DB
# Use the syntax below if you plan to include raw code in your project.
# from dbjson.db import DB 

# Initializing DB class from dbjson.main
db = DB()

# Test Data
data = {
  "id": 1,
  "first_name": "Vivyan",
  "last_name": "Treherne",
  "email": "[email protected]",
  "ip_address": "94.254.247.240"
}
collection = "users"

# Adding Record
data = db.createRecord(collection, data)
print(data)
# Response -> {'__id__': 'f00ae4e3ca8c3e318a68acc460e5f401', '__data__': {'id': 1, 'first_name': 'Vivyan', 'last_name': 'Treherne', 'email': '[email protected]', 'ip_address': '94.254.247.240'}}

# Updating Record
record_key = "f00ae4e3ca8c3e318a68acc460e5f401"
to_update = [
    {"email": "[email protected]"},
    {"ip_address": "google.com"}
]
data = db.updateRecord(collection, "f00ae4e3ca8c3e318a68acc460e5f401", to_update)
print(data)
# Response -> {'id': 1, 'first_name': 'Vivyan', 'last_name': 'Treherne', 'email': '[email protected]', 'ip_address': 'google.com'}

# Deleting Record
db.removeRecord(collection, record_key)
GitHub - ketanip/dbjson: This is a simple flat file database which stores its data in a JSON file.
This is a simple flat file database which stores its data in a JSON file. - GitHub - ketanip/dbjson: This is a simple flat file database which stores its data in a JSON file.

19. Flat-file-db (JavaScript)

Flat-file-db is an efficient flat file database for Node.js. It supports JSON and retains all data in memory. The database uses an append-only method for data persistence, which guarantees small file sizes and robust consistency. To utilize it, install it through npm and provide a database file to the flat-file-db constructor.

Although it is still operational and preferred by many projects, it has not received updates for years.

Install

npm install flat-file-db

Usage

var flatfile = require('flat-file-db');
var db = flatfile('/tmp/my.db');

db.on('open', function() {
	db.put('hello', {world:1});  // store some data
	console.log(db.get('hello')) // prints {world:1}

	db.put('hey', {world:2}, function() {
		// 'hey' is now fully persisted
	});
});
GitHub - mafintosh/flat-file-db: Fast in-process flat file database that caches all data in memory
Fast in-process flat file database that caches all data in memory - mafintosh/flat-file-db

20. TinyDB (Python)

TinyDB is a lightweight, document-oriented database that is written in pure Python, requiring no external servers or dependencies. With a simple and clean API, it is designed for ease of use. It allows storage of any document represented as a dictionary, similar to MongoDB.

For projects necessitating advanced features like multi-process or thread access, table indexing, an HTTP server, managing table relationships, or ACID guarantees, TinyDB might not be your optimal choice.

If your project calls for high-speed database performance, TinyDB might not be the ideal fit. In these instances, it's worth exploring other databases such as SQLite, Buzhug, CodernityDB, or MongoDB.

Features

  • Lightweight and document-oriented: TinyDB is efficient, storing any document as a dictionary, much like MongoDB.
  • Pure Python: TinyDB is written purely in Python and requires no external servers or dependencies.
  • User-friendly: TinyDB comes with a simple and clean API, making it easy and fun to use.
  • Compatibility: It is compatible with Python 3.5+ and PyPy.
  • Extensibility: TinyDB allows for customization by writing new storages or modifying the behaviour of storages with Middlewares.
  • Test Coverage: TinyDB boasts a 100% test coverage.

Example

>>> from tinydb import TinyDB, Query
>>> db = TinyDB('/path/to/db.json')
>>> db.insert({'int': 1, 'char': 'a'})
>>> db.insert({'int': 1, 'char': 'b'})

Query language

>>> User = Query()
>>> # Search for a field value
>>> db.search(User.name == 'John')
[{'name': 'John', 'age': 22}, {'name': 'John', 'age': 37}]

>>> # Combine two queries with logical and
>>> db.search((User.name == 'John') & (User.age <= 30))
[{'name': 'John', 'age': 22}]

>>> # Combine two queries with logical or
>>> db.search((User.name == 'John') | (User.name == 'Bob'))
[{'name': 'John', 'age': 22}, {'name': 'John', 'age': 37}, {'name': 'Bob', 'age': 42}]

>>> # Apply transformation to field with `map`
>>> db.search((User.age.map(lambda x: x + x) == 44))
>>> [{'name': 'John', 'age': 22}]

>>> # More possible comparisons:  !=  <  >  <=  >=
>>> # More possible checks: where(...).matches(regex), where(...).test(your_test_func)
GitHub - msiemens/tinydb: TinyDB is a lightweight document oriented database optimized for your happiness :)
TinyDB is a lightweight document oriented database optimized for your happiness :) - msiemens/tinydb
Welcome to TinyDB! — TinyDB 4.8.0 documentation

21. FlatDB (PHP)

FlatDB is an exceptionally efficient flat file database, masterfully designed to persist data using nothing more than PHP and flat files. It's the ideal solution when no other database is at your disposal.

Features

  • Lightweight and Secure
  • Easy to get started, just include flatdb.php to your app
  • No external dependencies
  • Built-in caching system for better performance
  • Powerful API with chaining methods
  • CRUD (create, read, update, delete) operations
  • Supports custom indexes
  • Supports WHERE(), LIMIT(), OFFSET(), ORDER(), FIND(), COUNT() and other methods
  • Supports multiple databases and tables
GitHub - onyxhat/FlatDB: Simple flat file database designed to persist data using just PHP and flat files.
Simple flat file database designed to persist data using just PHP and flat files. - onyxhat/FlatDB

22. flatfile (Go)

The package confidently provides a powerful utility to read fixed-width records from a flat-file database or text file, effortlessly wiping away the need for boilerplate code for data reading and mapping to struct fields.

The record layout mapping can be swiftly defined using struct tags, and, brilliantly, the package also packs an 'Unmarshal' method to seamlessly convert a record into a struct.

GitHub - ahmedalhulaibi/flatfile: A library to unmarshal structured data read from a flat-file (text file) into a struct using struct tags
A library to unmarshal structured data read from a flat-file (text file) into a struct using struct tags - ahmedalhulaibi/flatfile

23. PHP JLDB (JSON Lite DB)

Simple and powerfull tool that allows to use json file like a database. It provides collection of methods that you can use like a database query builder.

GitHub - mohamed-samir907/JLDB: Simple and powerfull tool that allows to use json file like a database. It provides collection of methods that you can use like a database query builder.
Simple and powerfull tool that allows to use json file like a database. It provides collection of methods that you can use like a database query builder. - mohamed-samir907/JLDB