MarkdownDB: Convert Your Markdown Files Into Structured Queryable Database
MarkdownDB is a lightweight JavaScript-based database system designed for developers and technical users who prefer working with Markdown files. It allows users to manage, query, and interact with data stored in Markdown format.
Specifically it:
- Parses your markdown files to extract structured data (frontmatter, tags etc) and builds a queryable index either in JSON files or a local SQLite database
- Provides a lightweight javascript API for querying the index and using the data files into your application
Ideal for:
MarkdownDB is ideal for developers looking for a simple and flexible way to store and retrieve data using Markdown files without the need for a full-fledged database system.
Features
- Rich metadata extracted including frontmatter, links and more.
- Lightweight and fast indexing 1000s of files in seconds.
- Open source and extensible via plugin system.
- Data in Markdown: Store and manage data directly within Markdown files, allowing easy integration with static sites and other Markdown-based workflows.
- Query Support: Query data using a simple API, making it easy to retrieve and manipulate data from Markdown files.
- Schema-less: No need to define schemas; the data structure is inferred from the content of the Markdown files.
- Search and Filtering: Built-in capabilities to search and filter content within Markdown files based on various criteria.
- Easy Integration: Can be easily integrated into existing projects, particularly those that rely on Markdown for content management.
- Simple API: Get a list of all or some Markdown files, filter them by frontmatter fields, and more.
Install and Use
Have a folder of markdown content
For example, your blog posts. Each file can have a YAML frontmatter header with metadata like title, date, tags, etc.
---
title: My first blog post
date: 2021-01-01
tags: [a, b, c]
author: John Doe
---
# My first blog post
This is my first blog post.
I'm using MarkdownDB to manage my blog posts.
Index the files with MarkdownDB
Use the npm mddb package to index Markdown files into an SQLite database. This will create a markdown.db file in the current directory. You can preview it with any SQLite viewer, e.g. https://sqlitebrowser.org/.
# npx mddb <path-to-folder-with-your-md-files>
npx mddb ./blog
Watching for Changes
To monitor files for changes and update the database accordingly, simply add the --watch flag to the command:
npx mddb ./blog --watch
This command will continuously watch for any modifications in the specified folder (./blog), automatically rebuilding the database whenever a change is detected.
Query your files with SQL...
E.g. get all the files with with tag a.
SELECT files.*
FROM files
INNER JOIN file_tags ON files._id = file_tags.file
WHERE file_tags.tag = 'a'
...or using MarkdownDB Node.js API in a framework of your choice!
Use our Node API to query your data for your blog, wiki, docs, digital garden, or anything you want!
Install mddb package in your project:
npm install mddb
Now, once the data is in the database, you can add the following script to your project (e.g. in /lib folder). It will allow you to establish a single connection to the database and use it across you app.
// @/lib/mddb.mjs
import { MarkdownDB } from "mddb";
const dbPath = "markdown.db";
const client = new MarkdownDB({
client: "sqlite3",
connection: {
filename: dbPath,
},
});
const clientPromise = client.init();
export default clientPromise;
Now, you can import it across your project to query the database, e.g.:
import clientPromise from "@/lib/mddb";
const mddb = await clientPromise;
const blogs = await mddb.getFiles({
folder: "blog",
extensions: ["md", "mdx"],
});
Computed Fields
This feature helps you define functions that compute additional fields you want to include.
Step 1: Define the Computed Field Function
Next, define a function that computes the additional field you want to include. In this example, we have a function named addTitle that extracts the title from the first heading in the AST (Abstract Syntax Tree) of a Markdown file.
const addTitle = (fileInfo, ast) => {
// Find the first header node in the AST
const headerNode = ast.children.find((node) => node.type === "heading");
// Extract the text content from the header node
const title = headerNode
? headerNode.children.map((child) => child.value).join("")
: "";
// Add the title property to the fileInfo
fileInfo.title = title;
};
Step 2: Indexing the Folder with Computed Fields
Now, use the client.indexFolder method to scan and index the folder containing your Markdown files. Pass the addTitle function in the computedFields option array to include the computed title in the database.
client.indexFolder(folderPath: "PATH_TO_FOLDER", customConfig: { computedFields: [addTitle] });
Configuring markdowndb.config.js
- Implement computed fields to dynamically calculate values based on specified logic or dependencies.
- Specify the patterns for including or excluding files in MarkdownDB.
Example Configuration
Here's an example markdowndb.config.js with custom configurations:
export default {
computedFields: [
(fileInfo, ast) => {
// Your custom logic here
},
],
include: ["docs/**/*.md"], // Include only files matching this pattern
exclude: ["drafts/**/*.md"], // Exclude those files matching this pattern
};
(Optional) Index your files in a prebuild script
{
"name": "my-mddb-app",
"scripts": {
...
"mddb": "mddb <path-to-your-content-folder>",
"prebuild": "npm run mddb"
},
...
}
With Next.js project
For example, in your Next.js project's pages, you could do:
// @/pages/blog/index.js
import React from "react";
import clientPromise from "@/lib/mddb.mjs";
export default function Blog({ blogs }) {
return (
<div>
<h1>Blog</h1>
<ul>
{blogs.map((blog) => (
<li key={blog.id}>
<a href={blog.url_path}>{blog.title}</a>
</li>
))}
</ul>
</div>
);
}
export const getStaticProps = async () => {
const mddb = await clientPromise;
// get all files that are not marked as draft in the frontmatter
const blogFiles = await mddb.getFiles({
frontmatter: {
draft: false,
},
});
const blogsList = blogFiles.map(({ metadata, url_path }) => ({
...metadata,
url_path,
}));
return {
props: {
blogs: blogsList,
},
};
};
License
MIT License
Resources & Downloads
datopian
