During my first employment at Indeed, I cloned every repository down to my machine. This approach worked for a while when the number of repositories was small. As the organization has grown, the solution quickly became unmanageable. While many people do not work across every repository, many are familiar with the pain of setting up a new machine. I wrote gitfs for a few reasons. First, to reduce the time spent setting up a new development environment. Second, to remove the need to figure out where all my projects need to be cloned. In this post, I discuss some challenges faced and lessons learned in writing my first file system.

gitfs in Action

gitfs is a FUSE file system that helps reduce the management of git repositories. It works by connecting to well defined api’s (GitHub, Bitbucket, and Gitlab) and fetching repository urls associated with the user. These urls are parsed into a virutal directory structure that can be navigated via the terminal on linux or osx.

[mjpitz@mjpitz ~/Development/code 1/1]
$ ls
github.com

[mjpitz@mjpitz ~/Development/code 1/1]
$ cd github.com/

[mjpitz@mjpitz ~/Development/code/github.com 1/1]
$ ls
indeedeng  indeedeng-alpha  mjpitz

[mjpitz@mjpitz ~/Development/code/github.com 1/1]
$ cd mjpitz/

[mjpitz@mjpitz ~/Development/code/github.com/mjpitz 1/1]
$ ls
OpenGrok           gitfs            jgrapht     proto2-3
consul-api         grpc-java        laas        rpi
docker-clickhouse  grpc.github.io   mya.sh  seo-portal
docker-utils       grpcsh           mp          serverless-plugin-simulate
dotfiles           hbase-docker     okhttp      simple-daemon-node
envoy              idea-framework   proctor     spring-config-repo
generator-idea     java-gitlab-api  proctorjs

[mjpitz@mjpitz ~/Development/code/github.com/mjpitz 1/1]
$ cd mya.sh/

[mjpitz@mjpitz ~/Development/code/github.com/mjpitz/mya.sh master 1/1]
$ ls
Gemfile       _drafts    _posts              go              statics
Gemfile.lock  _includes  _site               index.html
_config.yml   _layouts   docker-compose.yml  pages
_data         _plugins   error.html          s3_website.yml

[mjpitz@mjpitz ~/Development/code/github.com/mjpitz/mya.sh master 1/1]
$

Challenge 1 - Finding a complete example

The first big challenge that I encountered was finding a complete working example. I chose the bazil/fuse library since it provided a clean low level implementation. Using a few basic tutorials, I was able to implement a read-only version of the file system. Unfortunately, the tutorials often only implemented a couple of interfaces from the library. And finding a complete example proved to be very difficult. Eventually, I stumbled across cockroachdb/examples-go which provides a good example to work off of.

Using this reference, I implemented 2 structures. One that represented a file and one that represented a directory. As the project progressed, having the logic in two separate files became difficult to manage. Eventually, these implementations collapsed into a single INode structure. This made it easy to keep a lot of business logic in one place. For portability, I added an interface for quick reference detailing which methods need to be implemented.

package filesystem

import "bazil.org/fuse/fs"

type INode interface {
	// common node functions
	fs.Node
	fs.NodeSetattrer

	// directory functions
	fs.NodeStringLookuper
	fs.HandleReadDirAller
	fs.NodeMkdirer
	fs.NodeCreater
	fs.NodeRemover
	fs.NodeRenamer
	fs.NodeSymlinker

	// handle functions
	fs.NodeOpener
	fs.HandleWriter
	fs.HandleReader
	fs.NodeFsyncer
	fs.HandleFlusher
	fs.HandleReleaser

	// symlink functions
	fs.NodeReadlinker
}

Challenge 2 - Debugging

Debugging a file system can be intense. Since many operations happen in such a short period of time, a full set of logs can quickly fill your disk. First, I started by only logging errors. That solution was insufficient. In many cases, context from the request and wrapping structure would’ve helped debug issues. After iterating on the log a few times, I wound up adding an info log at the start of the method. It included details about the request, details about the structure, as well as what method was being invoked. From this, I was able to see the full sequence of operations on the file system. But it was a lot.

In many cases, the error logs were enough to understand what went wrong. To reduce the volume in the typical case, I implemented a DEBUG mode. By default, the info log is suppressed. When DEBUG is set to true, the info log and the additional details are logged to stdout. Since debugging now requires restarting the file system, I needed to understand reproduction steps before restarting. By understanding the reproduction steps well, I am able to reproduce the issue quickly, keeping the debug log short and easy to read.