Contributing to Dgraph

From Dgraph Wiki
Jump to: navigation, search

General

Wiki

  • Contributions to wiki are welcome and highly appreciated.
  • If you see any issues with the documentation in this wiki, please do one of two things:
    • Sign up and edit the wiki to fix the issue.
    • File a bug against us in Github Issues.[1]
  • To reference how Wiki Formatting works, please see MediaWiki Help:Formatting[2] page.
  • Wiki templates are a great way to bring various things to users' attention, including the validity and freshness of the page. You can see the list of all templates supported by Dgraph here.[3]
  • To upload images visit the Upload page.[4]
  • Wiki is kept fresh and up to date due to contributions from the community. Thank you!

Guidelines

Note:Over years of writing big scalable systems, I'm convinced that striving for simplicity wherever possible is the only way to build robust systems. This simplicity could be in design, could be in coding, or could be achieved by rewriting the whole module, you painstakingly finished yesterday. -mrjn
  • Pull requests are welcome, as long as you're willing to put in the effort to meet the guidelines.
  • Dgraph aims for clear, well written, maintainable code.
  • Simple and minimal approach to features, like Go.
  • Refactoring existing code now for better performance, better readability or better testability wins over adding a new feature.
  • Don't add a function to a module that you don't use right now, or doesn't clearly enable a planned functionality.
  • Don't ship a half done feature, which would need to be significantly altered to work fully.
  • Avoid Technical debt[5] like cancer.
  • Leave the code cleaner than when you began.

Style Guide

  • If you see any code which clearly violates the style guide, please fix it and send a pull request. No need to ask for permission.
  • We're following Go Code Review.[6]
  • Use gofmt for formatting your code.
  • Avoid unnecessary vertical spaces. Use your judgment or follow the code review comments.
  • Wrap your code and comments to 100 characters, unless doing so makes the code less legible.

Setting up your system

This section explains how to setup some tools and environments that we recommend for the development of Dgraph.

Note: If you're using Ubuntu, make sure you run sudo apt-get update before you proceed further. If you're using Arch Linux, make sure you have yaourt[7] installed.

Install Go 1.8

Manually

Download and install Go 1.8 from here.[8] Here I install the latest point release. We usually install to /usr/local/go. If you have an old version there, please delete it first. Using the non-source version is easiest, but do check the SHA256 checksums.

# check if Go is installed,
#   and uninstall from our recommended location
$ go version \
>    && rm -Rf /usr/local/go/
bash: go: command not found

# download the archive and secure hash files
$ wget -q https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz \
>         https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz.sha256

# visually compare the calculated secure hash to the expected value
$ sha256sum go1.8.linux-amd64.tar.gz; \
>       cat go1.8.linux-amd64.tar.gz.sha256; \
>      echo ""
53ab94104ee3923e228a2cb2116e5e462ad3ebaeea06ff04463479d7f12d27ca  go1.8.linux-amd64.tar.gz
53ab94104ee3923e228a2cb2116e5e462ad3ebaeea06ff04463479d7f12d27ca

# extract the Go binaries,
#   and remove the download files on success
$ tar -C /usr/local/ -xzf go1.8.linux-amd64.tar.gz \
>     && rm -f go1.8.linux-amd64.tar.gz*

Once installed, ensure that you set the GOPATH correctly as describe in the installation doc[9]. Setting these in your ~/.bashrc should be sufficient.

export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

Perform a minimal check for installation and setup success. See Test your installation[10] for a more complete test.

$ go version
go version go1.8 linux/amd64

Arch Linux

In Arch Linux, you can directly install Go from pacman.

$ sudo pacman -S go
$ go version
go version go1.8 linux/amd64

Mac

In Mac, you can also use homebrew. Homebrew[11] needs to be installed on Mac before using the command below.

$ brew tap homebrew/versions
$ brew install go
$ go version
go version go1.8 darwin/amd64

Install RocksDB

Dgraph depends on RocksDB[12] for storing posting lists.

Note:Dgraph maintains its own C API and Go wrapper around rocksdb. The current supported version of rocksdb is 5.1.4.

Arch Linux

In Arch Linux, rocksdb is available under AUR.

$ yaourt rocksdb
# And then select, aur/rocksdb for installation. Choose version 5.1.4, if available. The current version is v4.4.1.

This would install RocksDB library in /usr/lib. Make sure that your LD_LIBRARY_PATH is correctly pointing to it.

# In ~/.bashrc
export LD_LIBRARY_PATH="/usr/lib"

Otherwise, you can also install rocksdb from sources.

$ sudo pacman -S make gflags snappy zlib bzip2 gcc
$ wget https://github.com/facebook/rocksdb/archive/v5.1.4.tar.gz
$ tar -xzvf v5.1.4.tar.gz
$ cd rocksdb-5.1.4
$ make shared_lib
$ sudo make install
$ export LD_LIBRARY_PATH=/usr/local/lib

Ubuntu

# First install dependencies.
# For Ubuntu, follow the ones below. For others, refer to INSTALL file in rocksdb.
$ sudo apt-get update && sudo apt-get install libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev
$ wget https://github.com/facebook/rocksdb/archive/v5.1.4.tar.gz
$ tar -xzvf v5.1.4.tar.gz
$ cd rocksdb-5.1.4
$ make shared_lib # if you don't have make, run "sudo apt-get install make"
$ sudo make install

This would install RocksDB library in /usr/local/lib. Make sure that your LD_LIBRARY_PATH is correctly pointing to it.

# In ~/.bashrc
export LD_LIBRARY_PATH="/usr/local/lib"

Mac

In Mac, rocksdb available via homebrew is not compatible with dgraph. However Homebrew can be used to install dependencies. Homebrew[11] needs to be installed on Mac before using the command below.

If you are a first time developer, run

$ xcode-select --install

After this install rocksdb like this

$ brew tap homebrew/versions
# To install gcc 4.7.
$ brew install gcc47 --use-llvm
# Install dependencies
$ brew install snappy lz4
# Fetch v5.1.4 of rocksdb from github.
$ curl -fLs https://github.com/facebook/rocksdb/archive/v5.1.4.tar.gz | tar zxv
$ cd rocksdb-5.1.4
$ make shared_lib
$ make install

This would install RocksDB library in /usr/local/lib and the headers in /usr/local/include. To build dgraph you need to set your LIBRARY_PATH and C_INCLUDE_PATH to include these locations.

# In ~/.bashrc
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib
export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include

Install Dgraph

go get -v github.com/dgraph-io/dgraph/...

# The above should install the following binaries in your $GOPATH/bin: dgraph and dgraphloader

# Install some dependencies which are not vendored in (because they cause issues if you are using the Goclient)
go get -u google.golang.org/grpc

# Optionally, you can also run tests.
cd $GOPATH/src/github.com/dgraph-io/dgraph
./test

Install statically linked Dgraph

We provide binaries which are statically linked with major libraries such as RocksDB. To build these binaries, run:

$ go install -tags=embed github.com/dgraph-io/dgraph/...

Other softwares

Git

# Ubuntu
$ sudo apt-get install git

# Arch Linux
$ sudo pacman -S git

# Mac
$ xcode-select --install

GCC and G++

You might also need to get C and C++ compiler.

# Ubuntu
$ sudo apt-get install gcc g++ make cmake

# Arch Linux
$ sudo pacman -S gcc make cmake

# Mac
$ xcode-select --install

Vim (Optional)

We prefer using Vim at Dgraph. In case you wish to try Vim and want some help getting started, follow these steps as necessary to enable auto-completion, gofmt, go-imports in vim.

  • Install Vim
# Ubuntu
$ sudo apt-get install vim vim-nox

# Arch Linux
$ sudo apt-get install gvim
Warning:Ensure you have lua support. Run :echo has("lua") from within Vim and ensure it returns 1.
  • Install Vundle plugin manager
$ git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim
  • You could use this sample vimrc file[13] from Github as your .vimrc or pick relevant parts of it and add it to your existing .vimrc
  • Start vim and run these inside vim:
 :PluginInstall
 :GoUpdateBinaries
  • Get neocomplete.vim
$ git clone https://github.com/Shougo/neocomplete.vim ~/.vim/neocomplete.vim

Protocol buffers

We use protocol buffers to serialize data between our server and the Go client and also for inter-worker communication. If you make any changes to the .proto files, you would have to recompile them. To recompile them you would need

1. Protoc compiler which is required for compiling proto files used for gRPC communication. Get protoc version 3.0.0 from GitHub release page[14].

2. We use gogo protobuf[15] in Dgraph. To get the protocol buffer compiler plugin from gogo run

# It is important that the compiler plugin be in your $PATH, go get puts it under $GOPATH/bin which would usually be in your $PATH.
$ go get -u github.com/gogo/protobuf/protoc-gen-gofast

3. To compile the proto file using the protoc plugin and the gogo compiler plugin run from within the directory containing the .proto file.

$ protoc --gofast_out=plugins=grpc:. *.proto

This should generate the required .pb.go file.

Signed Commits

Note:This is a recommended step for people who intend to contribute to Dgraph on a regular basis.

Signed commits help in verifying the authenticity of the contributor. We use signed commits in Dgraph, and we prefer it, though it's not compulsory to have signed commits.

Getting a GPG key

To get a new GPG key, you can follow the Github page on gpg-keys[16] or just run:

$ gpg --gen-key

and choose RSA key of length 4096 bits when asked for desired options through prompts.

Importing existing keys

To use already existing GPG keys:

$ gpg --import public.key
$ gpg --allow-secret-key-import --import private.key

Where public.key and private.key contain your public and private keys respectively.

Set up git for signing commits

Add the public key (new/existing) to your Github account if it's not already added by following Github help pages[17].

$ gpg --list-secret-keys --keyid-format LONG
/Users/hubot/.gnupg/secring.gpg
------------------------------------
sec   4096R/3AA5C34371567BD2 2016-03-10 [expires: 2017-03-10]
uid                          Hubot 
ssb   4096R/42B317FD4BA89E7A 2016-03-10

$ git config --global user.signingkey 3AA5C34371567BD2

# To set all commits for a repository to be signed by default, run 
$ git config commit.gpgsign true

AWS

In addition to desktops, Dgraph team uses Amazon EC2 instances for development and testing. To get access to AWS console, someone needs to create your AWS account. They will provide you with username, password, access key and secret access key. Once you have these details, you can log in at https://dgraph.signin.aws.amazon.com/console.

When you try to log in for the first time, you will be asked to change your password. The password has to satisfy these constraints:

  • At least 25 characters long.
  • At least one uppercase, one lowercase, one number, one non-alphanumeric character.
Tip:Use a password manager. Unix pass[18] is free, open source and based on GPG encryption.


Troubleshooting

rdb/rdbc.cc fatal error: 'rocksdb/cache.h' file not found

There are many possibilities here. One possibility is that you didn't have the right version of RocksDB. Do check above for the right version and install it and remember to do a make install for RocksDB.

Another possibility is that our C RocksDB code has to be re-linked against the RocksDB lib (maybe because it is a new version) but the Go compiler does not do it by default because it detects no changes in our C RocksDB code. To fix this, there are two ways. One way is to touch rdbc.cc by adding some dummy comment and rebuild. Another way is to force a full rebuild of the binaries, like so.

go build -a github.com/dgraph-io/dgraph/cmd/...
go install github.com/dgraph-io/dgraph/...

Code Review Workflow

  • All contributors need to sign the Contributor License Agreement(CLA)[19].
Note:When you send out a pull request, you'll automatically be asked to sign the CLA. You don't need to look for it.
  • Pick an issue with a Help wanted label[20].
  • If the issue isn't currently present in the list, talk to us on discuss.dgraph.io[21].
  • Create a new feature branch from the current master head, write your code, and commit your changes locally.
  • If you prefer working on a forked repository, follow the steps in Forking to setup your system.
  • All the dependencies have to be checked in using Govendor[22]
  • Run govendor add +e to add the external dependencies that are not added to the vendor directory before committing the change.
  • We suggest you sign your commits[23]. If you are a keybase user, you could Setup Keybase GPG key[24] to sign commits.
  • Make sure you run go test ./... from the root.
  • Create a pull request[25].
  • Dgraph uses Reviewable[26] for code reviews and follows a rigorous code review process.
  • Address the comments, and repeat the cycle, until you get a LGTM by someone qualified.
  • Once you have an LGTM, squash your commits, merge locally to master and push.
  • Most new contributors aren't allowed to merge themselves, in that case, we'll do it for you.
Warning:Do not use the Github UI to squash and merge the commits. If you do that, the commit created as a result of squashing won't be signed as Github doesn't have your private key. So squash and merge locally.
Also, Github won't auto close your PR because it checks that the base branch(master) should contain all the commits from the HEAD branch(feature), which won't be the case if commits are squashed. So you would have to close it manually.

Forking Dgraph

If for some reason, you decide to fork Dgraph then you should do the following steps so that you can continue contribute.

  • Fork Dgraph into your namespace using Github[27].
  • Follow the steps to install Dgraph manually.
  • Add your forked repo as a remote to the main Dgraph repo $GOPATH/src/github.com/dgraph-io/dgraph like
git remote add myfork https://github.com/myfork/dgraph.git
  • Now you can make your change and push to your fork and then create a PR to the main Dgraph repository.

For more details on why we do this, you can read this article[28]

Merging PR

To merge a PR, we recommend that you follow the steps below. Say you want to merge a branch feature/abc.

$ git checkout master # Switch to master.
$ git pull # Update master.

# Merge and push the feature branch.
$ git merge --squash feature/abc # Squash your feature branch into one commit to master.
$ git commit -S -m "Some message" # GPG sign your commit by passing -S flag, unless you have set gpgsign=true in your ~/.gitconfig.
$ git push origin master

# Delete the feature branch.
$ git branch -d feature/abc # Delete local.
$ git push origin --delete feature/abc # Delete remote.

By the way, if you want to avoid merge conflicts, in your local branch, you can regularly sync using git pull origin master.

Version releases

Warning:This section is only relevant if you're doing a Dgraph release. Do not follow this to set up dev environment.

Building Binaries

Linux and Mac

  • Ensure you have RocksDB installed by following these instructions: Beginners_Guide#Install_RocksDB.
  • Binaries can be built using the build.sh script located here[29].
  • The script above would generate a gzipped tar bundle and a checksum.md5 file which should be uploaded to Github while doing a release.

Steps to do a Release

  • Follow these to create a release branch.
$ git checkout -b release-<version> master
Switched to a new branch "release-<version>"
# Bump up the version and do some last-minute changes if necessary and commit them.
$ git push origin release-<version>
  • Do some testing in the release branch by rebuilding the binaries in the release branch to ensure correctness. Currently, we manually bring up the server with the Freebase data[30] loaded by dgraphloader, perform some queries and ensure correctness of the results returned.
  • Follow these steps to do a release.
# Tag the final commit on the release branch.
$ git tag -s -a <version>
$ git checkout master
$ git merge --no-ff release-<version>
# Push the master branch to the upstream repository along with the tags
$ git push origin master 
$ git push origin --tags
  • Do a release on github[31] with the details about the version.
  • Once the release tag is pushed to Github, a new entry shows up in the releases page with the corresponding name. This has to be filled in with information about changes (Improvements, Bugs, Features) that were made in this version.
  • Upload binaries and checksum files for 64 bit Mac and Linux.
  • Update the Docker image.
Note: Keep the release in Draft mode until all the steps are complete.

Publishing a blogpost

Note: This section is mostly for the use of the Dgraph team. If you as a user want to publish a post on our blog then let us know.

We use Hugo[32] for our blog. When you want to publish a new post

  • You should create a new post in the repo[33] and keep it in draft mode.
  • Once you are done with the post, share it on discourse and gather feedback from the team. After making the final changes and choosing a nice pic, you can push to master after bringing the post out of draft mode.
  • Note that just pushing to master won't deploy the post. The blog repo has a public folder (which has the auto-generated html, css, js files) and it is added as gitsubmodule to the main repo. It points to this [34] repo. To deploy you can run the deploy[35] script in the root like ./deploy "Blogpost on cmux". This would generate the static files and push them to the master branch of the dgraph-io.github.io repo. As we use Github Pages, this means that the website is deployed automatically on a push to master.

References

  1. https://github.com/dgraph-io/dgraph/issues
  2. https://www.mediawiki.org/wiki/Help:Formatting
  3. https://wiki.dgraph.io/Special:AllPages/Template:
  4. https://wiki.dgraph.io/Special:Upload
  5. https://en.wikipedia.org/wiki/Technical_debt
  6. https://github.com/golang/go/wiki/CodeReviewComments
  7. https://archlinux.fr/yaourt-en
  8. https://golang.org/dl/
  9. https://golang.org/doc/code.html
  10. https://golang.org/doc/install#testing
  11. 11.0 11.1 http://brew.sh/
  12. https://github.com/facebook/rocksdb
  13. https://gist.github.com/ashwin95r/c12807427553e72d19c081bd7eabb804
  14. https://github.com/google/protobuf/releases
  15. https://github.com/gogo/protobuf
  16. https://help.github.com/articles/generating-a-new-gpg-key/
  17. https://help.github.com/articles/adding-a-new-gpg-key-to-your-github-account/
  18. https://www.passwordstore.org/
  19. https://cla-assistant.io/dgraph-io/dgraph
  20. https://github.com/dgraph-io/dgraph/labels/help_wanted
  21. https://discuss.dgraph.io
  22. "Govendor"
  23. https://help.github.com/articles/signing-commits-using-gpg/
  24. https://www.ahmadnassri.com/blog/github-gpg-keybase-pgp/
  25. https://help.github.com/articles/creating-a-pull-request/
  26. https://reviewable.io/
  27. https://github.com/dgraph-io/dgraph#fork-destination-box
  28. http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html
  29. https://github.com/dgraph-io/dgraph/blob/develop/build.sh
  30. https://github.com/dgraph-io/benchmarks/blob/master/data/22million.rdf.gz
  31. https://help.github.com/articles/creating-releases/
  32. https://github.com/spf13/hugo
  33. https://github.com/dgraph-io/open
  34. https://github.com/dgraph-io/dgraph-io.github.io
  35. https://github.com/dgraph-io/open/blob/master/deploy.sh