Connectors · Data sources

GitHub repository metadata

GitHub: public repositories' metadata.

In place

Anyone can contribute. Improve this connector Contribute a connector

Leaves your machineQueries onlyWhat you ask GitHub for: the data, places, periods or search words of each load. The key goes with the requests.

You needAn optional keyOptional: a GitHub personal access token (no scopes needed for public data) raises the limits from 60 to 5,000 requests an hour; create one at https://github.com/settings/tokens and set GITHUB_TOKEN.

DirectionBrings in

What is in it

Public GitHub repositories as research objects: search by words, topic, language, stars and creation date (up to 1,000 per search), and per repository its metadata (stars, forks, issues, licence, topics, languages, dates), contributor count and releases. Open-source ecosystems, software adoption, research software. Metadata only. Keyless (60 requests an hour); GITHUB_TOKEN raises the limits.

In e2er's words, from the source's definition.

github.com

The data stay with the source

The terms of GitHub grant no licence to redistribute the repository metadata. A published study does not pass these data on: publishing asks the researcher to confirm the terms, and the study's code loads the data again from the source (get_data.py). A reader who asks the authors for the data is sent to the source.

GitHub's Acceptable Use Policies (section 7, Information Usage Restrictions) allow research use of public non-personal information only if the resulting publications are open access, so the paper must be public. e2er publishes a study that used these data only with its paper public.

Terms of use for the data

Research use of public, non-personal information is allowed only if the publications are open access.
Personal information may be used only for the purpose its user authorized; e2er loads no e-mail addresses or contributor identities.
Repository contents belong to their owners under their own licences; e2er loads metadata only.
Respect GitHub's rate limits; do not share tokens to exceed them.

Full terms · docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies

The terms as e2er records them

GitHub Acceptable Use Policies, section 7 (https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies): "You may use information from our Service for the following reasons, regardless of whether the information was scraped, collected through our API, or obtained otherwise: […] Researchers may use public, non-personal information from the Service for research purposes, only if any publications resulting from that research are open access." "If you collect any personal information from the Service, you agree that you will only use that personal information for the purpose for which that User has authorized it." GitHub Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service), section H: "All use of the GitHub API is subject to these Terms of Service and the GitHub Privacy Statement." "You may not share API tokens to exceed GitHub's rate limitations." Section D: "By making a repository public, you grant other Users a nonexclusive, worldwide license to use, display, perform and reproduce (by forking) Your Content through the Service as permitted by GitHub's functionality. You may grant additional rights by adopting a license."

Listed with the open datasets: GitHub public repository metadata (REST API), GitHub, Inc.

How to cite

GitHub, Inc. GitHub REST API (version 2022-11-28). https://docs.github.com/en/rest.

The source publishes no citation format; this is the citation e2er writes.

Where it lives

Built into e2er.

Templates that can use it

Descriptive data study Empirical study Empirical study, pre-registered Event study in finance Policy evaluation (difference-in-differences)Spatial analysis Text analysis Time series and forecasting

Technical details

Setting: GITHUB_TOKEN

Code in e2er: src/modules/data/sources/github.py

e2er-data github repos: Search repositories, e.g. --topic machine-learning --language Python --min-stars 100 --created-after 2020-01-01 --sort stars --limit 200.
e2er-data github repo: Metadata of named repositories, with languages and contributors count, e.g. --repos pandas-dev/pandas,numpy/numpy.
e2er-data github releases: Releases of one repository, e.g. --repo pandas-dev/pandas.

Description

Data model

Discipline: General
Method family: not specified
Design: not specified
Research stage: Data acquisition
Authors: e2er authors (Software, Methodology)
Usage: not used in published research yet · templates that can use it: 8
Source: e2er repository · src/modules/data/sources/github.py · @d8fd8ab
Record: connector:github-data · JSON

Solid tags are declared by the source or mapped from its terms; dashed tags are inferred by a published rule. Hover a tag for its provenance.

On e2er.org

Loading…

Report a failure Propose a new version

Discussion