- Make the code more GitHub-specific
- Requires mirror hash to work with .gitattributes
- Use different API depending on whether PKG_SOURCE_VERSION is a
complete commit id or other ref types like tags
- Fix removing symbolic link
- pre-clean dir_untar for possible leftovers from previous run
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
Seems like the python download.py dl_method call causes serious
performance regression for fresh "make defconfig" as reported in
FS#1621. GitHub tarball download will also be disabled with this
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
A new python script scripts/download.py is added to fetch tarballs using
GitHub archive API [1], then repack in a reproducible way same as the
current DownloadMethod/git
GitHub imposes a 60 reqs/hour rate limit on unauthenticated API
access[2]. This affects fetching commit date for feeding tar --mtime=
argument. However, observation indicates that archive download is NOT
subject to this limit at the moment. In the rare cases where download
fails because of this, we will falback to using DownloadMethod/git
The missing piece in the GitHub API is that it cannot provide in the
tarball dependent submodules's source code. In that case, the
implementation will also fallback to using DownloadMethod/git
[1] Get archive link, https://developer.github.com/v3/repos/contents/#get-archive-link
[2] Rate limiting, https://developer.github.com/v3/#rate-limiting
v2 <- v1:
- allow passing multiple urls with --urls argument
- add commit ts cache. can be helpful on retry
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
Introduce a name-agnostic PROJECT_GIT variable poiting to
https://git.openwrt.org/ and declare LEDE_GIT and OPENWRT_GIT
as aliases to it.
After some transition time we can drop this alias variables.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Since $(DownloadMethod/unknown) is being invoked in the expansion of
$(call locked ...) anyway, you can't have an @ because the shell
doesn't know what to do with it.
Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>
This makes it easier to unify versioning of git based package downloads.
PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash
is used as PKG_VERSION, PKG_RELEASE should be used like normal packages.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This will attempt to automatically fix common mistakes like using MD5
instead of SHA256, using the MD5SUM variable instead of HASH, or even a
missing mirror file hash.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is intended to be used for a wide array of package sanity checks.
The first check that is implemented is for the hash of downloaded files.
It checks:
- Missing hash
- Use of SHA256 instead of MD5
- dl/<file> hash not matching hash in makefile
- deprecated MD5SUM variable
The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Since we've switched to preferring SHA256 over MD5, the old variable
name is misleading. Packages using the old name remain compatible.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This is going to be used to migrate the hand rolled git clone for the kernel
into using the git download method. The kernel uses custom options that we may
have to pass down.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Adds a slightly higher compression level to xz by default which roughly raises memory usage from 100MiB to about 200MiB during compression, about 10MiB for decompression. (Source: xz manpage)
Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
Apply a number of changes to the tarball generation in order to produce
identical files on different systems:
1) Use an explicit `gzip -cn` to avoid storing file mtime in the gzip header
2) Instruct `tar` to unconditionally use uid and gid 0 for archive members
3) Instruct `tar` to sort archive members by file name
4) For SCMs that do not preserve file modification times like Git or Mercurial,
use the date of the last commit to the repository and pass it as `--mtime`
value to `tar`
After these changes, locally produced tarballs generated from SCM checkouts
should be identical on any system, simplifying the mirroring of cache archives.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Some Git versions have issues following the HTTP->HTTPS redirect and since
the keyring package is fetched from this host, switching to HTTPS is a
sensible choice anyway.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Move the `--recursive` switch from `git clone` to `git submodule`
so that submodules are cloned for upstream branches where the
PKG_SOURCE_VERSION commit-ish has a different .gitmodules
configuration than the repository default.
This is, for example, required when the master branch for a source
package does not use submodules, but its topic branch for OpenWRT
does.
This changes the buildroot dependency from git-1.6.2 to git 1.7.12.2,
which was released September 2012.
Signed-off-by: Darik Horn <dajhorn@vanadac.com>
Signed-off-by: Karl Palsson <karlp@tweak.net.au>
SVN-Revision: 48830
Changeset r48416 broke the downloading of mirrored, packed scm checkouts.
Fix this by removing the "@" sign in front of the download command which is
now executed as part of a larger shell command under flock.
Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>
SVN-Revision: 48733
The Apache Software Foundation offers diverse download mirros.
For packaging Apache software a new alias @APACHE is defined.
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
SVN-Revision: 48270
I defined a new download method @SAVANNAH in include/download.mk and scripts/download.pl,
and converted quilt and qemu to use that method.
Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
SVN-Revision: 42840
When checking out git packages, buildroot doesn't seem to track the revisions
correctly of any submodules referenced by that project. As a result, the
submodule stays at whatever revision was referenced by the head of the master
branch. Running a 'git submodule update' after the checkout fixes this problem.
Signed-off-by: Owen Kirby <osk@exegin.com>
SVN-Revision: 38359
This patch adds support for darcs repositories (as sources of packages).
It does *not* add support for darcs:// URI scheme because such a scheme
do not exist (AFAIK). You must therefore manually set PKG_SOURCE_PROTO
to darcs in your Makefile (and use a regular http:// URI). You also have
to set PKG_SOURCE_VERSION to a string matching a tag contained in the
repository, and PKG_SOURCE_SUBDIR to something sensible
($(PACKAGE_NAME)-$(PACKAGE_SOURCE_VERSION) for instance).
Same rationale as for the previous patch ("useless to most, but small
and straightforward so why no add it?).
Best regards,
Signed-off-by: Gabriel Kerneis <kerneis@pps.jussieu.fr>
SVN-Revision: 23615
Some packages sources need to be checked out from server with unknown
certificates, silently accept by default in order not to stop builds, thanks rhk.
SVN-Revision: 20299
