Commit graph

64 commits

Author SHA1 Message Date
Yousong Zhou
9009efa18b download.mk: enable DownloadMethod/github_archive
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-07-05 01:30:57 +08:00
Yousong Zhou
04b9f85873 scripts/dl_github_archive.py: rename from download.py
- Make the code more GitHub-specific
 - Requires mirror hash to work with .gitattributes
 - Use different API depending on whether PKG_SOURCE_VERSION is a
   complete commit id or other ref types like tags
 - Fix removing symbolic link
 - pre-clean dir_untar for possible leftovers from previous run

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-07-05 01:30:57 +08:00
Yousong Zhou
e48ea13b3b download.mk: add more comments
Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-07-05 01:30:57 +08:00
Yousong Zhou
e15565a01c download.mk: restore the old dl_method implementation
Seems like the python download.py dl_method call causes serious
performance regression for fresh "make defconfig" as reported in
FS#1621.  GitHub tarball download will also be disabled with this

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-06-29 13:54:17 +08:00
Yousong Zhou
75ab064d2b build: download code from github using archive API
A new python script scripts/download.py is added to fetch tarballs using
GitHub archive API [1], then repack in a reproducible way same as the
current DownloadMethod/git

GitHub imposes a 60 reqs/hour rate limit on unauthenticated API
access[2].  This affects fetching commit date for feeding tar --mtime=
argument.  However, observation indicates that archive download is NOT
subject to this limit at the moment.  In the rare cases where download
fails because of this, we will falback to using DownloadMethod/git

The missing piece in the GitHub API is that it cannot provide in the
tarball dependent submodules's source code.  In that case, the
implementation will also fallback to using DownloadMethod/git

 [1] Get archive link, https://developer.github.com/v3/repos/contents/#get-archive-link
 [2] Rate limiting, https://developer.github.com/v3/#rate-limiting

v2 <- v1:

 - allow passing multiple urls with --urls argument
 - add commit ts cache.  can be helpful on retry

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
2018-06-27 10:51:27 +08:00
Jo-Philipp Wich
4700544e40 downloads.mk: introduce name-agnostic PROJECT_GIT variable
Introduce a name-agnostic PROJECT_GIT variable poiting to
https://git.openwrt.org/ and declare LEDE_GIT and OPENWRT_GIT
as aliases to it.

After some transition time we can drop this alias variables.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2018-01-10 21:27:31 +01:00
Philip Prindeville
76ba01a392 build: remove @ as it's causing an error
Since $(DownloadMethod/unknown) is being invoked in the expansion of
$(call locked ...) anyway, you can't have an @ because the shell
doesn't know what to do with it.

Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>
2017-09-20 08:50:49 +02:00
Felix Fietkau
5919cc2dc4 build: let make check warn about use of legacy PKG_MD5SUM variable in feeds
The variable rename change has been merged in OpenWrt now, so it's
possible to convert the feeds as well.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-13 10:23:43 +01:00
Felix Fietkau
84bd74057f build: use mkhash to replace various quirky md5sum/openssl calls
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2017-01-05 11:09:12 +01:00
Felix Fietkau
ffb0181a87 build: add defaults for PKG_SOURCE, PKG_SOURCE_SUBDIR, PKG_VERSION
This makes it easier to unify versioning of git based package downloads.
PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash
is used as PKG_VERSION, PKG_RELEASE should be used like normal packages.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-22 16:42:20 +01:00
Felix Fietkau
d49059693b build: add FIXUP option for make check
This will attempt to automatically fix common mistakes like using MD5
instead of SHA256, using the MD5SUM variable instead of HASH, or even a
missing mirror file hash.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-17 10:36:45 +01:00
Felix Fietkau
7a315b0b5d build: implement make check and make package/X/check
This is intended to be used for a wide array of package sanity checks.

The first check that is implemented is for the hash of downloaded files.
It checks:
  - Missing hash
  - Use of SHA256 instead of MD5
  - dl/<file> hash not matching hash in makefile
  - deprecated MD5SUM variable

The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt
is updated as well

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-17 10:36:25 +01:00
Felix Fietkau
6a5cc2d085 include/package.mk: sync default value for hash fallback with mirror hash
Simplifies dealing with PKG_HASH vs PKG_MD5SUM

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-16 15:40:07 +01:00
Felix Fietkau
881c5b47ec build: remove duplicate Download/default definition from include/host-build.mk
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-16 15:35:21 +01:00
Felix Fietkau
7416d2e046 build: replace MD5SUM variables with HASH
Since we've switched to preferring SHA256 over MD5, the old variable
name is misleading. Packages using the old name remain compatible.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-16 13:49:24 +01:00
Felix Fietkau
1b9a39c528 download.mk: improve download tarball reproducibility
Store only numeric user/group id. Group 0 is 'root' on linux and 'wheel' on
BSD and Mac OS X.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-13 12:16:26 +01:00
Felix Fietkau
19d3b78304 download.mk: remove code duplication in $(TAR) call
Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-13 12:16:26 +01:00
Felix Fietkau
dbbfd41118 download.mk: use $(error) instead of a regular shell error
Useful for further rework

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2016-12-13 12:16:25 +01:00
Florian Fainelli
b9aab34eb4 include/download.mk: Allow specify DownloadMethod specific options
This is going to be used to migrate the hand rolled git clone for the kernel
into using the git download method. The kernel uses custom options that we may
have to pass down.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
2016-12-04 11:41:51 +01:00
Daniel Engberg
cbc80805bd include/download.mk: Use -7e compression instead of -6 by default
Adds a slightly higher compression level to xz by default which roughly raises memory usage from 100MiB to about 200MiB during compression, about 10MiB for decompression. (Source: xz manpage)

Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
2016-10-05 15:15:19 +02:00
Jo-Philipp Wich
5d86dc791e include/download.mk: generate reproducable SCM tarballs
Apply a number of changes to the tarball generation in order to produce
identical files on different systems:

1) Use an explicit `gzip -cn` to avoid storing file mtime in the gzip header
2) Instruct `tar` to unconditionally use uid and gid 0 for archive members
3) Instruct `tar` to sort archive members by file name
4) For SCMs that do not preserve file modification times like Git or Mercurial,
   use the date of the last commit to the repository and pass it as `--mtime`
   value to `tar`

After these changes, locally produced tarballs generated from SCM checkouts
should be identical on any system, simplifying the mirroring of cache archives.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-10-05 00:45:59 +02:00
Jo-Philipp Wich
3a085dcbe3 download.mk: use HTTPS for git.lede-project.org
Some Git versions have issues following the HTTP->HTTPS redirect and since
the keyring package is fetched from this host, switching to HTTPS is a
sensible choice anyway.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
2016-06-01 15:18:42 +02:00
John Crispin
f5c4d963ff include: add lede git server url
Signed-off-by: John Crispin <john@phrozen.org>
2016-04-29 04:15:24 +02:00
Álvaro Fernández Rojas
694f060550 download: add @GITHUB download facility
Define a new alias (@GITHUB) for downloading raw files from github repos.

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
2016-04-03 22:02:46 +02:00
John Crispin
5c408ede6c download: add @GITHUB download facility
Define a new alias (@GITHUB) for downloading raw github repository files

Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>

SVN-Revision: 49138
2016-04-09 10:25:34 +00:00
Felix Fietkau
59121fdd69 buildroot: improve git submodule handling for packages
Move the `--recursive` switch from `git clone` to `git submodule`
so that submodules are cloned for upstream branches where the
PKG_SOURCE_VERSION commit-ish has a different .gitmodules
configuration than the repository default.

This is, for example, required when the master branch for a source
package does not use submodules, but its topic branch for OpenWRT
does.

This changes the buildroot dependency from git-1.6.2 to git 1.7.12.2,
which was released September 2012.

Signed-off-by: Darik Horn <dajhorn@vanadac.com>
Signed-off-by: Karl Palsson <karlp@tweak.net.au>

SVN-Revision: 48830
2016-02-29 20:12:25 +00:00
Jo-Philipp Wich
ac5b226735 download.mk: fix packed checkout mirroring support
Changeset r48416  broke the downloading of mirrored, packed scm checkouts.

Fix this by removing the "@" sign in front of the download command which is
now executed as part of a larger shell command under flock.

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

SVN-Revision: 48733
2016-02-17 08:44:37 +00:00
Felix Fietkau
c5db626ac5 build: download.mk: bugfix download.pl argument call order
Change wrong download.pl argument call order.
See download.pl argument list.

Signed-off-by: Florian Eckert <Eckert.Florian@googlemail.com>

SVN-Revision: 48427
2016-01-21 13:28:18 +00:00
Felix Fietkau
a81ad971df build: add locking for downloads (fixes race conditions with multiple variants)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 48416
2016-01-20 21:31:50 +00:00
Felix Fietkau
883b5b8191 build: add @APACHE download facility
The Apache Software Foundation offers diverse download mirros.

For packaging Apache software a new alias @APACHE is defined.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>

SVN-Revision: 48270
2016-01-17 10:47:32 +00:00
Felix Fietkau
11b7bc593c build: add a variable pointing to the main openwrt git repositories (useful if we want to support using a mirror later)
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 48117
2016-01-04 15:11:43 +00:00
Felix Fietkau
9b68fe961b build: add support for choosing a different url filename part than the output file
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 47591
2015-11-22 19:06:33 +00:00
John Crispin
b0a1c7dc0d include/download.mk: Add download mirrors for tools from GNU Savannah (bug #15184)
I defined a new download method @SAVANNAH in include/download.mk and scripts/download.pl,
and converted quilt and qemu to use that method.

Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>

SVN-Revision: 42840
2014-10-08 08:01:39 +00:00
Felix Fietkau
d1969d761f build: add support for packing version control downloads with xz
Signed-off-by: Felix Fietkau <nbd@openwrt.org>

SVN-Revision: 40446
2014-04-10 14:43:26 +00:00
Jo-Philipp Wich
97fc11af79 Incorrect checkout of git submodules
When checking out git packages, buildroot doesn't seem to track the revisions
correctly of any submodules referenced by that project. As a result, the
submodule stays at whatever revision was referenced by the head of the master
branch. Running a 'git submodule update' after the checkout fixes this problem.

Signed-off-by: Owen Kirby <osk@exegin.com>

SVN-Revision: 38359
2013-10-10 15:01:50 +00:00
Jo-Philipp Wich
5d253542d7 include: download.mk: correct tar argument order when packing checkouts
GNU TAR v1.26 will fail with "no such file or directory" errors when the
output file does not immediately follow the -f switch.

SVN-Revision: 34460
2012-12-02 18:35:47 +00:00
Felix Fietkau
e8ff034c8b download.mk: refuse to download mirror tarballs from packages using version control if no mirror md5sum was provided
SVN-Revision: 32079
2012-06-06 13:45:53 +00:00
Jens Muecke
6204553450 Enable recursive download of git sources.
SVN-Revision: 30967
2012-03-17 21:33:13 +00:00
Felix Fietkau
b568a64f8c add support for md5sum checks for mirrored tarballs of packages with version control source urls
SVN-Revision: 26260
2011-03-21 23:02:06 +00:00
Felix Fietkau
fd0deca65d download.mk: properly quote download urls
SVN-Revision: 26129
2011-03-13 18:38:43 +00:00
Jo-Philipp Wich
8fae8c0a6f download.mk: support https:// downloads
SVN-Revision: 24688
2010-12-18 21:43:10 +00:00
Florian Fainelli
d78a574019 add support for darcs repositories
This patch adds support for darcs repositories (as sources of packages).

It does *not* add support for darcs:// URI scheme because such a scheme
do not exist (AFAIK).  You must therefore manually set PKG_SOURCE_PROTO
to darcs in your Makefile (and use a regular http:// URI). You also have
to set PKG_SOURCE_VERSION to a string matching a tag contained in the
repository, and PKG_SOURCE_SUBDIR to something sensible
($(PACKAGE_NAME)-$(PACKAGE_SOURCE_VERSION) for instance).

Same rationale as for the previous patch ("useless to most, but small
and straightforward so why no add it?).

Best regards,

Signed-off-by: Gabriel Kerneis <kerneis@pps.jussieu.fr>

SVN-Revision: 23615
2010-10-24 12:36:44 +00:00
Nicolas Thill
4ba18f5916 download.mk: fix bzr download method
SVN-Revision: 20582
2010-03-29 13:06:19 +00:00
Jo-Philipp Wich
40781b7290 the svn fallback code added in r20322 is evaluated with each make invocation and produces a lot of spurious "svn: broken pipe" error - replace it with a more efficient version
SVN-Revision: 20377
2010-03-23 01:21:50 +00:00
Florian Fainelli
b195af191b fix svn download with older svn client versions
SVN-Revision: 20322
2010-03-20 17:18:47 +00:00
Florian Fainelli
08a790efb6 accept unknown server certificates by default on svn download
Some packages sources need to be checked out from server with unknown
certificates, silently accept by default in order not to stop builds, thanks rhk.

SVN-Revision: 20299
2010-03-19 13:44:47 +00:00
Travis Kemen
db7082d899 Updated post r19769, integrates the missing dl_method bzr hunk from #6370, uses the export subcommand for the vcs' that provide a usable version (hg archive doesn't work remotely, git archive isn't enabled on github), from #6700
SVN-Revision: 20222
2010-03-14 21:55:11 +00:00
Travis Kemen
f04ad5bb5f this patch adds support for bzr in include/download.mk, and fixes whitespace.
-Raphael

SVN-Revision: 19769
2010-02-20 04:48:07 +00:00
Jo-Philipp Wich
4d904cf081 allow download urls using the file:// schema (#6480)
SVN-Revision: 19120
2010-01-13 19:38:53 +00:00
Felix Fietkau
510bdd3f0e fix download reverse depends
SVN-Revision: 14907
2009-03-17 02:37:10 +00:00