| Message ID | 20230927165024.2342635-1-jstephan@baylibre.com |
|---|---|
| State | New |
| Headers | show |
| Series | devtool: modify: correctly handle multiple sources in SRC_URI | expand |
On Wed, 2023-09-27 at 18:50 +0200, Julien Stephan wrote: > [YOCTO #15162] > > When recipes such as bzip2 have several sources in SRC_URI, all the > sources are correctly extracted into devtool temporary workdir, but > currently, we only get back the primary source and all files declared > using "files://", then the devtool temporary workdir is deleted > > Fix this by moving all remaining files from devtool temporary workdir > into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", > and "recipe-sysroot-native") before deleting it > > Here is the stack trace from devtool modify/build bzip2: > > NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2 > ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None) > ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368 > Log data follows: > > DEBUG: Executing shell function do_install_ptest_base > > NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest > > sed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ > > ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile > > cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2 > > cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory > > WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/ > 1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress' > > WARNING: Backtrace (BB generated script): > > #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 > > #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 > > #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226 > ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1' > NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed. > > Summary: 1 task failed: > <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base > > Signed-off-by: Julien Stephan <jstephan@baylibre.com> > --- > scripts/lib/devtool/standard.py | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py > index d53fb810071..6c8ad7a6c8c 100644 > --- a/scripts/lib/devtool/standard.py > +++ b/scripts/lib/devtool/standard.py > @@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works > shutil.move(srcsubdir, srctree) > symlink_oelocal_files_srctree(d,srctree) > > + # In case SRC_URI contains multiple sources > + # they are extracted into the devtool temporary workdir > + # We need to move them into WORKDIR, otherwise they will be lost > + for path in os.listdir(os.path.join(tempdir, 'workdir')): > + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: > + workdir = d.getVar('WORKDIR') > + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) > + > if is_kernel_yocto: > logger.info('Copying kernel config to srctree') > shutil.copy2(os.path.join(tempdir, '.config'), srctree) Looks like a good start although I worry a bit about the hardcoded list of exceptions. Is it possible to add a test case to the devtool testsuite so this can't regress in future? Thanks, Richard
Le mer. 27 sept. 2023 à 22:36, Richard Purdie <richard.purdie@linuxfoundation.org> a écrit : > > On Wed, 2023-09-27 at 18:50 +0200, Julien Stephan wrote: > > [YOCTO #15162] > > > > When recipes such as bzip2 have several sources in SRC_URI, all the > > sources are correctly extracted into devtool temporary workdir, but > > currently, we only get back the primary source and all files declared > > using "files://", then the devtool temporary workdir is deleted > > > > Fix this by moving all remaining files from devtool temporary workdir > > into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", > > and "recipe-sysroot-native") before deleting it > > > > Here is the stack trace from devtool modify/build bzip2: > > > > NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2 > > ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None) > > ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368 > > Log data follows: > > > DEBUG: Executing shell function do_install_ptest_base > > > NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest > > > sed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ > > > ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile > > > cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2 > > > cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory > > > WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/ > > 1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress' > > > WARNING: Backtrace (BB generated script): > > > #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 > > > #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 > > > #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226 > > ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1' > > NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed. > > > > Summary: 1 task failed: > > <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base > > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com> > > --- > > scripts/lib/devtool/standard.py | 8 ++++++++ > > 1 file changed, 8 insertions(+) > > > > diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py > > index d53fb810071..6c8ad7a6c8c 100644 > > --- a/scripts/lib/devtool/standard.py > > +++ b/scripts/lib/devtool/standard.py > > @@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works > > shutil.move(srcsubdir, srctree) > > symlink_oelocal_files_srctree(d,srctree) > > > > + # In case SRC_URI contains multiple sources > > + # they are extracted into the devtool temporary workdir > > + # We need to move them into WORKDIR, otherwise they will be lost > > + for path in os.listdir(os.path.join(tempdir, 'workdir')): > > + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: > > + workdir = d.getVar('WORKDIR') > > + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) > > + > > if is_kernel_yocto: > > logger.info('Copying kernel config to srctree') > > shutil.copy2(os.path.join(tempdir, '.config'), srctree) > > Looks like a good start although I worry a bit about the hardcoded list > of exceptions. Hi Richard, Yes, I am not very happy with listing files but I don't know if I can easily get the list of secondary sources? Here are the alternatives I can see: * encapsulate the move into a try/catch block and ignore exceptions if the destination file/directory already exists * retrieve SRC_URI and manually build the list of secondary sources to move. This looks a little bit overkill here... What do you think? > > Is it possible to add a test case to the devtool testsuite so this > can't regress in future? Sure, I'll do it! Cheers Juliem > sure I will add a test for that. > Thanks, > > Richard
Le sam. 30 sept. 2023 à 12:01, Julien Stephan via lists.openembedded.org <jstephan=baylibre.com@lists.openembedded.org> a écrit : > > Le mer. 27 sept. 2023 à 22:36, Richard Purdie > <richard.purdie@linuxfoundation.org> a écrit : > > > > On Wed, 2023-09-27 at 18:50 +0200, Julien Stephan wrote: > > > [YOCTO #15162] > > > > > > When recipes such as bzip2 have several sources in SRC_URI, all the > > > sources are correctly extracted into devtool temporary workdir, but > > > currently, we only get back the primary source and all files declared > > > using "files://", then the devtool temporary workdir is deleted > > > > > > Fix this by moving all remaining files from devtool temporary workdir > > > into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", > > > and "recipe-sysroot-native") before deleting it > > > > > > Here is the stack trace from devtool modify/build bzip2: > > > > > > NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2 > > > ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None) > > > ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368 > > > Log data follows: > > > > DEBUG: Executing shell function do_install_ptest_base > > > > NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest > > > > sed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ > > > > ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile > > > > cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2 > > > > cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory > > > > WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/ > > > 1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress' > > > > WARNING: Backtrace (BB generated script): > > > > #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 > > > > #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 > > > > #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226 > > > ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1' > > > NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed. > > > > > > Summary: 1 task failed: > > > <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base > > > > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com> > > > --- > > > scripts/lib/devtool/standard.py | 8 ++++++++ > > > 1 file changed, 8 insertions(+) > > > > > > diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py > > > index d53fb810071..6c8ad7a6c8c 100644 > > > --- a/scripts/lib/devtool/standard.py > > > +++ b/scripts/lib/devtool/standard.py > > > @@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works > > > shutil.move(srcsubdir, srctree) > > > symlink_oelocal_files_srctree(d,srctree) > > > > > > + # In case SRC_URI contains multiple sources > > > + # they are extracted into the devtool temporary workdir > > > + # We need to move them into WORKDIR, otherwise they will be lost > > > + for path in os.listdir(os.path.join(tempdir, 'workdir')): > > > + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: > > > + workdir = d.getVar('WORKDIR') > > > + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) > > > + > > > if is_kernel_yocto: > > > logger.info('Copying kernel config to srctree') > > > shutil.copy2(os.path.join(tempdir, '.config'), srctree) > > > > Looks like a good start although I worry a bit about the hardcoded list > > of exceptions. > > > Hi Richard, > Yes, I am not very happy with listing files but I don't know if I can > easily get the list of secondary sources? > > Here are the alternatives I can see: > * encapsulate the move into a try/catch block and ignore exceptions if > the destination file/directory already exists > * retrieve SRC_URI and manually build the list of secondary sources to > move. This looks a little bit overkill here... > What do you think? > Hi Richard, I need to rework this patch. As is it doesn't work: if the recipe was already built then the sources I am trying to move during devtool modify from the devtool temporary workdir to WORKDIR already exist in WORKDIR. This makes several tests from devtool selftest to fail. Any suggestions on how I should handle this? Cheers Julien > > > > Is it possible to add a test case to the devtool testsuite so this > > can't regress in future? > > Sure, I'll do it! > > Cheers > Juliem > > > > > sure I will add a test for that. > > > > Thanks, > > > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#188455): https://lists.openembedded.org/g/openembedded-core/message/188455 > Mute This Topic: https://lists.openembedded.org/mt/101621289/4356723 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jstephan@baylibre.com] > -=-=-=-=-=-=-=-=-=-=-=- >
On Fri, 2023-10-06 at 12:34 +0200, Julien Stephan wrote: > Le sam. 30 sept. 2023 à 12:01, Julien Stephan via > lists.openembedded.org <jstephan=baylibre.com@lists.openembedded.org> > a écrit : > > > > Le mer. 27 sept. 2023 à 22:36, Richard Purdie > > <richard.purdie@linuxfoundation.org> a écrit : > > > > > > On Wed, 2023-09-27 at 18:50 +0200, Julien Stephan wrote: > > > > [YOCTO #15162] > > > > > > > > When recipes such as bzip2 have several sources in SRC_URI, all the > > > > sources are correctly extracted into devtool temporary workdir, but > > > > currently, we only get back the primary source and all files declared > > > > using "files://", then the devtool temporary workdir is deleted > > > > > > > > Fix this by moving all remaining files from devtool temporary workdir > > > > into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", > > > > and "recipe-sysroot-native") before deleting it > > > > > > > > Here is the stack trace from devtool modify/build bzip2: > > > > > > > > NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2 > > > > ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None) > > > > ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368 > > > > Log data follows: > > > > > DEBUG: Executing shell function do_install_ptest_base > > > > > NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest > > > > > sed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ > > > > > ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile > > > > > cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2 > > > > > cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory > > > > > WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/ > > > > 1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress' > > > > > WARNING: Backtrace (BB generated script): > > > > > #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 > > > > > #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 > > > > > #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226 > > > > ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1' > > > > NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed. > > > > > > > > Summary: 1 task failed: > > > > <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base > > > > > > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com> > > > > --- > > > > scripts/lib/devtool/standard.py | 8 ++++++++ > > > > 1 file changed, 8 insertions(+) > > > > > > > > diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py > > > > index d53fb810071..6c8ad7a6c8c 100644 > > > > --- a/scripts/lib/devtool/standard.py > > > > +++ b/scripts/lib/devtool/standard.py > > > > @@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works > > > > shutil.move(srcsubdir, srctree) > > > > symlink_oelocal_files_srctree(d,srctree) > > > > > > > > + # In case SRC_URI contains multiple sources > > > > + # they are extracted into the devtool temporary workdir > > > > + # We need to move them into WORKDIR, otherwise they will be lost > > > > + for path in os.listdir(os.path.join(tempdir, 'workdir')): > > > > + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: > > > > + workdir = d.getVar('WORKDIR') > > > > + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) > > > > + > > > > if is_kernel_yocto: > > > > logger.info('Copying kernel config to srctree') > > > > shutil.copy2(os.path.join(tempdir, '.config'), srctree) > > > > > > Looks like a good start although I worry a bit about the hardcoded list > > > of exceptions. > > > > > > Hi Richard, > > Yes, I am not very happy with listing files but I don't know if I can > > easily get the list of secondary sources? > > > > Here are the alternatives I can see: > > * encapsulate the move into a try/catch block and ignore exceptions if > > the destination file/directory already exists > > * retrieve SRC_URI and manually build the list of secondary sources to > > move. This looks a little bit overkill here... > > What do you think? > > I need to rework this patch. As is it doesn't work: if the recipe was > already built then the sources I am trying to move during devtool > modify from the devtool temporary workdir to WORKDIR already exist in > WORKDIR. > This makes several tests from devtool selftest to fail. > > Any suggestions on how I should handle this? There is some work on the bitbake list from Alberto which adds unpack tracing. I'm wondering if by default we should track some really basic form of unpack accesses which would then give you a list of target locations to merge? Cheers, Richard
Le ven. 6 oct. 2023 à 12:38, Richard Purdie <richard.purdie@linuxfoundation.org> a écrit : > > On Fri, 2023-10-06 at 12:34 +0200, Julien Stephan wrote: > > Le sam. 30 sept. 2023 à 12:01, Julien Stephan via > > lists.openembedded.org <jstephan=baylibre.com@lists.openembedded.org> > > a écrit : > > > > > > Le mer. 27 sept. 2023 à 22:36, Richard Purdie > > > <richard.purdie@linuxfoundation.org> a écrit : > > > > > > > > On Wed, 2023-09-27 at 18:50 +0200, Julien Stephan wrote: > > > > > [YOCTO #15162] > > > > > > > > > > When recipes such as bzip2 have several sources in SRC_URI, all the > > > > > sources are correctly extracted into devtool temporary workdir, but > > > > > currently, we only get back the primary source and all files declared > > > > > using "files://", then the devtool temporary workdir is deleted > > > > > > > > > > Fix this by moving all remaining files from devtool temporary workdir > > > > > into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", > > > > > and "recipe-sysroot-native") before deleting it > > > > > > > > > > Here is the stack trace from devtool modify/build bzip2: > > > > > > > > > > NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2 > > > > > ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None) > > > > > ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368 > > > > > Log data follows: > > > > > > DEBUG: Executing shell function do_install_ptest_base > > > > > > NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest > > > > > > sed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ > > > > > > ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile > > > > > > cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > > cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > > cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > > cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > > cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > > cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ > > > > > > ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2 > > > > > > cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory > > > > > > WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/ > > > > > 1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress' > > > > > > WARNING: Backtrace (BB generated script): > > > > > > #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 > > > > > > #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 > > > > > > #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226 > > > > > ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1' > > > > > NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed. > > > > > > > > > > Summary: 1 task failed: > > > > > <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base > > > > > > > > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com> > > > > > --- > > > > > scripts/lib/devtool/standard.py | 8 ++++++++ > > > > > 1 file changed, 8 insertions(+) > > > > > > > > > > diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py > > > > > index d53fb810071..6c8ad7a6c8c 100644 > > > > > --- a/scripts/lib/devtool/standard.py > > > > > +++ b/scripts/lib/devtool/standard.py > > > > > @@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works > > > > > shutil.move(srcsubdir, srctree) > > > > > symlink_oelocal_files_srctree(d,srctree) > > > > > > > > > > + # In case SRC_URI contains multiple sources > > > > > + # they are extracted into the devtool temporary workdir > > > > > + # We need to move them into WORKDIR, otherwise they will be lost > > > > > + for path in os.listdir(os.path.join(tempdir, 'workdir')): > > > > > + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: > > > > > + workdir = d.getVar('WORKDIR') > > > > > + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) > > > > > + > > > > > if is_kernel_yocto: > > > > > logger.info('Copying kernel config to srctree') > > > > > shutil.copy2(os.path.join(tempdir, '.config'), srctree) > > > > > > > > Looks like a good start although I worry a bit about the hardcoded list > > > > of exceptions. > > > > > > > > > Hi Richard, > > > Yes, I am not very happy with listing files but I don't know if I can > > > easily get the list of secondary sources? > > > > > > Here are the alternatives I can see: > > > * encapsulate the move into a try/catch block and ignore exceptions if > > > the destination file/directory already exists > > > * retrieve SRC_URI and manually build the list of secondary sources to > > > move. This looks a little bit overkill here... > > > What do you think? > > > > I need to rework this patch. As is it doesn't work: if the recipe was > > already built then the sources I am trying to move during devtool > > modify from the devtool temporary workdir to WORKDIR already exist in > > WORKDIR. > > This makes several tests from devtool selftest to fail. > > > > Any suggestions on how I should handle this? > > There is some work on the bitbake list from Alberto which adds unpack > tracing. I'm wondering if by default we should track some really basic > form of unpack accesses which would then give you a list of target > locations to merge? > Looks very interesting, let me check this :) Cheers Julien > Cheers, > > Richard > > > >
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index d53fb810071..6c8ad7a6c8c 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -684,6 +684,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works shutil.move(srcsubdir, srctree) symlink_oelocal_files_srctree(d,srctree) + # In case SRC_URI contains multiple sources + # they are extracted into the devtool temporary workdir + # We need to move them into WORKDIR, otherwise they will be lost + for path in os.listdir(os.path.join(tempdir, 'workdir')): + if not path in ["source-date-epoch", "recipe-sysroot", "recipe-sysroot-native"]: + workdir = d.getVar('WORKDIR') + shutil.move(os.path.join(tempdir, 'workdir', path), workdir) + if is_kernel_yocto: logger.info('Copying kernel config to srctree') shutil.copy2(os.path.join(tempdir, '.config'), srctree)
[YOCTO #15162] When recipes such as bzip2 have several sources in SRC_URI, all the sources are correctly extracted into devtool temporary workdir, but currently, we only get back the primary source and all files declared using "files://", then the devtool temporary workdir is deleted Fix this by moving all remaining files from devtool temporary workdir into actual WORKDIR (except "source-date-epoch", "recipe-sysroot", and "recipe-sysroot-native") before deleting it Here is the stack trace from devtool modify/build bzip2: NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2 ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None) ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368 Log data follows: | DEBUG: Executing shell function do_install_ptest_base | NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest | sed -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \ | ../../../../../../workspace/sources/bzip2/Makefile.am > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile | cp ../../../../../../workspace/sources/bzip2/sample1.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ | cp ../../../../../../workspace/sources/bzip2/sample2.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ | cp ../../../../../../workspace/sources/bzip2/sample3.ref <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ | cp ../../../../../../workspace/sources/bzip2/sample1.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ | cp ../../../../../../workspace/sources/bzip2/sample2.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ | cp ../../../../../../workspace/sources/bzip2/sample3.bz2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/ | ln -s /usr/bin/bzip2 <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2 | cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory | WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/ 1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress' | WARNING: Backtrace (BB generated script): | #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189 | #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158 | #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226 ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1' NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed. Summary: 1 task failed: <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base Signed-off-by: Julien Stephan <jstephan@baylibre.com> --- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)