Changes between Initial Version and Version 1 of FeatureRequests

Timestamp:

Sep 28, 2006, 11:33:08 AM (19 years ago)

Author:

anonymous

Comment:

--

FeatureRequests

@@ +1 @@
+= CMT feature requests =
+
+by V.Garonne     [[BR]]
+Date: 09.28.2006  [[BR]]
+
+
+== 1. Support for tags & macros in the cmt/project.cmt file (DavidQ) ==
+
+The particular motivation for this came from LCGCMT, where LHCb wish to install 
+it at CERN within an InstallArea, whereas ATLAS does. Currently this implies two separate  
+installations, but the ability to do something like:
+
+{{{
+macro use_strategy "LHCb" without_installarea "ATLAS" with_installarea
+build_strategy $(use_strategy)
+}}}
+
+would allow the use of a single installation. There was also a  
+request for support for the "author" keyword.
+
+== 2. Support for native_version use statements (DavidQ) ==
+
+This goes together with 3. The suggestion is for support for
+
+{{{
+use Foo * -native_version=1.2.3
+}}}
+
+This would require that Foo declared the Foo_native_version macro to  
+be set to 1.2.3 (in this example).
+
+== 3. Wildcard or prioritized project dependency support (DavidQ) ==
+
+At the moment CMT does not support wildcarded project dependencies.  
+This has two significant consequences:
+
+a. It is not easily possible to test a new version of a project  
+within the context of an existing project hierarchy.
+
+b. In some cases a project changes faster than the version of e.g. a  
+glue package to an external package. If a package within a higher level  
+project only depends upon the version of the external package, the project  
+needs to be versioned, even though no code changes have taken place.
+
+Take the example of the tdaq-common project. To first order, it only  
+depends upon the Boost, for which there is a glue package within the  
+LCGCMT project. The Boost version is stable over periods of many  
+months, but the LCGCMT project version changes much more frequently  
+because other packages (e.g. ROOT) change more frequently. Thus  
+currently tdaq-common would need to be changed and rebuilt at a rate  
+that's at least as high as that for LCGCMT, even though nothing has  
+changed. My proposal was that if a diamond project dependency existed  
+(in our case AtlasConditions depends upon tdaq-common, which depends  
+upon LCGCMT, and AtlasConditions also depends upon AtlasCore, which  
+depends upon Gaudi and then LCGCMT), then as long as one leg of the  
+diamond fully specified the project versions, the other leg could  
+"weaken" the dependencies with some sort of wildcarding. Thus as long  
+as exact versions of AtlasCore, Gaudi and LCGCMT were specified on  
+one branch, then tdaq-common could specify a weak dependency against  
+LCGCMT, and use the native_version package use statement described in  
+2. above to ensure that the correct version of Boost was used.
+
+I believe a similar scheme could be used to insert an updated project  
+version into an existing project tree for test purposes, although I  
+haven't thought it through yet.
+
+== 4. Support for <project> placeholder for patterns (DavidQ) ==
+
+CMT provides several placeholders for use within patterns, most  
+specifically <package>, which is replaced by the package name at  
+build time. I'd like to see <project> added in the same way. One  
+motivation for this is shown by the existing installed_library  
+pattern. This essentially has:
+{{{
+    macro_append <package>_linkopts " -L$(bin) -l<package> <extras> "
+}}}
+which is anyway incorrect (so needs to be fixed). The correct fix  
+would be something like:
+{{{
+    macro_append <package>_linkopts "" \
+                 <project>_without_installarea " -L$(<PACKAGE>ROOT)/$ 
+(<package>_tag)" ; \
+    macro_append <package>_linkopts            " -l<package> <extras> "
+}}}
+Which would use the library from within the package if no InstallArea  
+is present, and otherwise use the symlinked version from the  
+InstallArea.
+
+== Simplification of the -I<paths> (DavidQ) ==
+
+Currently we have a -I<path> entry for every used package, even  
+though all the package include directories are symlinked from the  
+InstallArea. This results in very long compilation command paths  
+which are bulky in logfiles and essentially opaque for humans to  
+read. The motivation for this was the use case where someone removes  
+a header file from a package, and wants to test whether another  
+package that has been checked out has been updated not to include the  
+now missing file. Without the long -I<path> the header file will  
+still be located within the underlying base release. At the time we  
+made this decision Christian proposed that the obsolete header file  
+should not be removed, but marked (via a pragma statement) to ensure  
+that it would cause clients to fail to compile. I now believe that he  
+was right and that we should adopt this strategy. I note that once  
+the updated package has been incorporated into the base release the  
+now obsolete header file can be deleted if so desired. I played with  
+bit to try to get this to work some time ago, but never got it  
+completed so now it needs to be resurrected. Note that this has to be  
+done per release rather than intrinsically by CMT since old releases  
+will still have the existing symlink structure.
+
+== 6. Revisit g77 include path (DavidQ) ==
+
+There's a horrible hack I put in because g77 has a character limit,  
+not only on the overall command line, but also on the -I<path>  
+options. This hack involves some symlinks in the project to other  
+projects and causes problems with search paths and is a real mess. We  
+should revisit this, perhaps in conjunction with 7. below.
+
+== 7. Investigate whether we can use gfortran instead of g77(DavidQ) ==
+
+Currently we use gfortran for the Saclay F90 code, and g77 for  
+everything else. I think we should look at using gfortran for  
+everything, since that would solved 6. above, and provide additional  
+uniformity. We would need to ensure that gfortran was shipped with  
+the kit, but that's done already.
+
+== 8. cmt co tag (DavidR) ==
+
+Another thing not vital but nice to have, which was suggested several  
+time in the past is to be able to specify a cmt co tag like :
+cmt co Reconstruction/RecExample/RecExCommon-01-02-03
+
+
+== 9. Make CMT a library rather than a static executable (SébastienB) ==
+
+This surely is a long-term item but I think it would ease the work of people 
+writting scripts (see b.) or building plugins for CMT.
+
+== 10. Have python bindings for CMT (SébastienB) ==
+
+Athena (and Gaudi, for that matters) would greatly benefit from having 
+official python bindings for CMT. Just have a look at how many python 
+wrappers have been written for CMT in the Atlas CVS repository.
+
+== 9. Introduce the concept of a Release (or some equivalent 'meta-data') (SébastienB) ==
+
+The central concept in CMT, according to the CMT-documentation, is a Package.
+(Looking at the source code, I'd argue it isn't completely true and would be 
+more inclined to say the central concept in CMT is the 'Use statement', 
+anyway...)
+I'd like to have the concept of a Release to be introduced. This would be used
+when one installs a (set of) project(s) with a frozen set of packages with 
+their version and dependencies (clients+uses).
+Iterrogating this Release object for package dependencies would be much faster
+than letting CMT rebuild and recompute each time the whole set of dependencies 
+whenever one does 'make'.
+I believe this would also speed-up the 'cmt show clients CustomerIsTheKing' 
+command.
+
+== 10. Separate the 'build' environment from the 'user' environment (SébastienB) ==
+
+Having CMT running into its own environment without touching the user 
+environment would allow a faster turn-around between nightlies/releases 
+(developer-centric) but also ease bug-reporting (user-centric) : the CMT 
+environment being described by a single file would dramatically reduce 
+confusion (while developers are trying to reproduce users' bugs) and ease 
+spotting mis-configuration.
+
+== 11. Introduce a mechanism to describe runtime dependencies between packages (SébastienB) ==
+
+Title says it all.
+
+== 12. Use checksums rather than timestamps to decide if a file has been modified (SébastienB) ==
+
+Hopefully, this will reduce re-compilation time.
+
+=== Additional remarks (AndreiG) ===
+
+we agreed that computing checksums can't be faster
+than testing time stamps; the use case he had in mind is when you
+inadvertently touched a file, and ended up with the same content but a
+different time stamp.
+
+=== Additional remarks (WimL) ===
+
+the point of checksums isn't to save time, it's to increase accuracy. If I
+revert my local sources to an earlier version from CVS, my object files and
+other build products have a time stamp that is newer, even as what they
+depend on has changed. Checksums will save you from having to remove all
+build directories.
+
+
+=== Additional remarks (AndreiG) ===
+
+Accuracy is important, but the compilation time is also
+important.  There is a trade off between the risk of forgetting to
+remove build dirs after going back to an earlier version of a package
+and increasing compilation for each of the builds.  I'd say that going
+"back in time" happens rarely, and therefore I am willing to accept
+the inconvenience of removing generated files by hand in such cases
+instead of increasing compilation time under routine circumstances
+(which is way too slow as it is).
+
+=== Additional remarks (WimL) ===
+
+> > I'd say that going "back in time" happens rarely
+
+YMMV. I do it a couple of times per day. Depends on work style, I guess.
+
+> > of increasing compilation time under routine circumstances
+
+There's a leverage effect. If I have to recompile my complete work directory
+b/c of a backtrack in one package, the total increase is enormous, and may
+surpass the cost of checksums in the routine circumstance. If I loose a day
+b/c of a subtlety, the scale goes even further in the direction of checksums.
+
+
+== 13. Profile and optimise CMT (SébastienB) ==
+
+It would be great to reduce the time to compile packages in Atlas. Surely a 
+chunk of this time comes from the packages themselves (templates, complicated 
+packages dependencies, unneeded dependencies and stuff like that).
+But a simple test on UserAnalysis (which falls exactly in the above caveats):
+{{{
+make -s  63.46s user 23.47s system 70% cpu 2:03.38  total (first build)
+make -s  10.83s user  7.67s system 96% cpu   19.083 total (no-op build)
+make -s  11.31s user  7.37s system 97% cpu   19.083 total (no-op build)
+}}}
+gives ~10 seconds for a no-op build...
+And for RecExCommon which is a pure python package:
+{{{
+make -s  31.83s user 24.17s system 97% cpu 57.578 total (first build)
+make -s  10.98s user  8.02s system 99% cpu 19.050 total (no-op build)
+make -s  10.65s user  8.08s system 96% cpu 19.384 total (no-op build)
+}}}
+=> still ~10seconds for a no-op build.
+(so I'd naively conclude it comes from CMT/Atlas-macros rather than the 
+content of the package)
+
+== 14. have a build-dir location which can be configured (SébastienB) ==
+
+Hence it would be possible to install all these possibly large .o,.so,.dict 
+files (especially in -dbg mode) under some /tmp/${USER}/build directory 
+without wasting AFS space.
+
+== 15. a mechanism to integrate 'plug-ins'(SébastienB) ==
+
+Plug-ins are hype.
+I believe the Atlas package structure and the dependencies between packages 
+would benefit of the integration of various Atlas-specific CMT plug-ins. Here 
+I am thinking about checkreq. If checkreq was integrated during the 
+development process of a package (and not 'just' at the nightly step) it 
+would improve the overall quality (or correctness) of cmt/requirements files
+content.
+One could think also to Atlas specific default cmt/requirements file which 
+would be used during the 'cmt create Foo Foo-00-00-00 Path/toFoo' command and
+check it sticks to Atlas conventions.
+Another useful plug-in, as we are heading towards gcc-3.4.x, would be a 
+precompiled-header plug-in.
+
+== 16. produce makefiles structured according to the "Recursive Make Considered Harmful" prescription (AndreiG) ==
+
+It would be very useful if cmt could produce makefiles structured
+according to the "Recursive Make Considered Harmful" prescription
+
+[1] http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html 
+
+This should give a huge speed up compared to "cmt br gmake" for
+someone who needs to recompile a bunch of checked out packages after
+modifying some of them.
+
+I imagine the following work model:
+
+- a developer checks out a set of packages he is working with,
+
+- issues a cmt br command to build a set of makefiles from the
+  "topmost" package in his set (or from a specially created test
+  package depending in all the checked out packages),
+
+- and from this point he only needs to run "gmake" from that same
+  location to recompile exactly what needs to be recompiled after
+  making modifications to any of his packages.  Or "gmake -j3"  :) 
+
+The role of the cmt here is to resolve package versions and create
+appropriate makefiles.  It only needs to be run once, unless you
+change the set of checked out packages, or the nightly you are working
+with.  Gmake run on the master makefile knows all the dependencies,
+and does the minimal job to compile just what is needed, and also to
+re-compute dependencies that are maintained in a granular way (one .d
+file per one .cxx file).  Having all the knowledge in one instance of
+gmake makes sure everything is updated properly - even if you use a
+-jN gmake option to build in parallel.  (I've heard that cmt develops
+a custom parallel build solution - but gmake is already well debugged,
+why not use that?)
+
+The global gmake approach [1] is used by ROOT, see
+$ROOTSYS/README/BUILDSYSTEM (and perhaps by many other projects).
+
+I also used it for a set of projects, and can assert that 
+
+1) It works beautifully,
+
+2) It IS easier to implement than it seems at first.
+
+I expect a large speedup because of a proper handling of dependencies
+that are stored and not recomputed unless needed.  Also, not asking
+cmt to re-resolve all the used package versions on each "gmake" should
+save us a lot.
+
+