LPM: Local Package Manager

Want to install software without root?

What is LPM?

LPM is a package manager that installs packages (e.g., tar ball) under your home directory. You do not need root privilege.

LPM does not take rpm/deb packages. It basically takes GNU-style tar balls. It can take other types of files if you write an LPM script that describes how to install it.

What is the difference with yum or apt?

If you do not have root, you cannot use yum or apt. LPM does not need root.

I can install a package (tar ball) to my home directory.

Yes, you may not need LPM if you can manually download a tar ball and can do './configure --prefix=$HOME/local && make && make install'. I was doing that but got much frustrated so I developed LPM. Suppose if you have accounts on several supercomputer centers:

  • You google the name of the software you want to install and download the latest version of the software. You may not want to copy-and-paste the URL from browser to your terminal more than once.
  • You have to specify to which directory you download the tar ball. You have your convention so you always specify the same directory (e.g., ~/archives) but you do not want to do that more than once.
  • You sometimes miss --prefix for configure, and get frustrated.
  • Some packages require us to set environmental variables after doing 'make install'. You may have to set PATH, MAN_PATH, LD_LIBRARY_PATH, or FOO_ROOT, etc., but you may believe that the process can be automated.
  • You like tcsh but most packages set only .bash_profile and it does not work.
  • You sometimes find that no uninstaller comes with the package. Packages created by using GNU automake can be uninstalled by 'make uninstall', but still with this case, you have to remove settings in .bash_profile or equivalent. Non-GNU-style packages are harder to uninstall.
  • You want to install modules of Perl, Python, or Ruby. LPM sets appropriate environmental variables, so you can easily install modules without root.
  • You want to install different verions of the same library (or program). You can switch between different sets of programs.

Usage

How to install a new package

When the target software has an LPM script in the default public repository (e.g., ttyrec):

$ lpm install ttyrec

When You wrote an LPM script for some software,

$ lpm install foo.lpm (Specify the absolute path if it is not in the current directory)

Install from GNU-style tar ball:

$ lpm install http://www.example.com/foo.tar.bz2 (You can specify the path of a local file instead of URL)

How to get the list of the installed packages.

lpm list

How to uninstall a package (e.g., foo).

lpm uninstall foo

How to update a package (e.g., foo).

lpm update foo

"lpm update" command executes "lpm freeze", "lpm uninstall", and "lpm install" sequentially.

When the last "lpm install" fails, the package disappears in the database. You can still roll it back by "lpm thaw" command.

Freezing a package (e.g., foo).

lpm freeze foo

LPM tracks installed files. We can save a package by creating a tarball that contains all the binaries (all the files) in the tarball. The created tarball will be located at ~/lcl/lib/lpm.

You can roll back a package with this command, which is especially useful when you fail to update the package (and it is removed from the database).

When two machines have the same version of operating system and your account names on them are the same, and you are lucky enough, you can copy the freezed binary tar balls between the machines, thaw them, and would be able to get it working (whether it happens depends on packages).

Show the list of the freezed packages.

lpm fridge

If the second argument (pattern) is given, you can search by regular expression.

Thaw a freezed package (e.g., foo)

lpm thaw foo

Thaw a specified freezed package. If that package is already installed, you encounter an error.

You might have to add the version number to avoid ambiguity when multiple versions of the package are freezed (e.g., lpm thaw ttyrec-1.7.0).

How to install a Perl module.

Do 'lpm initcpan' first. You need to do it just once; you can skip it when installing second module or later. We assume that you can use Perl CPAN, so we omit how to setup CPAN.

Install any module as usual. Alternatively, you can use 'lpm installcpan module_name'.

How to install Python package.

Do 'lpm install virtualenv'

Install any module as usual. We assume that you know how to use vertualenv.

How to list packages registered in repositories.

Do 'lpm listrepos', and you will see the list of available packages in repositories.

At this moment, lpm itself does not have search command. Use grep or whatever for search.

Questions we want you to ask.

How do I write a new LPM script?

See below.

Why LPM use '~/lcl'? I want to use ~/foo instaed.

Add '--local=foo'. For example, 'lpm initlocaldir --local=foo'. 'lpm initlocaldir' modifies lpm itself and '--local=foo' becomes the default, so you do not need to specify it every time.

ABC is missing in the default public repository. ABC should be there!

We cannot maintain all packages in the world.

We would be happy if you maintain an LPM script for some program and share it with people in the world!

I wrote an LPM script, but do not want to maintain it.

Something is better than nothing. If you send it to us, we put it on the public repository, and we are still happy with that. Someone in the future may write a better script based on your (old) script.

I tried LPM but it was not my taste. How do I unintall LPM?

Do 'lpm removelocaldir', and everything related to LPM will disappear.

Things that might be are comments in start-up scripts like .bashrc (the author is lazy and have not written a code for removing the comments).

Our system has a firewall that blocks access to sourceforge, and therefore 'lpm initlocaldir' fails. How should we do?

Download the latest (not necessarily but recommended) version of porg (see the web page), and give 'porgbase' option to 'lpm initlocaldir'.

e.g.) lpm initlocaldir --porgbase=/tmp/porg-0.1.tar.gz

lpm ssconfig command creates a ton of backup files, which are annoying to me.

When LPM modifies shell startup files, LPM saves the original files for safety. If you trust LPM and you do not need any backup files, edit the source code of LPM. Search the line that contains "use constant DEFAULT_BACK_UP => 1;", and change 1 to 0. Then LPM will not create any backup files. "lpm updatelpm" keeps the settings, so you do not have to worry when you update LPM.

How to write an LPM script

Basics

Lines that start with '#' (without quotes) are considered as comments. Empty lines are just ignored.

If the target application is a GNU-style tar ball, (i.e., you can do './configure && make && make install' to install it)

Fill the URL in the line that contains 'source='.

# The line below is used as the description of the package when the script is uploaded.
# Description: netcat
#
# The line below shows what Linux distribution (or MacOS) you tested the package with. Shown on the web if uploaded.
# Tested: CentOS 6.4
#
# Describe dependent packages. It will not be processed automatically, so you can write anything human-readable.
# However, if we have a corresponding LPM package (in the central repository), use the package name. You can omit the line if the package does not have any dependency.
# Depends:
#
# Use note tag (in comment lines) if you have long explanation that does not fit within a single line.
# <note>
# It depends on almost nothing.
# Should be supereasy to install.
# </note>
#
# source URL. It specifies where to download from.
source=http://internap.dl.sourceforge.net/sourceforge/netcat/netcat-0.7.1.tar.bz2
# URL. Just for information. If you upload the script, it will be linked to this URL.
url=http://netcat.sourceforge.net/
# download the package
download
# extract files
extract
# configure
configure
# make
make
# make install
makeinstall

source is a URL from which you can download a tar ball. It usually starts with http:// or ftp://.

source can also be a git/hg/svn repository URL. If you download the source tree from a git repository, then you can specify the git path (git://...) as source. LPM infers the repository type (git, mercurial, subversion are) from the URL, but you may explicitly prepend the repository type and a comma to the URL in order to disambiguate (e.g., source=git,http://foo.example.com/reponame). You do not need extract command when you downloaded things from a repository.

Please specify the URL in 'url=...' if you want to share the script with others in the public repository. You will see the link in the package list page.

I am a developer of LPM scripts and am installing a commercial package, so I can't make LPM download it automatically.

First, use 'manualdownload' instead of 'download' in LPM script. 'manualdownload' just checks if the package is in the archive directory (Default: ~/lcl/archive).

If there is not, LPM shows a message that asks you to manually download the package. If you know the license name, give it as an argument for 'manualdownload', and then it will be shown to user.

For manualdownload, specify just a file name in 'source='; you do not have to specify an actual URL.

I am just a user of LPM, and want to install commercial packages that cannot be downloaded without getting licenses. However, I have a lot of machines without shared file system. Is there any way to save my time?

Yes, you can create a file, ~/lcl/.lpmlocalrepo (the exact directory may change according to --local option), in which you describe a remote directory for which only you have access.

When a commercial (or non-free) package is not available locally, LPM will try to secure copy (scp) it from the directory specified in .lpmlocalrepo.

It uses scp, so you can describe anything that should be passed to scp command line. Below are some examples.

# You can write comments. The format is the same as scp. Try 'man scp' for details. username@ssh.example.com:/home/username/secretdir # You can also pass scp options if needed. -l username -I /home/myuser/.ssh/id_rsa_lpm ssh.example.com:/home/username/secretdir

I am installing a Java application, so I don't need to make it.

Comment out the line that starts with 'make' in the above example.

LPM does not recognize the name of the program and/or the version number.

If the file name of the tar ball does not follow the GNU style, specify them manually. LPM uses a simple regular expression to extract them, so it may fail with complex names.

The most common pattern is the version number and the package name are separated by underscore, not by hyphen (e.g., boost_1_49_0. boost-1.49.0 is okay.).

source=http://example.com/really-difficult-to-read-SoftwareName0.1.bin2.22fc.example.ver.zip
name=SoftwareName
ver=0.1.bin2.22fc

If you download files from a repository, the version number is automatically set to the repository revision just after the repository is cloned (or checked out by subversion). You may overwrite the version number by adding ver command after download.

The top directory in the tar ball does not follow the GNU style, so 'make && ./configure' fails.

foo-1.2.3.tar.gz (or .tar.bz2) is the correct file name for foo version 1.2.3. The name of the top directory (the top-most directory in the tar ball) must be 'foo-1.2.3'.

For tar balls with other naming convention, you must set the top directory manually.

topdirname=SoftwareName_0.1.bin2.22fc

Package foo cannot be intalled by just 'make install'. We need to do something more complex.

Write a shell script for installation, and put it between 'custominstall' and 'EOC' in LPM script. You can assume that the current directory is the top directory when you execute the first line of the custominstall section.

# install
custominstall
cd FastQC
cp -r uk $LIB_DIR/java
mkdir $SHARE_DIR/FastQC
cp -r *.txt *.ico *.bat Help $SHARE_DIR/FastQC
EOC

You can use the following environmental variables in 'custominstall' or 'shell' blocks. Here we assume that the local directory is ~/lcl (you can specify it by --local option) and that the file name of the tar ball is 'hogehoge-1.3.2.tar.gz'.

Env. var. nameExample
$LOCAL_DIR~/lcl
$ARCHIVE_DIR~/lcl/archive
$BIN_DIR~/lcl/bin
$LIB_DIR~/lcl/lib
$VAR_DIR~/lcl/var
$OPT_DIR~/lcl/var
$SHARE_DIR~/lcl/share
$MAN_DIR~/lcl/man
$BUILD_DIR~/lcl/build
$INCLUDE_DIR~/lcl/include
$LPMLIB_DIR~/lcl/lib/lpm
$PACKAGE_NAMEfoobar
$PACKAGE_VER1.3.2
$ARCHIVE_FILE~/lcl/archive/foobar-1.3.2.tar.gz
$OSlinux or darwin
$ARCHx86_64 or x86
$RAW_ARCHx86_64, i386, i686, ...
$DIST_TYPE'rhel' represents Cent OS, RedHat Enterprise Linux, Fedora Linux, and Scientific Linux, 'suse' represents SuSE Linux, 'ubuntu' represents Ubuntu. 'mac' on Mac OS.
$DIST_DESCReturns a distribution-specific string. For example, it is 'Red Hat Enterprise Linux Server release 6.4 (Santiago)' on RHEL 6.4.
If you give --local=foo to initialize the LPM local directory, these values become ~/foo/archive, ~/foo/bin, ..., respectively.

In the example above (custominstall), we are using normal cp command. However, all the files generated in custominstall block are automatically tracked by porg, so you can uninstall it even if an uninstall script does not come with the package. The only exception is that porg fails to track generated files when the installer uses statically linked binaries such as Oracle's Java VM to create files. porg depends on dynamically linked glibc to track files, so kernel calls cannot be monitored when you use statically linked binaries. Don't worry. We usually do not see such installers.

What should I do with binary distributions?

See the subsection above. In short, you have to describe how to install it.

If you wish to change the download path according to the machine architecture, see below.

# Use this URL for Linux
if{$osname eq 'linux'} source=linux_binary.tar.gz
# Use this URL for Mac
if{$osname eq 'darwin'} source=macos_binary.tar.gz
# Use this URL for 64bit (Intel/AMD x86_64)
if{$btype eq 'x86_64'} source=binary.x86_64.tar.gz
# Use this URL for 32bit (Intel/AMD x86)
if{$btype eq 'x86'} source=binary.x86.tar.gz
# Use this URL for Linux 64bit (Intel/AMD x86_64)
if{$btype eq 'x86_64' && $osname eq 'linux'} source=linux_binary.x86_64.tar.gz
# Use this URL for Mac 64bit (Intel x86_64)
if{$btype eq 'x86_64' && $osname eq 'darwin'} source=macos_binary.x86_64.tar.gz

An expression inside "{}" will be evaluated as Perl expression. Note that the parentheses are different from what are normally used in Perl. $osname and $btype were predefined variables. You cannot use code blocks (i.e., "if" affects to a single line.)

Here are the most commonly used variables.

Variable nameDescription
$osnameis 'linux' on Linux, 'darwin' on Mac OSX.
$btype'x86_64' on 64bit OS, 'x86' on 32bit OS.
$dist_info->{type}'rhel' represents Cent OS, RedHat Enterprise Linux, Fedora Linux, and Scientific Linux, 'suse' represents SuSE Linux, 'ubuntu' represents Ubuntu. 'mac' on Mac OS.
$dist_info->{description}Returns a distribution-specific string. For example, it is 'Red Hat Enterprise Linux Server release 6.4 (Santiago)' on RHEL 6.4.

LPM finds the latest version of porg at sourceforge. Can I use that feature for other software?

getlatest=package URL

You can use the getlatest command. Please replace 'package' with the package name and 'URL' with the URL (of the file listing). LPM will download and analyse the HTML page to figure out which version is the latest. 'source', 'name', and 'ver' will be set automatically as long as the package is named in the GNU style (package-ver.tar.(gz|bz2|xz)).

I want to add configure options.

You can add options just after 'configure'.

configure --disable=lib-foo

Instaed of doing 'make', I want to execute more complex commands.

A block surrounded by 'shell' and 'EOC' are executed instead of just doing 'make'. It look much like 'custominstall'/'EOC' block, but 'shell' does not trace created files.

The current directory is always the top directory when you execute the first line of a 'shell' block. Even if you do 'cd' in a shell block, it does not affect to other 'shell'/'custominstall' blocks.

I want to set up environmental variables.

If you want to setup some environmental variables when you login, write like this:

loadstartup
setlogin
export FOO=abc
...
EOC
savestartup

'export FOO=' is replaced with 'setenv FOO ' for tcsh/csh, all you have to prepare is a bash script. Appending idioms like ‘export FOO=$FOO:bar’ will be automatically converted to an equivalent csh script (although the converted script might be long), so just write a bash script.

If you want to setup some environmental variables for all shells (interactive/non-interactive), write like this:

loadstartup
setini
export FOO=abc
...
EOC
savestartup

I want to check the hash of a tar ball.

You can use either md5 or sha256.

md5 9fcbe4fbb11f9168c675a9ab05879e7a
sha256 5baf33b4232d267acb1e1ef8c0606b0bad5866473837a3fd22f714fca29efcbf

A tar ball in my hand is bogus. I want to modify a bit of some file in the tar ball.

We often see that situation, such as 'I want to add -fPIC to this line of Makefile!'.

replaceregexp
file_name
search_regular_expression
replace_regular_expression

replaceregexp command uses four lines. Differnt syntaxes that occupy only one line might be possible, but you might get frustrated when you replace strings with spaces.

file_name is the relative path to be modified (remind that the current directory is the top directory). LPM searches the file from the beginning, and when it finds a line that matches search_regular_expression, it applies replace_regular_expression

Do you get it? The example below finds a line of CFLAGS setting in Makefile, and add '-fPIC' option.

replaceregexp
Makefile
/CFLAGS/
s/CFLAGS=/CFLAGS=-fPIC /

I want to do 'make' in parallel.

Use LPM_MAKE_OPTION.

Users can set the number of CPU cores used for build.

export LPM_MAKE_OPTION=-j8
lpm install foo (make runs with 8 threads)

I want to update LPM.

Do 'lpm updateself'

lpm updateself

If you would like to use the latest development version (unstable), type 'lpm updatelpmdevel'.

Can't download from https sites.

Some web sites do not allow downloading from minor software (likely to be automatic downloading software).

Please add --wget option to use wget for downloading.

I want to use several local directories.

Suppose that you are going to try a set of programs with gcc and Intel C to see the difference in performance.

First, you install the programs to ~/gcc, using gcc.

$ export CC=gcc
$ lpm initlocaldir --local=gcc
$ exec $SHELL -l
$ export CC=gcc (CC might be reset at this point)
$ lpm install targetsoft1 --local=gcc
$ lpm install targetsoft2 --local=gcc
$ lpm install targetsoft3 --local=gcc

Let's turn off ~/gcc directory.

lpm ssconfig gcc off

Make sure that ~/gcc is inactive

lpm ssconfig

Relogin to reset environmental variables.

Next, we use icc (Intel C compiler) and install the programs into ~/icc.

$ export CC=icc
$ lpm initlocaldir --local=icc
$ exec $SHELL -l
$ export CC=icc (Again, CC might be reset at this point)
$ lpm install targetsoft1 --local=icc
$ lpm install targetsoft2 --local=icc
$ lpm install targetsoft3 --local=icc

You can do benchmarks with targetsoft1,2,3 compiled by icc. Once the benchmarks with the icc binaries finish, you turn on gcc instead.

$ lpm ssconfig icc off
$ lpm ssconfig gcc on

Please relogin to reset environmental variables.

Next you can do benchmarks with the gcc binaries.

I could not download because wget complains that an SSL certificate could not be verified.

A textbook says that if you cannot verify the certificate, simply you should not download it. You know the security risks and still want to download it? Okay, you can add --wnocert option to lpm in order to turn off the security feature.

If extra plugins/modules will be installed without using LPM (e.g., R packages, ruby modules), how do we track such files as under a parent package (e.g., R, ruby)?

Create a file named '.lpm_save_under_this_dir' in such directories in an LPM script.

When 'lpm freeze' finds that directory, it also adds files under that directory to the archive. Note that directories specified in this way will not be removed by 'lpm uninstall'.