Question about LD_PRELOAD, environment, masking an app

[matiasbatero]

Hi!

I want to execute applications that have been built for another distros (like archlinux). I have all dependency tree packages, i merged all of them on a directory, including libc, all.

This full directory, have the next form:

Code: Select all

full_package_with_deps 
./etc
./usr
   ./bin  <-- here is my executable (elf)
   ./lib   <-- here is glib dep version, and ld-linker
   {...etc etc...}
./var 

etc etc.

I need to fake the executable inside on "full_package_with_deps/usr/bin/app" to avoid the app to look on my current / filesystem, where my libs have different versions, and different paths.

To do that.. i'm using: Environment variables.

Code: Select all

LD_PRELOAD="path_to_glib_version" + "ld-linux.. current package linker"
LD_LOAD_LIBRARY="path_to_usr_lib"
./usr/bin/app

---

The question is, there is a way to set a full custom environment, (not for sandboxing, isolation) in order to ensure the execution of any app, without using chroots, jails?

The main objetive of all of this, is to build a system that allows to run any app from any distro, using only packages from repos. Like a portable-bundle.

For example:
Fedora: X app --> ( 1 + 23 dep packages RPM)
Download all, extract, merge.. and set a custom enviroment, to mask executables to point at their own libs directory.

In my opinion, the last limitation is the version kernel for compatibility.
There is a inherent disadvantage: the weight of the app. (But, this can be avoided using a modern filesystem that uses copy-on-write.


----

Chroot evironments, are usefull, but it has some problems. For example, if you have an audio app running inside, and you want to connect throught jack, with another app running on host-system, or in another chroot jail. Is not possible, as far i know.

----

The restrictions to do this system is:
1) Use binaries (no compilation). Use all binaries that exists on repos.
2) Use system posix built-ins to run apps. (The objetive is portability)
3) Make integration transparent.
4) The host, and other target systems, cannot be modified in order to run portable apps.
5) No lost in perfomance execution.

---

I you have some ideas to help me to do this, you are welcome.
Also, i need your opinions, and some analysis: Good, and Bad things for you.

Regards.

[jamesbond]

Matias,

The forum has a lousy PM system. I replied to your PM and the reply got stuck in my "outbox" for days. Anyway, I'm publishing my response to the PM here - I hope you don't mind. If you do mind, let me know I'll remove this post.

---

I want to build a system that allows, build portable packages using different distro-repositories.

This is going to be difficult. You just need to follow the discussion on skype 4.3 thread as an excellent example of the difficulty.

1) In puppy SFS's are very common, but i don't know how it is implemented. For example, if you have a SFS with app, it is mixed with / directory, throught unionfs? ¿ ( layer1=/ + layer2=SFS ) ?

Yes. See http://distro.ibiblio.org/fatdog/web/fa ... ystem.html.

I read some comments, that is not possible, because is redundant to union / with itself. BUT, you said in some comment that the / is already a unionfs, you can append another layer over this directory.

Correct, you can add SFS to "/" if "/" is already a unionfs. Otherwise, you can't.

If its correct, it is only true for puppy?

It is true of any OS that uses unionfs as "/". Puppy is only one of them, there are many others: Slax, AntiX, and a few others. Not sure about DebianDog.

because i'm on Debian Squeeze now, and i can't append layers over /. (Maybe my / is not a unionfs , but i don't know how to set it)

You can't. Setting unionfs for "/" is done at boot time, in initrd. You can't do it when your system is already up and running.

2) It's possible to load SFS without using / path? For example, using a custom folder to make unions?

Yes, very possible. If your kernel has unionfs/aufs module, then it is very simple. If it does not, but at least it has FUSE module, then you can use unionfs-fuse (slower, though).

3) Chroot is another method to launch app's. But i think that it has problems. For example.. you chroot an audio app, and you have qjack in host.. jackd cannot see the process to request a connection. So integration could be problematic.

You can. But it must be done carefully - you must allow some files from the "host" to show up under chroot, too. This usually requires some sort of bind-mounts.

And you can combine SFS with chroot - load SFS under a certain mountpoint, and then do a unionfs to put r/w layer on top of that (e.g using tmpfs), and then chroot to that. Example: http://www.lightofdawn.org/wiki/wiki.cg ... koInFatdog.

Resume: My idea is:
> Build a package using any distro repo (with all dependency tree inside of course) avoiding to compile and make pkgs from source. (Only using available resources). (Glib can be a problem, but if it is included in package, can be resolved)

You have summarised succintly what "woof-next" from Woof-CE does

> Execute it transparently over any system.
> Avoid virtualization and chroot.

Not always possible if you want to avoid chroot. If you want to avoid chroot you can't bring the original distros glibc. This usually is not a problem, but if the packages requires newer glibc than the one installed on the host, it won't work.

I want to know, what are the inherent limitations. If this can be applicable.
And what do you think about this, if you have an alternative idea, or help.

If you want to avoid compiling, then you'll have to go with chroot method. Make the SFS, create aufs layers on top of it, use bind-mounts to bring /tmp, /dev, /proc, /sys and a few other important directories (perhaps /var too) to it, then chroot. It isn't perfect, but it is close.

If you *do* want to compile, then the best way is to install everything to /opt/your-pkgname - including the libraries, and then set LD_LIBRARY_PATH to point to it. This is how libreoffice does it, this is how firefox does it - they have everything including the libraries in their own directory, all executed under a wrapper script. Thus, no chroot or aufs needed.

-----


As for your post above, with LD_LIBRARY_PATH you can redirect almost everything, except:
a) glibc
b) files and libraries that that path hardcoded in them (RPATH)
c) hardcoded data file locations.

cheers

[amigo]

I guess I'll also answer here as I also have a PM in my inbox from matias.

Unioning only works on live CD distros because aufs is not inlcuded in the kernel of any standard (installable) distros. And since uafs is implemented with a kernel source-code patch, it can't be simply added to an existing setup unless you replace the kernel with your own.

Using LD_PRELOAD and PATH(s) is quick and elegant but will not work for things located under /etc or /usr/share -it only lets you point towards libraries and binaries.

So, chroot is the only thing that can be used everywhere and always. Your example with jack should also work, it's simply a matter of making everything needed visible and available under the chroot directory before chrooting in there. You do this with the 'bind' option to mount.

The whole matter boils down to this: there is no way to cleanly and reliably use 'any' app from 'any' distro anywhere else. The whole woof concept is fatally wounded from the start. Even if it was limited to building from a single distro and version, *for* a single distro and version(some puppy), there would still be problems which would require a different solution. There is no magic way to make programs 'just work' outside of the original context.

It is possible to build a sane live distro, but this requires using sane methods, like using the original toolchain to compile any new programs which are not included in the original distro, or any programs which are included but must be re-compiled to alter the options and/or dependencies. The other place where puppy really 'drops the ball', is that, when installing to hard disk, most of the liveCD infrastructure should be replace with their normal counterparts. This means, for example, using real init processes which differ wildly from the processes used by any liveCD.

[matiasbatero]

Matias,

The forum has a lousy PM system. I replied to your PM and the reply got stuck in my "outbox" for days. Anyway, I'm publishing my response to the PM here - I hope you don't mind. If you do mind, let me know I'll remove this post.

-----
cheers

Hi jamesbond!, thank you so much for your response. There is no problem for me! Keep your response here! (PM's and your posts came at the same time)

You have summarised succintly what "woof-next" from Woof-CE does

Wow, i will check this.

Not always possible if you want to avoid chroot. If you want to avoid chroot you can't bring the original distros glibc. This usually is not a problem, but if the packages requires newer glibc than the one installed on the host, it won't work.

LD_PRELOAD, can load a different GLIBC. That can help in some aspects. But, maybe each app have a custom analysis and can be less practical.

If you *do* want to compile, then the best way is to install everything to /opt/your-pkgname - including the libraries, and then set LD_LIBRARY_PATH to point to it. This is how libreoffice does it, this is how firefox does it - they have everything including the libraries in their own directory, all executed under a wrapper script. Thus, no chroot or aufs needed.

Yes, compiling, seems to be the more clean method.

But, then you have mentioned this:

With LD_LIBRARY_PATH you can redirect almost everything, except:
a) glibc
b) files and libraries that that path hardcoded in them (RPATH)
c) hardcoded data file locations.

How to solve, RPATH: Look this:
http://linux.die.net/man/1/chrpath
https://nixos.org/patchelf.html

User Amigo, says in this post, that LD_PRELOAD/LIBRARY, helps with binary and libs, that is true.. and he says that cannot redirect /usr/share.. so problems can be started for that point.

If you want to avoid compiling, then you'll have to go with chroot method. Make the SFS, create aufs layers on top of it, use bind-mounts to bring /tmp, /dev, /proc, /sys and a few other important directories (perhaps /var too) to it, then chroot. It isn't perfect, but it is close.

It seems the best portable way to do this. It isn't perfect because, this solution eats more space, i think.


Your answers have a good information for me, i will analyze some things and your links, and i will coming with some more questions

[matiasbatero]

I guess I'll also answer here as I also have a PM in my inbox from matias.

Of course!! Your response is welcome!

Unioning only works on live CD distros because aufs is not inlcuded in the kernel of any standard (installable) distros. And since uafs is implemented with a kernel source-code patch, it can't be simply added to an existing setup unless you replace the kernel with your own.

wow, i thought that unionfs were available on standard distros.

Using LD_PRELOAD and PATH(s) is quick and elegant but will not work for things located under /etc or /usr/share -it only lets you point towards libraries and binaries.

Yes, LD_xxx seems elegant. /usr/share or /etc cannot be redirected. It is a problem. Maybe for some apps, not for all. But, that reason is important, because it not seems to be a general solution.

So, chroot is the only thing that can be used everywhere and always. Your example with jack should also work, it's simply a matter of making everything needed visible and available under the chroot directory before chrooting in there. You do this with the 'bind' option to mount.

Yes, i will read some jamesbond links about this. Chroot with mount --bind points, for this moment, is the most portable method.

On puppy linux, chroot can run smoothless because, it never ask for permissions. But, in other distros, chroot approach requires permissions for any app to be executed. Yes, it can be solved. But, the solution needs to be very solid.

The whole matter boils down to this: there is no way to cleanly and reliably use 'any' app from 'any' distro anywhere else. The whole woof concept is fatally wounded from the start. Even if it was limited to building from a single distro and version, *for* a single distro and version(some puppy), there would still be problems which would require a different solution. There is no magic way to make programs 'just work' outside of the original context.

I think, that there is no way to make ALL programs working outside of the original context. But, if the solution can be great for 80% is acceptable.
By the way, at this moment, there is no linux distro that can run all available packages from repos. And for such apps, always requires manual intervention to get it running.

It is possible to build a sane live distro, but this requires using sane methods, like using the original toolchain to compile any new programs which are not included in the original distro, or any programs which are included but must be re-compiled to alter the options and/or dependencies. The other place where puppy really 'drops the ball', is that, when installing to hard disk, most of the liveCD infrastructure should be replace with their normal counterparts. This means, for example, using real init processes which differ wildly from the processes used by any liveCD.

I agree with you. The big problem on all linux distros, is all management is very centralized. And apps, are built to be friend to that way to do the things. A real solution, is "take the bull by its horns". Build all using sane concepts, solid implementations that can manage the flexibility.
Check: GoboLinux , the team redefines the unix-standard-tree. And look, what are their results. With a re-definition, they can avoid to use package-managers. http://www.gobolinux.org/

Thanks for your answer.
PD: Sorry for my english