DotPup Downloader

[klhrevolutionist]

I've started working on a new dotpup downloader, not as fancy as Marks.
Simply put it reads a txt file from my website as of now, but I am hoping to get it to read dotpups.txt from multiple websites.

You can see the code here, http://puppylinux.pastebin.com/713756

& if you modify it please post the link
in your reply to this post so that everyone who may contribute can do just that.

Download the alien/unleashed pkg here,
http://www.greenvilleroad.com/puppy/Downloader.tar.gz

!!!UPDATED!!!
Alright, I now have figured out how to get it to launch the dotpuprox.sh
Nathan F. if you can add your code to this to make the multiple dotpups.txt possible, we can release this.
I have been messing around, and I believe it will be possible to install unleashed packages as well. I am testing and will report back. But that is later.....

[Nathan F]

Simply put it reads a txt file from my website as of now, but I am hoping to get it to read dotpups.txt from multiple websites.

Easy enough. Create a config file, say /root/.dotpup/config, and place your url on it. Additional url's can go each on a new line. This way it's configurable. Then read with cat:

Code: Select all

URL1=`cat /root/.dotpup/config | head -n 1`
URL2=`cat /root/.dotpup/config | head -n 2 | grep -v "$URL1"`
URL3=`cat /root/.dotpup/config | head -n 3 | grep -v "$URL1" | grep -v "$URL2"`

This would give three options, and be easily changed via a simple text editor if one of the sites happened to move. Probably a better way to do this somewhere but this method springs to mind for me.

I'm working on something related but a bit different, I think it would be worthwhile to keep up with each other on how the projects are being coded. I'm thinking what you are working on might be a good candidate for inclusion in the new SVN repo, since what the pastebin site you're using is all about is marginally version control anyway. Well give it some thought.

Nathan

[Lobster]

Now we are cooking . . .

There is a conversation with Rarsa about svn
appears at end of post

I too started working on on a simple package manager (3 different version)
I started using a simple top 10 downloads / addons
and used my welcome.pup as a basis
I was not happy with that and moved to a menu scroll bar
then I looked at MU's code and this led me to the .pup pages
and got me thinking about HTML and XUL again . . .

I would say Kenny that the progress bar is a simple and important component.

How does this tie in with .pet?
What about the sticky links to software?

I did not get far with the code (on my other Puppy machine) - will post if I find it . . .

There are several missunderstandings here. I will try to clarify them.

> If not able to add projects are people gonna bother?

Any user can send an email to svnadmin@puppylinux.org to request having a
repository added or if he/she is the owner of a project, having someone
else added to the 'write' list.
This process should be quite responsive, meanwhile he/she can keep working
on the project as usual. I am requesting volunteers to help administering
the site. It will require a learning curve and following certain
guidelines that I will publish soon. Those guidelines will be quite simple
but are necessary.

> My suggestion is > making subversion as open as possible to start with
> so people are> encouraged to use it.
> It is an approach I used on the wiki - hope that makes some sort of sense
It makes sense if you compare it with the approach used for the DotPups
page. Someone has to administer it to make sure that the integrity is
preserved.
Managing a Subversion repository is not too complex but ain't easy either.
The administrator must have certain knowledge and follow the established
certain guidelines. Once you see the structure and guidelines you will
understand.
The Wiki has a free structure. A repository should not or becomes useless
quite fast.
The subversion repository is open for anyone to read and particular
projects will have owners that decide who can contribute. The only
difference with today's environment is that it will be more open and
organized.

> Extras is a program written in script
I asked specific questions because I must to know where to put it. If it a
script that shows documentation then it may belong to the documentation
repository if it is a script that downloads the apps then it may belong to
the scripts repository, etc.
Please read further down and let me know where you think it belongs.
> Will the unleased programs be in Subversion?
Another clarification: Subversion is NOT a package repository. An FTP site
does a better job at that.

The purpose of subversion is to help coordinate during the development of
a project (be it a document, an image, a script or an executable)

Applications developed by third parties (e.g. geany, jwm, etc) should
already have a repository for this purpose.

This repository is for Puppy specific content. Apps or documents or images
developed specifically for puppy.

Regarding the Unleashed packages. What should belong into this repository
is the patches, instructions, development environment set-up scripts and
packaging scripts required to create the unleashed packages. I will create
an example of what I mean and I will include with the guidelines.

I have overused the words "repository" and "Project". The reason is that
the difference between them is more practical than technical and depends
on the particular context where it's used. For subversion it also depends
on the choosen layout of the directory structure that organizes the
projects.

Repository: A collection of related projects that share common
administrative properties.

Project: A collection of related files for the purpose of generating an
independent deliverable.

I had originaly thought that we would have project per repository so I was
referring to them as "projects". After experimenting and analysing the
content that will go into subversion I realized that that was not the most
efficient way.

There are many ways to organize projects in repositories but for now I've
settled in the following structure;

puppy-documentation repository
manpages project
trunk
tags
branches
manual project
trunk
tags
branches
etc...
puppy-scripts repository
jwmconfiguration project
trunk
tags
branches
network wizard project
etc...
puppy-miniapps repository
freemem project
...
mini-volume project
etc...
puppy-patches repository (this will only include the patches and
instructions/scripts to apply them)
dillo project
geany project
etc...
etc...

I hope you get the idea. Projects will be added to existing repositories
unless they don't fit any of them, in which case we will create another
repository for that category of applications.

Regarding access:
- There will be a few adminsitrators with full access to everything
- Some other people will have write access to specific repositories
- Some other people will have write access just to specific projects
- Everyone will have read access to all the projects.

[rarsa]

I've started working on a new dotpup downloader, not as fancy as Marks.

I think it would be beneficial to build uppon what Mark has created.

In any event, here are some ideas. I would recommend as a matter of practice to list all the ideas (requirements) you have or get from other people and keep adding them as you get them, then prioritize them regularly implementing the ones with a higher priority first. Don't try to implement all of them at once.

1. Handling multiple repositories

Simply put it reads a txt file from my website as of now, but I am hoping to get it to read dotpups.txt from multiple websites.

The way Nathan suggests seems the one favoured by other package managers work (e.g. yum). That way the user is responsible for deciding what repositories to trust.

2. Packages organization structure.
I would recommend start with a flat structure as the one you have but for phase 2 use a dynamic tree structure that way the app can organize the apps in different ways. XDG can be used for this.

3. Regarding DotPets, There is no reason why a tool like this cannot be extended to install other kind of packages.

Feel free to contribute in anyway. As of now it is using code from GuestToo, rarsa and myself.

Do you want to have this project on subversion?

[klhrevolutionist]

As for adding these lines of code nathan f. I am going by trial & error. I have not got the slightest clue of where or what any of it means, only that I edit, edit and edit till something works. however, as lobster mentioned I too have a problem of discarding what I am working on because I cannot figure it out.
As for the progress bar, I think it makes and end-user feel all warm inside lobster.

As for adding onto mu dotpup-downloader, I have tried and tried using puppybasic and wxbasic but that is harder for me than xdialog.

Now we always come upon this topic of the downloader because we all feel that there is more needed. But we never get very far, other than the .pet And yet it is not going to far either. And at this point I would rather not use subversion, although this needs to be addressed.

We all have unique ideas of what features need to be there. MU jumped outta nowhere and really helped us out, but over time we noticed that his dotpup-downloader has flaws.
Mainly bandwidth flaws Personally I have tried pushing for something that involves p2p. And nathan has his ideas, etc.. So can we come together & start using subversion as well as pastebin.com and get this train a rolling ?

With subversion & pastebin we can make sure all of the work does'nt fall on one person, and I know that would help ease the pain.

http://puppylinux.pastebin.com/

On that note we should look into adjusting this app for our use,
http://stud4.tuwien.ac.at/~e0225855/swap/swap.html

screenshot
http://stud4.tuwien.ac.at/~e0225855/swap/swap.png

[Nathan F]

Okay, Kenny, Lobster, you've forced my hand a little now. I didn't want to hijack your thread to say what my project was, but people are now getting right to the root of the matter. My project is so far a CLI only version of Pupget with a few customizations from me. I've been writing it from the beginning with the intention of creating an interface for it once everything is working as it should. It's also at the moment set up to work with packages that have the .pet extension, and I have the beginnings of a wrapper script that accesses it when you click on a .pet in rox. So yes, there is work being done on the concept. I know jmarsden was working on a similar project but I haven't seen him post anything to the forum in a couple months now.

I've already asked to add this to our subversion repository because I think it would be a great project to work on as a community. I've got a few more ideas I'd like to incorporate into it and some are already outlined somewhat, like basic dependency resolution and more in depth package info. I've already coded local install, web install, retrieval only, and search options. The only reason I haven't shown it to anyone yet is that I haven't coded the uninstall function yet.

A lot of the code was borrowed from Pupget, but I rewrote quite a bit and reorganized it to make it easier to maintain and separate the gui from the application, and make it easier to have other programs interact with it. A lot of the code is just my own invention, too. The main reason I used code from Pupget was because I didn't want to reinvent the wheel, so to speek.

I'll gladly work together on this or any other related project. I'd like to point out that even if we do at some point make a switch to a .pet package format that in no way means that .pups are obsolete, we have way too many of them to just discard them at this point. Heck, I still use MU's downloader a lot myself. So I'd say keep working on the downloader in some form or another. Whether that means reworking MU's downloader or going from scratch is another question entirely, and one that should probably be discussed with MU's input. For all we know he's already working on something or at least has a few ideas up his sleave, that MU is a prolific fellow after all.

Nathan

[rarsa]

I've already asked to add this to our subversion repository because I think it would be a great project to work on as a community.

Nathan,
I haven't received any request from you to add it.

Kindly send an email to svnadmin@puppylinux.org with your email and the current version of the script development folder (whatever you want to have in subversin for that project).

[Nathan F]

Okay, I resent it. I don't know what happened before, I sent an email a couple days ago. Explains why there wasn't a response.

Nathan

[rarsa]

I don't know what happened before, I sent an email a couple days ago. Explains why there wasn't a response.

Hi Nathan,

This is a servage account. Barry had warned me that the account would request confirmation as it does with his account. I didn't think it would be SO strict.

Please add antispam@servage.net to your Contacts list or "authorized senders" list or disable spam protection until you receive the confirmation email. You will only need to do this once as servage only asks for confirmation for the first email.

[klhrevolutionist]

Alright, I now have figured out how to get it to launch the dotpuprox.sh

Nathan F. if you can add your code to this to make the multiple dotpups.txt possible, we can release this.

http://puppylinux.pastebin.com/713756

http://www.GreenvilleRoad.com/puppy/Downloader.tar.gz

I have been messing around, and I believe it will be possible to install unleashed packages as well. I am testing and will report back.

[Nathan F]

I'm adding the code for this but am unable to test it right now, I'm in a hotel on dialup right now. I'll try and comment the code as well as possible to make it easy to follow. I changed the format I mentioned earlier, and it is now a comma delimited list which is cleaner to code for. Right now it's hard coded to use three repos, but it could be changed in a way to allow for as many or as few as you would want. At this point though I like it this way because it keeps the code cleaner and shorter. Okay, done.

link: http://puppylinux.pastebin.com/714257

This works (in theory, like I said, untested) by reading mirrors.txt, which must be in this format:

Code: Select all

http://site1/dotpups,ftp://site2/dotpups,http://site3/dotpups

It will then download dotpups.txt from each site and append the lines onto dotpups.txt inside your appdirectory.

I have been messing around, and I believe it will be possible to install unleashed packages as well. I am testing and will report back.

Like I said, I'm working on this myself. I've got a fully functional installer now that just lacks a gui, and the syntax for accessing it is pretty easy.

Code: Select all

pkgtool -install pkg # installs a package you've already downloaded
pkgtool -get pkg # just download the package
pkgtool -getinstall pkg # download and install the package
pkgtool -search pkg # search for a package by name

I just recoded it slightly today so you can now install without even typing the version number. It will ask for confirmation and then download and install. Uses a scheme somewhat like what I put into your code to use multiple mirrors. It will try each one for the package in turn and only gives up if none of them have it.

So I think this tool is a pretty good start if you want to install an unleashed package from the command line. There are two drawbacks right now. I need to recode the actual install function slightly for compatibility with Puppy2, and still no uninstall function.

I've got some work done on a prototype version that is designed for a new dotpet format. My version of a dotpet is a stock unleashed package, but zipped up with an md5sum and a couple other things just like a dotpup, and even click to install. So we could have built in file integrity checking and a host of other possibilities. I'm even planning to code for basic dependency resolution so it will function a bit like apt in a way. Interested?

Nathan

[Nathan F]

Please add antispam@servage.net to your Contacts list or "authorized senders" list or disable spam protection until you receive the confirmation email. You will only need to do this once as servage only asks for confirmation for the first email.

Done, hopefully you sill get the message soon.

Nathan

[Lobster]

I really like pastebin.com

- it seems an intuitive and simple way to share code. Your code as is did not seem to run Nathan - but you are coding 'blind' - so will await any changes - or maybe KLH will get it working . . .

Talking of .apt - How will this all fit in with MU's Debian installer?
Longterm it seems no one cares what is being installed from where (.sfs anyone) as long as they have programs that work. Then developers can choose which is the easiest way to package programs for Puppy or use Debian - that would be neat . . . is that feasible?

Nathan the Uninstaller - something in pupget is very important. I remember the first time I installed Linux.
After weeks I finally got it working. "At last!" I thought - then I tried to install software - another learning curve. So I left it for a few years.

Pupget is an example of how simple things can be
Add
Remove
Barrys pupget is the best installer of any distro. However . . . we have a lot more software available thanks to GuestToos .pup

Rarsa in 'Trust Metrics' everyone who is trusted is admin, or given increased powers. One of the conditions I foresaw - coding from the web, that svn does not seem to cover, is being used by Nathan. Klh has come up with a solution that works . . . simply.

What I am trying to understand is how using svn which adds an admin layer is to be used by those who are cats and those who think they need herding? (that is what the situation seems like at the moment) I know you see advantages to subversion and I trust that but as yet am wondering at the implementation . . .
What might be useful is if you added stable versions from pastebin.com to subversion? Really trying to understand this?

[klhrevolutionist]

This is great Nathan. And yes I have it working Lobster. The only issue that comes to mind is how to update mirrors.txt ?

Now as for the dotpups.txt Are they being updated each time the downloader starts ? And should we act as most other distro's and automatically do this without letting the user choose ? Note: MU gives the user the option of updating the list.

Try and implent installing unleashed pkgs.

We have come this far, in only 61 lines of code and three files.

Lobster, you are crusty with ideas.. What have you in mind ?

[Nathan F]

This is great Nathan. And yes I have it working Lobster. The only issue that comes to mind is how to update mirrors.txt ?

The idea is to provide a sane list to the user to begin with, and allow more experienced users edit it if they want to. It's a real simple format, so I don't see a problem with it getting screwed up too often. And it would be easy to patch up if it did.

Now as for the dotpups.txt Are they being updated each time the downloader starts ? And should we act as most other distro's and automatically do this without letting the user choose ? Note: MU gives the user the option of updating the list.

Yes, currently the way it's set up they would be updating at every start. You are right that this should be optional. Try using a simple yesno box in xdialog. If you need a hint go ahead and look at MU's.

Try and implent installing unleashed pkgs.

You mean implement it into this? It might be possible to recode slightly so the script can tell if it's dealing with a dotpup, a pupget, or a dotpet and call the appropriate installer. But I need to guify my unleashed package installer first, and there needs to be a consensus about what would constitute a dotpet package before that gets put into practice. I'll be working on all of those concepts.

Talking of .apt - How will this all fit in with MU's Debian installer?

I don't think it does, actually. Unless you consider that a lot of new dotpups could be made by using the Debian installer, which is exactly what Mark seems to be doing a lot of lately.

Pupget is an example of how simple things can be
Add
Remove
Barrys pupget is the best installer of any distro. However . . . we have a lot more software available thanks to GuestToos .pup

True, but seen from another standpoint it's a very inflexible system. The list of avialable packages is hard coded into the program itself at the time the iso is created, and there is no way to adjust mirrors other than hacking the script itself, and it doesn't run without X which is a big deal with something like Onebone. Plus, it's really slow. Painfully slow on an older computer.

I'm trying to address some of these issues without reinventing the wheel, and right now I feel like I'm on the right track. I'll be posting the code pretty soon.

About subversion, I think the biggest problem so far is just that nobody stepped forward right away to use it. It's almost painfully easy once you've used it a couple times. I think once a few projects are up there the benefits will speak for themselves.

You are correct though, that pastebin seems to work wonderfully, especially for a small project like this. I forsee some problems with using it on anything larger than a single script, though. When you have multiple files in a project then svn or cvs start to make a lot more sense.

Nathan

[Lobster]

This is great Nathan. And yes I have it working Lobster. The only issue that comes to mind is how to update mirrors.txt ?

I would suggest that subversion is an ideal place for a mirror list. Or perhaps the mirror list can be provided periodically as a dotpup.

For now I would work on including mirrors.txt with the simple installer (this was my idea to just have the top 10 extra packages)

===================
Top 10 Extra software add ons

OpenOffice
Firefox
Opera
Flock
NVU
Audacity
XaraLX
Java
Codecs
Cups

more software?
================

[klhrevolutionist]

Nathan F. as you were typing I was already looking at Mu's script for giving user the option of updating or not. And had already implented.

Another issue resolved...

[klhrevolutionist]

As the New DotPup-downloader is closely based on InternetRadio.pup & the wiki is going to be used at least temporarly or permanently for the mirrors.txt I have edited PizzasGood script "UpdateRadio" that allows the mirrors.txt to be updated. The acl for this page include raffy, lobster, mu, rarsa and nathan f. & myself along with Barry of course.. This is to ensure Quality Control for mirrors

http://puppylinux.org/wikka/MirrorsList

I am also expermienting with Mu's wrapper that tells when there is a new post on the forum, although I am again editing it to notify user's that the mirrors.txt wiki page has been updated. Thus allowing rarsa to create yet another jwm tray app if he will of course..... This would only be a notifier and could be edited to notify pups of other important upgrades ?

Yes, I got stuck on the implentation of installing unleashed pkgs, but Nathan f. already figured this problem out with his cli-app, which could be utilized for doing the job... Will have to wait on that, no biggie.

Other than that this project is going further than I had imagined thanks for the input and help of course. I wonder if I am missing anything ? Dear Lobster I had to take the progress bar out... Could not get it to function with the rest of the script. Maybe later.

Any other ideas please post. Will try at least....

[klhrevolutionist]

This another development release. Please look over the scripts and if you have solutions please post your reults back here. www.GreenvilleRoad.com/puppy/Downloader.tar.gz

Results
-----

AppRun

As of now AppRun is updating the dotpups.txt
Nathan f. has this setup to where it wants to use ftp. But for those
that do not have ftp myself personally, this is a problem. So it should be possible
to use http & ftp, or not ?

------

AppRun2

this is the completed part. This simply reads the dotpups.tx file and shows the
readily available dotpups that are ready to be downloaded. Future goal would be a
better interface / gui with categories and such...

------

AppRun3

Since the downloader is based upon the InternetRadio.pup, I have used Pizzasgood
companion script UpdateRadio. However, using it from the commandline you will see
that it is malfunctioning. This is caused by line 38
This scripts intention is to update the mirrors.txt file from the wiki.
Should we use the wiki or something more efficient ?

------

Notifier

The Notifier is an old script of MU, it sits on ones desktop and notifies them of
a new post, reply, topic on the forum. I have adjusted the settings to where it will
notify the user when the wiki page for the mirrors.txt has been updated. Meaning
that a new repository is available or has been removed either way it will notify.
As ubuntu has an update on their taskbar, I thought this would be a great jwm tray
app. Where one can update the mirrors.txt file and possibly be used for many more
updating functions...

------

End Result.

AppRun I only have http and not ftp, So I have no way of knowing
if it works correctly or not.

AppRun2 Still waiting to implement the unleashed installer. I personally believe the unleashed pkgs. should be using the dotpups.txt file This is to keep things less complicated.

AppRun3 if we decide to use the wiki for the mirrors.txt file then line 38 will need
to be adjusted to our needs, or I may have not entered my url correctly to the wiki ?
http://puppylinux.org/wikka/MirrorsList compare with http://puppylinux.org/wikka/RadioStations

Notifier if Apprun3 is not be used I see no reason to keep the Notifier. If AppRun3
will be used, then this will be a great sidekick to the downloader.

[Nathan F]

So it should be possible to use http & ftp, or not ?

The update is using wget, which can download using ftp or http. So it should work with http sites just as well as ftp sites. You currently have mirrors.txt in the wrong format, should look something like this:

Code: Select all

http://www.GreenvilleRoad.com/puppy,ftp://texasangel.org/pub/dotpups,http://grafpup.com/download/dotpups

In other words just a comma delimited list of sites.

Also, your dotpups.txt file on GreenvilleRoad has too many newlines at the end. The code will not work this way, it must have one, and only one, newline or it will not work right with Xdialog. So the formatting of dotpups.txt is very, very important.

I fixed a few things and simplified the code that launches the dotpup handler. It will now install a dotpup using it's real name instead of 'file.pup', and it requires a lot less code to do so. The update is attached. Note I didn't make it into an unleashed package, I just tarred up the roxapp folder. So just extract it in my-roxapps.

I've also given some thought, and what I have for an unleashed installer could work now as is. There just wouldn't be a gui. So call it from the script like this:

Code: Select all

rxvt -e pkgtool -getinstall pkgname.tar.gz

That would launch it in an rxvt window. I'll upload it in a few days after I adjust for Puppy2. Then we just need code that will determine whether it's a dotpup or an unleashed, so it can call the appropriate installer for it. Shouldn't be too hard to code this, we just need to read the file extension most likely.

Nathan