speeding up scripts
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
Thanks to amigo for the read line trick.
I used it to throw together a jwm_menu_create script that completes in 1/15th the time of fixmenus using only shell commands (and one echo -e, because some of Puppy's busyboxes don't have printf)
usage:
jwm_menu_create > ${HOME}/.jwmrc && jwm -restart #by default it will just go to stdout
Todo:
1. What the heck is up the default*.desktop file stubs - do I need a work around or are they misplaced/malformed (they are missing most entries)
2. Localization - just needs another case statement, but few .desktop files support it anyways ... and my Locale is already default
Edit It is now in this thread
http://www.murga-linux.com/puppy/viewtopic.php?t=70804
I used it to throw together a jwm_menu_create script that completes in 1/15th the time of fixmenus using only shell commands (and one echo -e, because some of Puppy's busyboxes don't have printf)
usage:
jwm_menu_create > ${HOME}/.jwmrc && jwm -restart #by default it will just go to stdout
Todo:
1. What the heck is up the default*.desktop file stubs - do I need a work around or are they misplaced/malformed (they are missing most entries)
2. Localization - just needs another case statement, but few .desktop files support it anyways ... and my Locale is already default
Edit It is now in this thread
http://www.murga-linux.com/puppy/viewtopic.php?t=70804
Last edited by technosaurus on Tue 16 Aug 2011, 09:00, edited 1 time in total.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].
Thanks, this is very educational. I don't use the jwm so the script isn't much help for me, but the idea behind it is very helpful for future projects.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]
Once again I started a thread I'm too thick to follow This thread went over my head about 2 pages ago.. I'm gonna read, re-read, and re-read again until it makes sense! But I'm still loving the results!jamesbond wrote:Thanks, this is very educational.
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]
- thunor
- Posts: 350
- Joined: Thu 14 Oct 2010, 15:24
- Location: Minas Tirith, in the Pelennor Fields fighting the Easterlings
- Contact:
Re: speeding up scripts
Does <action>exec $SHELL -c 'my_function_name'</action> solve it? Somebody using Ubuntu+dash reported a similar problem and this was his fix.sc0ttman wrote:1. Is it possible to execute a command defined in the main script, from within a GTK-Dialog button? If so, that would be great, but at the moment, anything like this in the GTKDialog GUIsimply returns (in the terminal)Code: Select all
<button> <action>my_function_name</action> </button>
Code: Select all
my_function_name: command not found
Regards,
Thunor
Re: speeding up scripts
I'll have a look, thanks for the tipthunor wrote:Does <action>exec $SHELL -c 'my_function_name'</action> solve it? Somebody using Ubuntu+dash reported a similar problem and this was his fix.sc0ttman wrote:1. Is it possible to execute a command defined in the main script, from within a GTK-Dialog button? If so, that would be great, but at the moment, anything like this in the GTKDialog GUIsimply returns (in the terminal)Code: Select all
<button> <action>my_function_name</action> </button>
Code: Select all
my_function_name: command not found
Regards,
Thunor
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]
Hello,
you can simplify in:
it's because of sub shell, in Ubuntu is dash
and dash does not support "export-f"
do the same for <input>
if <input> is a function
you can simplify in:
Code: Select all
<action>bash -c 'my_function_name'</action>
and dash does not support "export-f"
do the same for <input>
if <input> is a function
speeding up scripts using arrays
when you have correctly formatted files there is another way to pull out values
*so if you start out creating a correctly formatted file
getting info is easy
here is an example using a .desktop
we all know that desktops are not formatted correctly or is there an official
line by line template that is used by all linux distros
but we could make a template
then read it with this
using shell and arrays
--------------------------------------------------------------------------------------------
sample output
# echo ${desktop_array[0]//|/ }
[Desktop Entry]
#
# echo ${desktop_array[1]//|/ }
Encoding=UTF-8
#
# echo ${desktop_array[2]//|/ }
Name=Axel download accelerator
#
# echo ${desktop_array[3]//|/ }
Icon=mini-ftp.xpm
#
# echo ${desktop_array[4]//|/ }
Comment=Axel download accelerator
#
# echo ${desktop_array[5]//|/ }
Exec=puppydownload
#
# echo ${desktop_array[6]//|/ }
Terminal=false
#
# echo ${desktop_array[7]//|/ }
Type=Application
#
# echo ${desktop_array[8]//|/ }
Categories=X-Internet
#
# echo ${desktop_array[9]//|/ }
GenericName=Axel download accelerator
#
when you have correctly formatted files there is another way to pull out values
*so if you start out creating a correctly formatted file
getting info is easy
here is an example using a .desktop
we all know that desktops are not formatted correctly or is there an official
line by line template that is used by all linux distros
but we could make a template
then read it with this
using shell and arrays
--------------------------------------------------------------------------------------------
Code: Select all
# advanced arrary use from info in advanced bash scripting modified by Joe Arose
# for using correctly formatted templates
# this separates the strings using a pipe symbol this allows for spaces in strings to be read
# to later be converted back to spaces
# if not done this way every space is a new arrary value causing undesired word splitting
# this fixes many problems with getting just one line of information
#this is beautiful it replaces the pipe for a space
#echo ${desktop_array[4]//|/ }
#how that reads is echo array 4 and //substitue all , pipe symbols, with a space
# it looks a bit like sed but using only the shell
# set this to what you want
fileplace=/usr/share/applications/
filename=Axel-download-accelerator.desktop
desktop_array=( `cat "$fileplace$filename" | tr ' ' '|'`)
echo ${desktop_array[@]//|/ }
echo ${desktop_array[0]//|/ }
echo ${desktop_array[1]//|/ }
echo ${desktop_array[2]//|/ }
echo ${desktop_array[3]//|/ }
echo ${desktop_array[4]//|/ }
echo ${desktop_array[5]//|/ }
echo ${desktop_array[6]//|/ }
echo ${desktop_array[7]//|/ }
echo ${desktop_array[8]//|/ }
echo ${desktop_array[9]//|/ }
echo ${desktop_array[10]//|/ }
sample output
# echo ${desktop_array[0]//|/ }
[Desktop Entry]
#
# echo ${desktop_array[1]//|/ }
Encoding=UTF-8
#
# echo ${desktop_array[2]//|/ }
Name=Axel download accelerator
#
# echo ${desktop_array[3]//|/ }
Icon=mini-ftp.xpm
#
# echo ${desktop_array[4]//|/ }
Comment=Axel download accelerator
#
# echo ${desktop_array[5]//|/ }
Exec=puppydownload
#
# echo ${desktop_array[6]//|/ }
Terminal=false
#
# echo ${desktop_array[7]//|/ }
Type=Application
#
# echo ${desktop_array[8]//|/ }
Categories=X-Internet
#
# echo ${desktop_array[9]//|/ }
GenericName=Axel download accelerator
#
I modified amigo's code to make desktops now write to and read from an array
*thanks for that snippet Gilbert it was really clever using the case for a pre filter grep and very fast
original code from amigo below
*thanks for that snippet Gilbert it was really clever using the case for a pre filter grep and very fast
Code: Select all
> test.file
for DESKTOP_FILE in /usr/share/applications/* ; do
#for DESKTOP_FILE in /usr/share/applications/Editra.desktop ; do
while read LINE ; do
case $LINE in
Name=*) NAME="${LINE[@]}"'|' ;;
Icon=*) ICON="${LINE[@]}"'|' ;;
#Terminal=*)
#Type=*)
Categories=*) CATS="${LINE[@]}"'|' ;;
Exec=*) EXEC="${LINE[@]}"'|' ;;
Comment=*) COMM="${LINE[@]}"'|' ;;
esac
done < $DESKTOP_FILE
echo $NAME$ICON$CATS$EXEC
# To test the extract function below, use the following line instead of above
echo '[Desktop_Entry]|'$NAME$ICON$CATS$EXEC$COMM | tr ' ' '_'>> test.file
done
desktop_array2=( `cat "$HOME/test.file"`)
# read line by line of the formatted array
# one problem was the Name= and the comment didnt get the spaces converted into
# the pipe symbol because the pipe was after the string with spaces
# but this gets fixed with underscores in the last echo using tr ' ' '_'
# how to read line by line from a correctly forrmatted array now
# just a sample of 5 diferent desktops read from the array desktop_array2
echo ${desktop_array2[0]//|/ } | tr ' ' '\n' | tr '_' ' '>testing0.txt
echo ${desktop_array2[1]//|/ } | tr ' ' '\n' | tr '_' ' '>testing1.txt
echo ${desktop_array2[2]//|/ } | tr ' ' '\n' | tr '_' ' '>testing2.txt
echo ${desktop_array2[3]//|/ } | tr ' ' '\n' | tr '_' ' '>testing3.txt
echo ${desktop_array2[4]//|/ } | tr ' ' '\n' | tr '_' ' '>testing4.txt
original code from amigo below
Code: Select all
> test.file
for DESKTOP_FILE in /usr/share/applications/* ; do
#for DESKTOP_FILE in /usr/share/applications/Editra.desktop ; do
OUT=
while read LINE ; do
case $LINE in
Name=*) NAME="${LINE#*=}"'|' ;;
Comment=*) OUT=$OUT"${LINE#*=}"'|' ;;
Icon=*) ICON="${LINE#*=}"'|' ;;
#Terminal=*)
#Type=*)
Categories=*) CATS="${LINE#*=}"'|' ;;
Exec=*) EXEC="${LINE#*=}"'|' ;;
#Comment=*) COMM="${LINE#*=}" ;;
esac
done < $DESKTOP_FILE
echo $NAME$ICON$CATS$EXEC
# To test the extract function below, use the following line instead of above
# echo $NAME$ICON$CATS$EXEC >> test.file
done
Last edited by big_bass on Fri 19 Aug 2011, 23:39, edited 3 times in total.
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
I only use bash
* but if you are using another shell this runs only in bash*
you can take advantage of arrays using bash
(I credited amigo in a prior thread for the code base and pasted the original code a few posts ago )* I will include a diff patch for clarity
I put a safety in there all files get generated in /tmp/desktop
so if you are happy with the results you could manually copy over the
old desktops (thats your call) the user woud have to repackage all the packages with the new desktops to do the job right
I replaced the underscore with the "+" because some files may use underscore in names
* but if you are using another shell this runs only in bash*
you can take advantage of arrays using bash
(I credited amigo in a prior thread for the code base and pasted the original code a few posts ago )* I will include a diff patch for clarity
I put a safety in there all files get generated in /tmp/desktop
so if you are happy with the results you could manually copy over the
old desktops (thats your call) the user woud have to repackage all the packages with the new desktops to do the job right
I replaced the underscore with the "+" because some files may use underscore in names
Code: Select all
#!/bin/bash
make a template of all the desktops and lets you view them in /tmp/desktops
so none of your original desktops get overwritten
builds an array for speeding up any scripts that search info from the desktop
now all your desktops will have a format and an organized template
the first line is [Desktop Entry],Name,Icon,Categories,Exec,Comment
and thats what you will expect to see when you search the desktops for info
#[Desktop Entry]
#Name=AbiWord
#Icon=/usr/share/icons/abiword_48.png
#Categories=Application;Office;WordProcessor;GNOME;GTK;X-Red-Hat-Base;
#Exec=abiword
#Comment=Compose, edit, and view documents
# make a template of the desktops regenerate all desktops to the new simple template
# removes poorly formatted desktops and creates a standard which allows later for easy reading of strings
# into an array to speed up scripts since the newly generated desktops maintain a standard format
# less commands are needed to filter data for output this is where all the time is wasted
# having to parse poorly formatted files from the start if you have organized files
# everything is fast and easy to parse the code
:>arraytest.txt
mkdir -p /tmp/desktops
for DESKTOP_FILE in /usr/share/applications/* ; do
#for DESKTOP_FILE in /usr/share/applications/Editra.desktop ; do
while read LINE ; do
case $LINE in
Name=*) NAME="${LINE[@]}"'|' ;;
Icon=*) ICON="${LINE[@]}"'|' ;;
#Terminal=*)
#Type=*)
Categories=*) CATS="${LINE[@]}"'|' ;;
Exec=*) EXEC="${LINE[@]}"'|' ;;
Comment=*) COMM="${LINE[@]}"'|' ;;
esac
done < $DESKTOP_FILE
echo $NAME$ICON$CATS$EXEC
# To test the extract function below, use the following line instead of above
# fixes spaces in the string names by replacing them with a "+" making a correctly formatted array
echo '[Desktop+Entry]|'$NAME$ICON$CATS$EXEC$COMM | tr ' ' '+' >>arraytest.txt
#uncomment if you want to generate all new desktops in /temp/desktops
echo '[Desktop+Entry]|'$NAME$ICON$CATS$EXEC$COMM | tr ' ' '+' | tr '| ' ' ' | tr ' ' '\n' | tr '+' ' '>/tmp/desktops/`basename $DESKTOP_FILE`
done
- Attachments
-
- amigo-orig.patch.gz
- this is only to show the differences between amigos original work and my modifications
- (737 Bytes) Downloaded 291 times
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
here is an example I cooked up that includes recursion, string manipulation, and integer math
A=$(($1/16))
$1 is the input, this stores the "div" of the input by 16 (div is integer only without a remainder so 15 div 16 is 0 but 17 div 16 is 1)
HEX=0123456789ABCDEF
this shows how strings are really just an array of characters
[ $A -gt 15 ] && dec2hex ${A} || printf ${HEX:$A:1}
this is the recursive part, if the "div" is greater than 16, then we haven't gone enough hex place values, so call ourself with the div to shift back one ... note that nothing further happens until the last place value is reached (div is < 16) and the all recursive calls to dec2hex return... this last one only will print its div (in HEX format) the value after the first ":" is the starting point of the substring, and the value after the second ":" is the length of the substring
printf ${HEX:$(($1%16)):1}
similar to above printf statement except that it prints the "mod" (the remainder of input div 16) ... notice that all recursive calls will execute this code
now just to show the value of using functions instead of external scripts
try it like this:
Code: Select all
#!/bin/ash
A=$(($1/16))
HEX=0123456789ABCDEF
[ $A -gt 15 ] && dec2hex ${A} || printf ${HEX:$A:1}
printf ${HEX:$(($1%16)):1}
$1 is the input, this stores the "div" of the input by 16 (div is integer only without a remainder so 15 div 16 is 0 but 17 div 16 is 1)
HEX=0123456789ABCDEF
this shows how strings are really just an array of characters
[ $A -gt 15 ] && dec2hex ${A} || printf ${HEX:$A:1}
this is the recursive part, if the "div" is greater than 16, then we haven't gone enough hex place values, so call ourself with the div to shift back one ... note that nothing further happens until the last place value is reached (div is < 16) and the all recursive calls to dec2hex return... this last one only will print its div (in HEX format) the value after the first ":" is the starting point of the substring, and the value after the second ":" is the length of the substring
printf ${HEX:$(($1%16)):1}
similar to above printf statement except that it prints the "mod" (the remainder of input div 16) ... notice that all recursive calls will execute this code
now just to show the value of using functions instead of external scripts
try it like this:
Code: Select all
#!/bin/ash
dec2hex(){
A=$(($1/16))
HEX=0123456789ABCDEF
[ $A -gt 15 ] && dec2hex ${A} || printf ${HEX:$A:1}
printf ${HEX:$(($1%16)):1}
}
dec2hex $1
- # time dec2hex 999999999999999999
DE0B6B3A763FFFF
real 0m0.034s
user 0m0.032s
sys 0m0.024s
# time dec2hex 999999999999999999
and if you want a generalized format for any baseCode: Select all
#!/bin/ash #note that base64 is traditionally A...Za...z0...9+/ (yeah wtf) dec2baseN(){ A=$(($1/$2)) STRING=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/ [ $A -ge $2 ] && dec2baseN ${A} $2 || printf ${STRING:$A:1} printf ${STRING:$(($1%$2)):1} } dec2baseN $1 $2
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].
Does anyone know: if you source another file, are functions from it run in a new subshell? I don't see why they would be... but I don't know.
Do you know a good gtkdialog program? Please post a link here
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
- technosaurus
- Posts: 4853
- Joined: Mon 19 May 2008, 01:24
- Location: Blue Springs, MO
- Contact:
No, but it does take considerably longer, due to the extra file read.
Things that should be sourced include (not limited to...just examples
Localization (b/c you are only source 1 of X)
Configuration files (can be used/modified by other programs or the user)
A single file that contains all of your needed functions.
Things that usually shouldn't be sourced
Lots of files with a single or a few functions (each read adds time)
A self generated file (you can normally use a variable)
Things that should be sourced include (not limited to...just examples
Localization (b/c you are only source 1 of X)
Configuration files (can be used/modified by other programs or the user)
A single file that contains all of your needed functions.
Things that usually shouldn't be sourced
Lots of files with a single or a few functions (each read adds time)
A self generated file (you can normally use a variable)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].
Ah, thanks, I confirmed that by testing, tooNo
I don't know if anybody here would find it useful, but I see the Arch people maintain "A library providing UI functions for shell scripts"
https://github.com/Dieterbe/libui-sh#readme
Do you know a good gtkdialog program? Please post a link here
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
When you source a file, it is just as if you 'pasted' the code from it into wherever you sourced in your calling script.
I have one very large shell project -well over 10,000 lines of code. It used all be in one file, but when it got over 3,000 lines I began splitting it up. I do like using functions greatly and name the files after the main function they contain. But, I wouldn't do this for a single small function. In my large project, some of the files containing functions are sourced selectively, according to whether the functions are needed or not.
About the time needed for reading the file in -it takes time each time you check a file attribute using 'test' or '[' and we never shy from that right? And, reading in the contents of the file won't take much longer. And it usually takes much less time than starting some external program -the startup latency from lodaing that into the cache and linking usually takes longer than simply opening a file for read.
Thanks for mentioning 'libui-sh'. I don't think I had seent had one before. There are a couple of others, also, which include a standard set of dialog boxes that will work with several different tools -dialog, Xdialog, zenity, gdialog or even just plain shell. I've found that most of them are too big and overreaching. The ideal setup for creating dialogs would cover a few standard screens like menu-selection, input-box, info-box, yes-no, etc.
I've put lots of time into learning to cerate faster scripts by focusing on replacing often-used standard tools like basename, dirname, cut, cat, etc., with small blocks of bash code.
basename?
${VAR##*/}
dirname?
${VAR%/*}
cat?
while something ; do
done < file-you-wanted-to-cat
I have one very large shell project -well over 10,000 lines of code. It used all be in one file, but when it got over 3,000 lines I began splitting it up. I do like using functions greatly and name the files after the main function they contain. But, I wouldn't do this for a single small function. In my large project, some of the files containing functions are sourced selectively, according to whether the functions are needed or not.
About the time needed for reading the file in -it takes time each time you check a file attribute using 'test' or '[' and we never shy from that right? And, reading in the contents of the file won't take much longer. And it usually takes much less time than starting some external program -the startup latency from lodaing that into the cache and linking usually takes longer than simply opening a file for read.
Thanks for mentioning 'libui-sh'. I don't think I had seent had one before. There are a couple of others, also, which include a standard set of dialog boxes that will work with several different tools -dialog, Xdialog, zenity, gdialog or even just plain shell. I've found that most of them are too big and overreaching. The ideal setup for creating dialogs would cover a few standard screens like menu-selection, input-box, info-box, yes-no, etc.
I've put lots of time into learning to cerate faster scripts by focusing on replacing often-used standard tools like basename, dirname, cut, cat, etc., with small blocks of bash code.
basename?
${VAR##*/}
dirname?
${VAR%/*}
cat?
while something ; do
done < file-you-wanted-to-cat
Does anyone know if there is a difference between something like `sed -i ... somefile` and `sed ... < somefile`?
I don't think there is, but then why are there both options? Is it because the < isn't available in certain shells or something?
I don't think there is, but then why are there both options? Is it because the < isn't available in certain shells or something?
Do you know a good gtkdialog program? Please post a link here
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
the sed -i
makes the changes in the "original file" permanent
and the other command without the -i just lets you see what would happen
only the terminal shows the results the original file is unchanged
sed is a great tool but... it looks like taking modem noise and then converting it
into a human readable language
sed <modem_noise # is used to generate the command options
makes the changes in the "original file" permanent
and the other command without the -i just lets you see what would happen
only the terminal shows the results the original file is unchanged
sed is a great tool but... it looks like taking modem noise and then converting it
into a human readable language
sed <modem_noise # is used to generate the command options
- Attachments
-
- sed1line.txt.gz
- (6.87 KiB) Downloaded 270 times
True! I guess I should be in bed. The reason I wasn't seeing any difference is because the line in the script I was working on wasn't actually necessary for the data I was testing it with
Thanks.
Thanks.
Do you know a good gtkdialog program? Please post a link here
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
Classic Puppy quotes
ROOT FOREVER
GTK2 FOREVER
- thunor
- Posts: 350
- Joined: Thu 14 Oct 2010, 15:24
- Location: Minas Tirith, in the Pelennor Fields fighting the Easterlings
- Contact:
I've got some fast portable shell scripting tips that I'd like to note.
I'm off out in a minute so I'll add some more later.
Regards,
Thunor
Code: Select all
## With bash you can read the first line from a file speedily like this:
echo `<file`
## This you might think is the ash/dash solution:
cat file
## But I've actually found this to be the fastest for ash, bash and dash:
read -r input < file
echo $input
Regards,
Thunor
Last edited by thunor on Sun 04 Dec 2011, 23:16, edited 1 time in total.