Can I cat files in random order?

[Taavi]

If I have a script like this

Code: Select all

for file in *
do
cat "$file"  >  /tmp/xxx/xxx.tex

is there a way to get the files in random order? Or should I use find and how is it done with it?

Googling googling but can't find a solution.

Taavi

[Bruce B]

random with duplicates
random with duplicates and all files not necessarily displayed
random with no duplicates
random with no duplicates and all files displayed
random with no duplicates and all files all files not necessarily displayed

yes to all, but the instructions would vary considerably

[Taavi]

Random with no duplicates and all files displayed - that's the case.

[Bruce B]

That would (could) require the most code over the other
options.

Random in programming is very different than random in
Las Vegas.

Bash 'random' generates numbers between 0 and 32767.

In the beginning we get 'hits' fairly fast, then as we near
the end of this list, we may have to run through thousands
of loops to get a match.

Keeping in mind we are not Las Vegas and we are dealing
with true random, the deck is not stacked for or against,
when true random matches and only when it matches we
get a hit.

But the computer does this fairly fast.

Files have names and they probably aren't named by
numbers.

Thinking about how to do it, I think I'd work in RAM and
generate one numbered symlink for each file name.

Each time a numbered symlink was processed delete it,
then it can never be matched again.

What I'd like (actually demand) if you want me to outline a
script, is you must be extremely specific about what you
are doing and why.

In other words you have to communicate with such clarity
that the would be programmer has no question
about what you want, all he need to is translate your
English into Bash script.

[Taavi]

Hi,
I appreciate your help. I try to explain what I'm aiming at. So first I have to translate my plans into english. Hope my englih is good enough to that.

I'm doing my writings with latex. I have made some scripts for that.

When I get ideas - maybe two lines for a song or poem or an interesting sentence I've heard - I write them down in note book or piece of paper, if I don't occur to sit by my computer. Later I write them to PC.

I collect those ideas in one directory. I save them as latex files without preampble and end tags. I use date and time with seconds for filenames. I use *.pox -ending for those files so I can open them with a special script in Rox. So those files are named for example like 20090527122130.pox.

I have a directory structure like /writings/year/pieces for these *.pox files.

I have a script to collect those files in one latex file and run latex on it and open the result in xdvi.

I put this script here.

Code: Select all

#!/bin/bash
YEAR=`date +%Y`

# delete old and make new temp directories
rm /tmp/ltx/pieces/*.*
mkdir -p /tmp/ltx/pieces

# make preamble for the *.tex file
echo "\documentclass[12pt,a4paper,finnish]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{babel}
\usepackage[left=40mm,top=30mm,right=40mm,bottom=20mm,nohead]{geometry}
\setlength{\parindent}{0mm}
\begin{document}" >> /tmp/ltx/pieces/onefile.tex

#cat *.pox files
cd /mnt/home/writings/"$YEAR"/pieces
for file in *.pox
do
cat "$file"  >>  /tmp/ltx/pieces/onefile.tex
echo "
\vspace{1mm}

\rule{50mm}{0.1mm}

\vspace{1mm}
" >>  /tmp/ltx/pieces/onefile.tex
done

# get end tag for the *.tex file
echo "
\end{document}" >> /tmp/ltx/pieces/onefile.tex

# run latex on the result and open the produced *.dvi file
cd /tmp/ltx/pieces
latex onefile.tex
xdvi -expertmode 4 -xoffset -0.1 -yoffset -0.1 -geometry 400x500 onefile.dvi

Now I'd like to get this script to cat those *.pox files in random order as I guess it could be more creative that way - connecting those little pieces differently.

I hope I wasn't as unclear as I use to be (at least my wife argues I am).

And if there's anything to correct in my script please tell me as I'm still in the beginning of learning to write these.

Taavi

[Bruce B]

Weaknesses and Strengths

A serious weakness here for me is latex format.

You seem to have a good grasp of scripting so this can make of for my bad latex knowledge.

You're written English is super.

Q: cat(ing) to a monitor can get very weird, especially when using \ as backslash has special meaning in Linux, do understand the implication of \

Q: What are the MAX number of files you will ever, ever need to process?

Q: Will their be any chance of characters which would not be legible in your viewing program?

I'll try and get some thoughts posted.

[Bruce B]

Code: Select all

#!/bin/bash
# file name is tavi for purpose of our
# discussion

# we build it by functions, the primary
# call is main - main calls other functions
# we can test functions independently by
# commenting out functions in main

main() {


	simple_random

}

simple_random() {

# : is always true so the while loop is
# infinite

# we limit the random numbers to from 
# 1 to 10000 with %10000+1
# as noted earlier RANDOM has a range
# from 0 to 32767
# this example shows how we can control
# the range although even more control
# is possible, but keeping things simple

# the important thing is your full 
# understanding of this simple loop
# play around with it 

# run it - change %value and learn
# terminate with Ctrl+C


while :
do
    let NUM=${RANDOM}%10000+1
	echo $NUM
done

}

main

[Bruce B]

Add the numbered_symlinks function to script and test but keep simple_random, I only removed it for forum posting.

Code: Select all

#!/bin/bash
main() {

#	simple_random
        numbered_symlinks
}

simple_random() {

    echo data cut for brevity

}

numbered_symlinks() {

# cd to proper directory for pox files
# to be processed


    [ ! -d /tmp/ltx ] && mkdir /tmp/ltx
    [ ! -d /tmp/ltx/links ] && mkdir /tmp/ltx/links
    [ ! -d /tmp/ltx/pieces ] && mkdir /tmp/ltx/pieces
    CDIR=`pwd`
    LINKDIR=/tmp/ltx/links
    CNT=0
    for i in *pox # or *.pox
    do
        CNT=`expr $CNT + 1`
        echo ln -sf ${CDIR}/${i} ${LINKDIR}/${CNT}
        # ln -sf ${CDIR}/${i} ${LINKDIR}/${CNT}
    done

# Explained, use the echo first to see it it
# looks good. Then uncomment the actual command for
# a trial run

# Purpose in numbered symlinks: Simply to make
# (random + all + one time only) easier to implement

# Note: not tested at all

}

main

[Taavi]

Hi,

and thanks for teaching me this.

Now I have here function to get random numbers in the range I can set.

Then I have other one to get numbered symlinks for files in a directory. That worked fine. (And I have learned other way to delete and make temp directories.)

I guess the next step is to use simple_random to get symlinks listed in random order so that every numer turns up just once?

[Bruce B]

Attached bash official reference manual for 4.xx

Also, I'm using;

GNU bash, version 4.0.10(1)-release (i686-pc-linux-gnu)

And it's working great with Puppy

[Bruce B]

Taavi,

You're clearly are a step ahead.

The primary objective for me is your learning and looks like you are doing it.

My next step will be a slightly more complex function, which understanding the building blocks should be easy to read, modify and control.

I expect to present it within 24 hours.

Ultimately it will be you polishing up the template for your use.

I don't know your time schedule but look back any time within the next 24 from time of this post.

Bruce

[Bruce B]

(And I have learned other way to delete and make temp directories.)

I notice I didn't comment on this so here is the explanation

[ is an actual command
] is a notation that [ is completed

[ is synonymous with 'test' which is also a command

[ is internal to bash and Puppy also has an external [ command

When we have an internal command and an external command the internal command is used.

Results from expression tests are true or false

[ ! -d /tmp/ltx ] && mkdir /tmp/ltx

! means not
-d means directory
the test is to see if the directory /tmp/ltx does NOT exist.

if it does not exist, the result is true, which allows the && to execute and make the directory.

Anything other than true and the command following the && will not execute.

[Bruce B]

Taavi,

In order to work on the script, I made another script to
batch rename files to your extension.

I can write a script like that in just a matter of a few
minutes. If there are bugs or features wanted I can do
that later.

When sharing scripts for others, I like to do a lot of
testing. Meaning to say: I take a lot of extra time.

In this particular case I also commented the script very
well.

If I did a good job, you should be able to read the script
and follow it with good understanding. If anything is not
understood please ask.

The script should be useful to yourself and others,
because renaming multiple file extensions on the
command line isn't so simple as mv *.txt *.doc

Hoping you and others find the script useful as well as
instructive. Script attached as .zip file

The # followed by text are instructional only

Bruce

Comment script

Code: Select all

#!/bin/bash

# a bash script is read from top to bottom
# it executes commands in the same manner

# the functions are read in and become
# programs within the script

# note the first command is the last line
# of the file, which is the call to main

# you can follow the entire program flow
# by reading the commands in main

# in outlining the program, I wanted to
# (first) verify the arguments and show
# help and exit as needed.

# (second) if arguments seem good then
# show the  user a preview of the changes
# to be made before actually making them

# (third) get user explicit consent
# before making the changes

# (forth) if it is good to go, make the
# changes

# note you can do things like rename
# filenames.pet filenames.pet.tar.gz


main() {

    checkarguments "$@" # exits on error
    preview "$@"
    permission # exits if permission not
    # granted

    # below is an unnecessary double check
    # uses of x is a way of making sure
    # either side of the comparision is
    # never empty, quotes are also commonly
    # used and shown in this script
    
    [ x$REPLY = xy ] && mvextension "$@"
    echo "" # for formatting only

}


checkarguments() {

    # in this example we will nest functions.
    # 'showhelp' will be a function within
    # 'checkarguments'


    showhelp() {

    # $0 is the full path and name of program
    # basename $0 is only the name of the
    # program - stripping the path makes for
    # nicer looking on screen display

    # using the backslash \" allows quoting
    # of the program name

    # the initial name of the program is mvext
    # but the script is structured such that
    # it can be renamed

    
    prgm=`basename $0`

      echo ""
      echo "Purpose of \"$prgm\" is to batch rename"
      echo "file extensions"
      echo
      echo "Usage: \"$prgm\" <extension> <newextension>"
      echo ""
      echo "Note: \"$prgm\" will not make any changes or"
      echo "overwrite files without your consent"
      echo ""
      exit

    }


    # if either are true help is shown
    # the "" are used to make sure neither side
    # of our string comparison test is ever
    # empty

    # the purpose of the argument check is to
    # prevent the program from running in the
    # absence of an argument and show help
    # when it is obvious the user wants or
    # needs help

    # example usage of double string comparisons
    # -o means "or" if either test is true
    # showhelp runs
    
    [ "$1" = "" -o "$2" = "" ] && showhelp
    [ "$1" = "--help" -o "$1" = "-h" ] && showhelp

}


preview() {

    # for loop will process all files matching our
    # criteria of *$1
    
    # basename is used for stripping filenames to
    # our criteria - see man basename or
    # basename --help for more information

    # we bracket variables such as $i to ${i} to
    # prevent any confusion by bash as to the
    # variable name

    for i in *"$1"
    do
        fn=`basename $i "$1"`
        echo " mv ${fn}"$1" ${fn}"$2""
    done

}

permission() {

    # read used without specifying a variable, makes
    # a variable called REPLY, which will be used

    # the 'read' command is for getting user input
    # during the operation of the program

    read -n 1 -p " Proceed with actual renaming [y,n] "
    [ x$REPLY != xy ] && echo "" && exit

}

mvextension() {


    # in this function we do the actual renaming,
    # the --interactive will halt in the event of
    # a potential overwrite
    
    for i in *"$1"
    do
        fn=`basename "$i" "$1"`
        mv --interactive --verbose ${fn}"$1" ${fn}"$2"
    done

}

main "$@"

Script with no comments which is
easier to read for some

Code: Select all

#!/bin/bash

main() {

    checkarguments "$@"
    preview "$@"
    permission
    [ x$REPLY = xy ] && mvextension "$@"
    echo ""

}


checkarguments() {

    showhelp() {

    prgm=`basename $0`

      echo ""
      echo "Purpose of \"$prgm\" is to batch rename"
      echo "file extensions"
      echo
      echo "Usage: \"$prgm\" <extension> <newextension>"
      echo ""
      echo "Note: \"$prgm\" will not make any changes or"
      echo "overwrite files without your consent"
      echo ""
      exit

    }

    
    [ "$1" = "" -o "$2" = "" ] && showhelp
    [ "$1" = "--help" -o "$1" = "-h" ] && showhelp

}


preview() {

    for i in *"$1"
    do
        fn=`basename $i "$1"`
        echo " mv ${fn}"$1" ${fn}"$2""
    done

}

permission() {

    read -n 1 -p " Proceed with actual renaming [y,n] "
    [ x$REPLY != xy ] && echo "" && exit

}

mvextension() {

    for i in *"$1"
    do
        fn=`basename "$i" "$1"`
        mv --interactive --verbose ${fn}"$1" ${fn}"$2"
    done

}

main "$@"

[Taavi]

Thanks,

I was in a course for amateur theatre players and get just home. I'll examine this one more closely tomorrow morning. It seems that you have commented it so clearly that I should understand how it works.

I have downloaded the bash reference, thanks for that too.