process:  download files (script: base-get-book.sh )
    #####################################
          wget -O INFO-1 "http://www...page 1
          wget -O INFO-2 "http://www...page 2
          wget -O INFO-3 "http://www...page 3
    #####################################
  process:  strip out formatting to get specific content
     (script: extract-text-book.sh)
    #####################################
    #!/bin/bash
    #####################################
    mkdir INFO
    for x in `ls INFO-*`
        do
    #####################################
    cat $x | grep "text $x" | sed 's/New American/ New American/g' | \
            sed 's/[^<]*<\/span>/ /g'  | \
            sed 's/<[^>]*>//g' | sed 's/\ [[^]*\]]//g' | sed 's/ \([1-9]\)/\n\1/g'  > $x.txt
    #####################################
    mv $x INFO
        done
    #####################################
  process:  make sure all files have proper names for sorting
    ls > fix
    #####################################
    vi fix
        remove lines that have digits that will sort: e.g.   filename-10.txt
        leave lines that need to have a zero added, then:
    :%s/.*/mv & &/g
        results:     mv filename-1.txt filename-1.txt
        then cursor to the number on the right, and insert a 0 before 1, to get:  filename-01.txt
        repeat (usually 1-9) until done.
    :wq
    #####################################
sh ./fix ; rm -f fix
    #####################################
    ls -al (should show all files sorted nicely)
    #####################################
  process:  remove remaining internal brackets
    use script to remove brackets:
                sh ./remove-brackets.sh *
    #####################################
   script: remove-brackets.sh
    #!/bin/bash
    echo "remove bracketed footnotes"
    perl -pi -e 's/\[.\]//g' *.txt
    perl -pi -e 's/\[..\]//g' *.txt
    #####################################
 process:  edit all the files to remove errors, line up on one line
        vi INFO-*
    after making sure there is a number on the left and all the text on one line, then
    :%s/.*/INFO 9:&/g
    :wn
    :%s/.*/INFO 10:&/g
    :wn
    :%s/.*/INFO 11:&/g
    :wn
        #  where INFO is the book or file name, and 9 is the chapter.
        # The ampersand places the text string with line number to the right
        repeat until done editing all files
    #####################################
 process:  create unique summary file and then copy to other locations
    cat INFO-* > INFO_number.txt
    copy files to proper directories and other servers:
    cp INFO-*.txt ../SORTED-INFO
    scp -r ../SORTED-INFO/ 192.168.11.11:/home/luser/FILES/SORTED-INFO
    or
     rsync -r ../SORTED-INFO/ 192.168.11.11:/home/luser/FILES/SORTED-INFO
    #####################################
 repeat until all 1,189 chapters are cleaned up and sorted...
    then test to see that there will be 31,102 lines, and based on the version, 781,621 words
    cat INFO-* > total-info.txt ; cat total-info.txt | wc
(note: as of 3/7/2017 I'm on 14/66)
    

1. A quick comparison of commands for MS use:
   http://johnmeister.com/linux/Microsoft/PowerShell-vs-BASH-commands-compared.html
2. http://johnmeister.com/linux/Scripts/wget-perl-WX-details.html
3. http://johnmeister.com/linux/Scritps/perl-fix-spaces-remove-brackets.sh.html
4. the art of Linux Sys Admin outline, sample vi video
5. SHELL - history, process, commands
6. discussion on ASCII vs. UTF-8, etc.

#!/bin/bash
echo "remove bracketed footnotes"
perl -pi -e 's/\[.\]//g' *.txt
perl -pi -e 's/\[..\]//g' *.txt

###############################################################################################
#!/bin/bash
echo "remove bracketed footnotes, fixed commas and semicolon spacing"
perl -pi -e 's/\[.\]//g' *.txt  # remove bracketed text such as a footnote
sperl -pi -e 's/\[..\]//g' *.txt # remove bracketed text such as a footnote
perl -pi -e 's/,/, /g' *.txt    # adds a space after a comma
perl -pi -e 's/,  /, /g' *.txt  # removes any extra spaces after a comma after adding a space
perl -pi -e 's/;/; /g' *.txt    # adds a space after a semicolon
perl -pi -e 's/:/: /g' *.txt    # adds a space after a colon
perl -pi -e 's$\.$. $g' *.txt   # adds a space after a period
perl -pi -e 's/?/? /g' *.txt    # adds a space after a question mark
###############################################################################################
#  the following sed and perl string replace all the lines below, used brackets for upper-case
###############################################################################################
sed -i 's/[A-Z]/ /g             # add space before uppercase letter
perl -pi -e 's/  / /g           # change two spaces to one
###############################################################################################
# perl -pi -e 's/A/A /g' *.txt    # adds a space after an uppercase A
# perl -pi -e 's/A  /A /g' *.txt  # removes any extra spaces before an uppercase A
# perl -pi -e 's/T/T /g' *.txt    # adds a space after an uppercase T
# perl -pi -e 's/T  /T /g' *.txt  # removes any extra spaces before an uppercase T
# perl -pi -e 's/W/W /g' *.txt    # adds a space after an uppercase W
# perl -pi -e 's/W  /W /g' *.txt  # removes any extra spaces before an uppercase W
# perl -pi -e 's/Y/Y /g' *.txt    # adds a space after an uppercase Y
# perl -pi -e 's/Y  /Y /g' *.txt  # removes any extra spaces before an uppercase Y
###############################################################################################
#   need to figure out bracket in perl:  perl -pi -e  's/(.*)/<img src=\"$1\"><br>$1<hr>/'
###############################################################################################

   The filter for the numbers doesn't always get it right, so I've added the filters above. 

    #####################################
    cat $x | grep "text $x" | sed 's/New American/ New American/g' | \
            sed 's/[^<]*<\/span>/ /g'  | \
            sed 's/<[^>]*>//g' | sed 's/\ [[^]*\]]//g' | sed 's/ \([1-9]\)/\n\1/g'  > $x.txt
    #####################################

MET on Thursday this week... normally on Friday...

Notes and the script from the session:

--> more merge-files-project.sh.txt
#!/bin/bash
#   john 30 Mar 2017 merge-files-project.sh.txt
#  three objectives:
# 1) file 1 first, then file 2
    (add new line in process to create combined file) - maybe paste isn't the right tool?
# 2) copy files, then annotate each line with designator - file1X.txt file2Y.txt
# 3) preserve original, annotated and combined in plain text
        - file1, file1X, file2, file2X AND:  BOTHfiles.txt
# 4) create file with html table to show cells for each line,
        two cells in one row... 4.3M files! - BOTHfiles.html
#  paste -d '\n' file1.txt file2.txt
# perl -pi -e  's/(.*)/<img src=\"$1\"><br>$1<hr>/'
# KJV.txt NAS.txt
################################################
cp KJV.txt K1.txt
cp NAS.txt N1.txt
paste K1.txt N1.txt > KN-test1.txt   #  file with one long line for both files
paste -d '\n' K1.txt N1.txt > KN-test2.txt
        #  file with two separate lines in sequence and in order
################################################
#   sed 's/$/ [KJV]/' <KJV.txt >KVJb.txt
#   perl -pe 's/$/ (kjv)/' <KJV.txt >KVJb.txt
#    need to find global replacement, and also how to insert into a string, if possible
################################################
cp KJV.txt KJVb.txt
cp NAS.txt NASb.txt
perl -pi -e  's/(.*)/$1(kjv)/' KJVb.txt
perl -pi -e  's/(.*)/$1(nas)/' NASb.txt
paste -d '\n' KJVb.txt NASb.txt > KNb.txt   #  file with two separate lines in sequence and in order
head KNb.txt                        # put a head command in script to test
read -p 'hit enter'
################################################
sed -i 's$.*$<tr><td>&<p></td>$g' K1.txt
        # using sed to add html tags on each line
sed -i 's$.*$<td>&<p></td></tr>$g' N1.txt
        # using sed to add html tags on each line
head -n 3 K1.txt
head -n 3 N1.txt
read -p 'hit enter'
#######################
echo "<html><title>TEST</title><center><table border='10' \
        width='95%'>" > KN1.txt-testX.html
paste K1.txt N1.txt >> KN1.txt-testX.html #  the paste command reads one line from each file
echo "</table></center> <a href='copyright-info.txt'>copyright info</a></html>" \
    >> KN1.txt-testX.html
   # adding closing table and link
#######################
#  perl -pi -e 's$Dan 9:$<b><em>Dan 9:</em></b>$g' KN27-Daniel-Ch9.html
    # decided to emphasis the book and chapter
# head KN27-Daniel-Ch9.html                        # put a head command in script to test
#######################
head KN1.txt-testX.html                        # put a head command in script to test
#######################

Notes for this session:

 
 
 win10 freebie: ALL COMMANDS from Win explorer:  mkdir "c:\ALL_CMDS.{ED7BA470-8E54-465E-825C-99712043E01C}"
http://johnmeister.com/linux/Microsoft/SHOW_ALL_COMMANDS-win7-8-10.html 

 got MATE working on SuSE:   --> cat /etc/sysconfig/displaymanager | grep -v -E '(^#|^$)'

DISPLAYMANAGER_XSERVER="Xorg"
DISPLAYMANAGER="sddm"
DISPLAYMANAGER_REMOTE_ACCESS="yes"
DISPLAYMANAGER_ROOT_LOGIN_REMOTE="no"
DISPLAYMANAGER_STARTS_XSERVER="yes"
DISPLAYMANAGER_XSERVER_TCP_PORT_6000_OPEN="yes"
DISPLAYMANAGER_AUTOLOGIN="john"
Default:     "yes"
DISPLAYMANAGER_PASSWORD_LESS_LOGIN="yes"
DISPLAYMANAGER_AD_INTEGRATION="no"
DISPLAYMANAGER_SHUTDOWN="auto"

 this is SuSE still runnig xcfe, will fix...: --> cat /etc/sysconfig/displaymanager | grep -v -E '(^#|^$)'

DISPLAYMANAGER_XSERVER="mate-desktop"
DISPLAYMANAGER="lightdm"
DISPLAYMANAGER_REMOTE_ACCESS="yes"
DISPLAYMANAGER_ROOT_LOGIN_REMOTE="yes"
DISPLAYMANAGER_STARTS_XSERVER="yes"
DISPLAYMANAGER_XSERVER_TCP_PORT_6000_OPEN="yes"
DISPLAYMANAGER_PASSWORD_LESS_LOGIN="no"
DISPLAYMANAGER_AD_INTEGRATION="no"
DISPLAYMANAGER_SHUTDOWN="auto"
DISPLAYMANAGER_AUTOLOGIN=""


 further refinement of process to build study tools, worked through comparing line by line, editing and checking:
--> more mergefiles.sh.txt
#!/bin/bash
#   john 30 Mar 2017 merge-files-project.sh.txt
################################################
cp KJV.txt KJVb.txt
cp NAS.txt NASb.txt
perl -pi -e  's/(.*)/$1(kjv)/' KJVb.txt
perl -pi -e  's/(.*)/$1(nas)/' NASb.txt
paste -d '\n' KJVb.txt NASb.txt > KNb.txt
#######################

 To test I'd view the combined file, if I found an error, 

I'd edit the NAS version and repeat, put it all on one command line:
    --> vi NAS.txt ; sh ./merge-files-project.sh.txt ; more KNb.txt

The first books I configured had more issues, but once my updated scripts had been
used it went quickly; took about 3 evenings to finish.

This also found the one discrepancy between the translations, as noted in 3Jn 1:15
, where I added a line to the kjv to match:

--> grep "3Jn 1:1" KNb.txt
    #  note: if you forget the quotes you get everything that starts out 3Jn and the name of the file as well...
    #  note: the indent on the second line for nas - this is found in the NAS.txt file, one space leading
...
3Jn 1:13 I had many things to write, but I will not with ink and pen write unto thee: (kjv)
 3Jn 1:13  I had many things to write to you, \
but I am not willing to write them to you with pen and ink; (nas)
3Jn 1:14 But I trust I shall shortly see thee, and we shall speak face to face. \
Peace be to thee. Our friends salute thee. Greet the friends by name.(kjv)
 3Jn 1:14 but I hope to see you shortly, and we will speak face to face. (nas)
3Jn 1:15 THIS VERSE DOES NOT APPEAR IN THE KJV BUT DOES IN THE NASB.(kjv)
 3Jn 1:15  Peace be to you. The friends greet you. Greet the friends by name. (nas)


 So, now I have several versions to work with for searching key words and phrases.
 The kjv with a verse on each line with and without (kjv), and the same for nas.
The combined version has (kjv) and (nas) for clarity.
 The NAS.txt and KJV.txt are the base copies, but I have both translations by chapter and by book.
 Using grep I have control for searches on the combined file showing any differences in the translations.

Looking for a common phrase:
------------------------------------------------
--> grep -i "Jesus wept" KNb.txt
Jn 11:35 Jesus wept. (kjv)
 Jn 11:35  Jesus wept. (nas)

------------------------------------------------

------------------------------------------------
When I check a phrase initially I'll often have the screen flash by with lots of info, so this helps:
    -->  grep -i "know that I am the Lord." KNb.txt | wc -l
     154
------------------------------------------------
So, then I use a pipe and grep another key word to narrow the results...
But sometimes the phrase or keys aren't the same in both and I'll only see one version:
------------------------------------------------
--> grep -i "you will know that I am the Lord." KNb.txt | grep Zion
 Joe 3:17  Then you will know that I am the Lord your God, Dwelling in Zion, My holy mountain.
        So Jerusalem will be holy, And strangers will pass through it no more. (nas)

------------------------------------------------
By reducing the phrase and dropping "you will" from the string, then both versions are shown.
------------------------------------------------

--> grep -i "know that I am the Lord." KNb.txt | grep Zion
Joe 3:17 So shall ye know that I am the LORD your God dwelling in Zion, my holy mountain: then
         shall Jerusalem be holy, and there shall no strangers pass through her any more. (kjv)
 Joe 3:17  Then you will know that I am the Lord your God, Dwelling in Zion, My holy mountain.
         So Jerusalem will be holy, And strangers will pass through it no more. (nas)
-----------------------------------------------
another variation that may work:
--> grep -i "[shall ye|will] know that I am the Lord." KNb.txt  | grep Zion
Joe 3:17 So shall ye know that I am the LORD your God dwelling in Zion, my holy mountain:
    then shall Jerusalem be holy, and there shall no strangers pass through her any more. (kjv)
 Joe 3:17  Then you will know that I am the Lord your God, Dwelling in Zion, My holy mountain.
    So Jerusalem will be holy, And strangers will pass through it no more. (nas)

-----------------------------------------------

When I know there is a difference in the translations, or suspect it, 

and I have the reference, I can grep on that:

--> grep "Jn 8:58" KNb.txt
Jn 8:58 Jesus said unto them, Verily, verily, I say unto you, Before Abraham was, I am. (kjv)
 Jn 8:58  Jesus said to them," Truly, truly, I say to you, before Abraham was born, I am."(nas)


 now I can use the file KNb.txt, or KJV.txt, NAS.txt to search out phrases or key words
------------------------------------------------
--> grep -i "know that I am the Lord." KNb.txt | more  # both translations with (kjv|nas) at the end
--> grep -i "know that I am the Lord." KJV.txt | more  # KJV without (kjv) at the end
--> grep -i "know that I am the Lord." KJVb.txt | more   # KJV with (kjv) at the end
--> grep -i "know that I am the Lord." NAS.txt | more  # NAS without (nas) at the end
--> grep -i "know that I am the Lord." NASb.txt | more  # NAS with (nas) at the end

 an example of a passage to search out is one using "I am", a statement that declared diety.
In the Torah when Moses asked who he should say told him, the response was "I AM that I AM".
------------------------------------------------
So, I search:  -->  grep -i "I am that I am" KNb.txt
Exo 3:14 And God said unto Moses, I AM THAT I AM: and he said,
  Thus shalt thou say unto the children of Israel, I AM hath sent me unto you. (kjv)
------------------------------------------------
ok... but where's the nas version?  No problem, simply grep for the passage:
--> grep "Exo 3:14" KNb.txt

Exo 3:14 And God said unto Moses, I AM THAT I AM: and he said,
Thus shalt thou say unto the children of Israel, I AM hath sent me unto you. (kjv)

 Exo 3:14 God said to Moses, "I A M W H O I A M"; and He said,
"Thus you shall say to the sons of Israel, 'I A M has sent me to you.'"(nas)
------------------------------------------------
AND GUESS WHAT?  I just found another edit I need to fix in NAS.txt... :) That's why it didn't show up.
Perhaps I should write a script to update all my computers where I have these files now...
------------------------------------------------
But, continuing the search for the other statement of diety...
------------------------------------------------
--> grep -i "I am" KNb.txt | wc -l
    1641
------------------------------------------------
well, need to narrow that down a bit
------------------------------------------------
--gt; grep -i "I am" KNb.txt | grep Jesus | wc -l
      66
------------------------------------------------
still too many verses... going to guess the passage is in the Gospels
------------------------------------------------
--> grep -i "I am" KNb.txt | grep Jesus | grep -E '(Mat|Mk|Lk|Jn)' | wc -l
33
------------------------------------------------
Ah, just remembered, Abraham was mentioned in the context... adding a grep:
------------------------------------------------
--> grep -i "I am" KNb.txt | grep Jesus | grep -E '(Mat|Mk|Lk|Jn)' | grep Abraham
Jn 8:58 Jesus said unto them, Verily, verily, I say unto you, Before Abraham was, I am. (kjv)
 Jn 8:58  Jesus said to them," Truly, truly, I say to you, before Abraham was born, I am."(nas)
------------------------------------------------
Of course I could just memorize the verse with the passage reference... well, I did, but I forgot it...
------------------------------------------------
So... now I've got a tool that allows me to search on key words, phrases and known references!
------------------------------------------------
I'm a happy student... and I need to fix /etc/sysconfig/displaymanager and test,
and find a way of checking and fixing NAS.txt in Exo 3:14,
and coming up with a way of copying or updating all my systems where I have these notes...
------------------------------------------------
Ok, if we make it this far and have time...
fixing NAS.txt - LET'S use grep and awk to find any other instances
like Exo 3:14 and create a script or command that lists those passages for us to review,
and then possibly create a script using perl or sed to fix those known instances specifically
so we don't create other problems in the file.  We will need to review those passages
to make sure they are the same as the source document.

It's tedious and laborius, but important, and using these powerful regex tools in Linux
at the command line makes the creation and editing of this tool possible.

And if we still have time we can generate an rsync script to update other directories and systems...

 

Notes for this session:

 discuss possible add ons for SuSE  
http://johnmeister.com/linux/SysAdmin/SuSE-add-ons.html

 using CSV (comma separated values) in a script to gather information (part of the Art of Linux Sys Admin series)
 http://johnmeister.com/linux/Scripts/collect-sys-info.sh.html
- automate system documentation!

 CSV at the command line - using gnumeric:
    --> ssconvert spreadsheet.xls newfile.csv    # (will this convert csv to xls? RTM anyone?)
    # a good use for gnumeric!
    --> apt-get install gnumeric  or --> zypper install gnumeric  or --> yum install gnumeric

 using the office suite at the command line (YMMV) - results on Linux with libreoffice ok...
    created a directory... need to try other options.

     CSV at the command line - using libreoffice:
        -->  libreoffice --headless --convert-to csv $filename --outdir $outdir
            # this worked, created a directory
     CSV at the command line - using OpenOffice:
        --> soffice --headless --convert-to csv test-file.xls test.csv
    #  not recommended on a mac, haven't tried on Linux or Win!
    # tried this with "advice" from a search - it created a ".lock" file owned by root in:
        (following advice on the internet is NOT recommended)
    #  further advice says delete .lock... # BETTER - chown!!!
      (I should never do what I find on the internet! (without thinking)... and not on a mac)
    # /Users/luser/Library/Application Support/OpenOffice/4  removing it removed all of my
        configuration and history - thinking gnumeric on linux best...

 
 prep material to put into CSV to import into spreadsheet
 collect chapters into one book for Proof of Concept (POC):
  --> cat 11n_1Ki-* > 1Kings.txt ; perl -pi -e 's/1 Ki/1Ki/g' 1Kings.txt
        # fix space in naming

  --> cat 1Kings.txt  | tr " " "\n" | grep ^[A-Z] | sort | uniq -d > 1Kings-KEYWORDS.txt
            # vi to clean up sp ch

  --> cat 1Kings-NAS-KEYWORDS.txt | wc -l
     119

 create script to take KEYWORDS and list times found and references:
#!/bin/bash
# Find-Keywords-1Kings.sh.txt   13 Apr 2017 jm
##########################################
# list of names: 1Kings-KEYWORDS.txt # complete book: 1Kings.txt
##########################################
FILE="Names-found-in-1st-Kings.txt"   # output, names to references
echo `date +'%Y_%m_%d-%H%M'` > $FILE   # way to reset the file and / or keep track
echo "======================"  >> $FILE   # eye candy separator
for x in `cat 1Kings-NAS-KEYWORDS.txt`    #   x is the name to search on
    do
     echo $x >> $FILE                       #  list the name
     echo "----------" >> $FILE             #  eye candy
     grep $x 1Kings.txt | wc -l >> $FILE    #  number of times name is found in book
     echo "----------" >> $FILE             #  eye candy
     grep $x 1Kings.txt | awk '{print $1" "$2}' >> $FILE   # print book and reference
     echo "======================"  >> $FILE   # eye candy
    done
##########################################
OUTPUT:
##########################################
# 2017_04_13-2209
# ======================
# Abel-meholah,
# ----------
#        2
# ----------
# 1Ki 4:12
# 1Ki 19:16
# ======================
 NEXT CSV - modify script to insert commas for CSV formatting... (not complete, need to wrap lines) 
##########################################
#!/bin/bash
# Find-Keywords-SET-AS-CSV-1Kings.sh.txt   13 Apr 2017 jm
##########################################
# 1Kings-NAS-KEYWORDS.txt - grep'd Names, manually edited file
# 1Kings.txt  #  NAS 1 Kings text
##########################################
FILE="Names-found-in-1st-Kings-CSV-FORMAT.csv.txt"
for x in `cat 1Kings-NAS-KEYWORDS.txt`
    do
     echo $x >> $FILE        # no eye candy, name, followed by a comma, number of times, references
     echo ", ">> $FILE
     grep $x 1Kings.txt | wc -l >> $FILE






     echo ", " >> $FILE
     grep $x 1Kings.txt | awk '{print $1" "$2", "}' >> $FILE
     echo "====================== \n"  >> $FILE
    done
##########################################
# OUTPUT:
##########################################
Abel-meholah
,
       2
,
1Ki 4:12,
1Ki 19:16,
======================
Abiathar
,
       9
,
1Ki 1:7,
1Ki 1:19,
1Ki 1:25,

 

 using grep to get the number of times a word appears in the NAS, and getting the references for study
  694  grep -i love NAS.txt | wc -l   #    573
  695  grep -i hate NAS.txt | wc -l   #    312
  696  grep -i Lord NAS.txt | wc -l   #   6707
  697  grep -i sin NAS.txt | wc -l   #    1566
  698  grep John NAS.txt | wc -l   #       136
  699  grep Jesus NAS.txt | wc -l   #      946
  700  grep Joseph NAS.txt | wc -l   #     233
  701  grep Rebecca NAS.txt | wc -l   #      0
  702  grep Rebeka NAS.txt | wc -l   #      31
  703  grep Paul NAS.txt | wc -l   #       161
  704  grep -i faith NAS.txt | wc -l   #   404
  705  grep -i faithful NAS.txt | wc -l  # 163

--> grep -i faith NAS.txt | awk '{print $1" "$2}'
Gen 32:10
Gen 47:29
Lev 5:15
...
Rev 19:11
Rev 21:5
Rev 22:6

using search tool, catgrep, and basics on Apache web server and directives

  search text tool http://johnmeister.com/linux/Scripts/SEARCH-TEXT
 script and all files needed to use found at:  http://johnmeister.com/bible/SEARCH-KJV/

  use catgrep to identify existing config:

  http://johnmeister.com/linux/Scripts/catgrep.sh.html - "catgrep /etc/apache2/default-server.conf" 

  understand the basics: http://johnmeister.com/linux/SysAdmin/Apache2-setup.html
1) install apache2 - zypper in apache2
2) establish base configuration in /etc/apache2/default-server.conf
3) verify document root and log locations
4) systemctl enable apache2  (*chkconfig)
* --> chkconfig -l | grep apache
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native systemd configuration.
apache2                   0:off  1:off  2:off  3:off  4:off  5:on   6:off
5) systemctl start apache2

  examining existing directives, THESE DIRECTIVES WORK with Apache2 on SuSE 13.2, but NOT in LEAP 42.2:
DocumentRoot "/srv/www/htdocs"
<Directory "/srv/www/htdocs">
 AllowOverride FileInfo AuthConfig Limit Indexes Options MultiViews Indexes IncludesNoExec  FollowSymLinks
 IndexOptions IgnoreCase FancyIndexing FoldersFirst NameWidth=* DescriptionWidth=* SuppressHTMLPreamble
#SymLinksIfOwnerMatch
 <Limit GET POST OPTIONS PROPFIND>
 Order allow,deny
  Allow from all
 </Limit>
 <LimitExcept GET POST OPTIONS PROPFIND>
  Order deny,allow Deny from all
 </LimitExcept>
</Directory>

  review or create systemd service

 - http://johnmeister.com/linux/Scripts/Sample-systemd-script.html 
 any changes to website can be made throughout using:
    find . -type f -name README.html -exec cp README.html {} \;

using tools to extract key information on a system

Using search tool and text editing

Apache 2.4 migration

using ssh, rsync, ftp, NFS

1. basic: http://johnmeister.com/linux/FileSystems/man.rsync.txt
2. config info: http://johnmeister.com/linux/Scripts/fix-ftp.sh.html
3. config info: http://johnmeister.com/linux/Notes/ftp-fixes-for-vsftp.conf.html
4. basics: http://johnmeister.com/linux/Notes/RSYNC-cmd.html
5. examples: http://johnmeister.com/linux/Scripts/rsync-directories.sh.html
6. cron job examples: http://johnmeister.com/linux/FileSystems/rsync-disks-cron.sh.html
7. cron job examples: http://johnmeister.com/linux/SysAdmin/Cron-Rsync-sync-up-drives.sh.html
8. using rsync to update: http://johnmeister.com/linux/Scripts/setup-diverse-systems-prototype.sh.html
9. mention (again): http://johnmeister.com/linux/Scripts/EXAMPLE-OSYNC/
10. NFS option: http://johnmeister.com/linux/FileSystems/NFS-Server-setup.html
11. - if time: (demonstrate WinSCP on XP in a VM on Linux or a Mac)

using basic commands and the vi editor.

 After helping a few friends with Linux recently I see the need
    for less speed and more detail... so...
Let's review basics of .bashrc, .History, setting up .ssh
    and converting a Windows Batch file to Linux to check network performance.
http://johnmeister.com/linux/Notes/bashrc-simple.html
http://johnmeister.com/linux/Notes/SSH/quick-ssh.html
 http://johnmeister.com/linux/Microsoft/LOADTEST-OS-COMPARISON.html 
directory with excel, batch file and large file 
http://johnmeister.com/linux/Microsoft/NETWORK-PERFORMANCE/Network-Time-Test-GENERIC.bat.html
oh, and one slightly advanced moment:
http://johnmeister.com/linux/Notes/setup-LAMP-on-MINT.html

100 lunchtime sessions since 2014!

Networking basics and testing performance

BASIC SETUP:  ifconfig lan0 192.168.11.42 netmask 255.255.255.0 broadcast 192.168.11.255
ADD ROUTE:    route add default gw 192.168.11.1
COMMANDS:     ifconfig -a, ip addr, ping -c 3 8.8.8.8, route, netstat -ar, traceroute 8.8.8.8, nslookup johnmeister.com/linux
(note: Microsoft commands offer fewer options and change a few names: e.g. ifconfig --> ipconfig, traceroute --> tracert.)
FILES:    /etc/hosts, /etc/resolv.conf, /etc/nsswitch.conf, /etc/hostname, /etc/sysconfig/network (YMMV)

http://johnmeister.com/linux/Notes/NETWORK/basic-networking.html


 http://johnmeister.com/linux/Microsoft/NETWORK-PERFORMANCE/Network-Time-Test-GENERIC.bat.html

 examining the spreadsheet formulas (CSV)

 http://johnmeister.com/linux/Microsoft/NETWORK-PERFORMANCE/Network-Time-Test-Generic-BASH.sh.html

grep basics and using catgrep to analyze config files

http://johnmeister.com/linux/Scripts/catgrep.sh.html
    #  this is the key string:  grep -E -v '(^\s*#|^$)'

 clean copy to copy... copy to ~/bin/catgrep, chmod and update path in .bashrc

#!/bin/bash
################################################################################
# 24 jun 2016 - john meister copyright 2016 http://johnmeister.com/linux
#  cat's file, then greps out blank lines and comments, offers option to save
# 22 jul 2016 - added sed string to strip leading slash off echo /etc | sed 's/^\///'
# 6 jun 2017 - finally got the white space string working: grep -E -v '(^\s*#|^$)'
# NOTE:  create ~/bin directory, chmod to 755, add ~/bin to PATH - test:
#    ex: --> which catgrep
#      /home/luser/bin/catgrep
################################################################################
USAGE="catgrep filename # removes commented and blank lines, option to save"
echo ' --------------------------------------------------------------------'
echo " DISPLAYING file: -->   $1    <--  without commented or blank lines"
echo ' --------------------------------------------------------------------'
cat $1 | grep -E -v '(^\s*#|^$)'  #   the \s is for white spaces
VFL="`echo "$1" | sed 's/^\///' | tr '/' '-'`" ; export VFL
echo '-----------------------------------------------------------------'
echo "save info in local directory?  if yes, type: y"
echo '-----------------------------------------------------------------'
read answer  ;   if [ $answer = "y" ]     # note, adding brackets for upper/lower case would be [Y,y]
    then
    echo "$1"  | tee ~/FYI-$VFL-content-`date +%Y%b%d-%H%M`.txt
    cat $1 | grep -E -v '(^\s*#|^$)' | tee -a ~/FYI-$VFL-content-`date +%Y%b%d-%H%M`.txt
    #### NOTE: use redirect or append on line above instead of tee -a to reduce screen clutter
    echo '-----------------------------------------------------------------'
    echo " info written to:      ~/FYI-$VFL-content-`date +%Y%b%d-%H%M`.txt "
    echo '-----------------------------------------------------------------'
    else
    echo '      ### ------------------------------------'
    echo '      ###       not written.'
    echo '      ### ------------------------------------'
fi

additional details:
http://johnmeister.com/linux/Notes/REGULAR-EXPRESSIONS-simple.html

system logs: dmesg and journalctl

dmesg # provides kernel information # part of util-linux package, see Linux Kernel Archive; man dmesg # sudo dmesg --clear
sudo journalctl  # sudo journalctl -f  # sudo journalctl | grep -i error # journalctl | grep -i Error | grep ssh | grep -v "Jun 0"


Migration from Apache 2.2 to 2.4
changes to apache2 involved replacing two lines regarding permissions, index directives continued to work:
#               #Order allow,deny# Allow from all -->  Require all granted
#               # Order deny,allow# Deny from all -->  Require all denied

--> apachectl configtest #  Syntax OK
--> systemctl enable apache2
--> systemctl start apache2

--> systemctl status apache2
. apache2.service - The Apache Webserver
   Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-06-07 00:34:44 PDT; 5s ago
 Main PID: 2422 (httpd-prefork)
   Status: "Processing requests..."
    Tasks: 6

   CGroup: /system.slice/apache2.service
           - 2422 /usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -DSYSTEMD...
           - 2429 /usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -DSYSTEMD...
           - 2430 /usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -DSYSTEMD...
           - 2431 /usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -DSYSTEMD...
           - 2432 /usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -DSYSTEMD...
           - 2433 /usr/sbin/httpd-prefork -f /etc/apache2/httpd.conf -DSYSCONFIG -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d/ -DSYSTEMD...

Jun 07 00:34:44 linuxsystem systemd[1]: Starting The Apache Webserver...
Jun 07 00:34:44 linuxsystem systemd[1]: Started The Apache Webserver.

 HAPPY SYS ADMIN DAY - July 28, 2017!!! 

Prepping for LIVE Oreilly Media Windows 10 BASH sessions - test run...

basic .bashrc: http://johnmeister.com/linux/Microsoft/Win10-BASH/W10-bashrc.txt
bashrc delta from default: http://johnmeister.com/linux/Microsoft/Win10-BASH/W10-bashrc-DELTA.txt
summary of shopt (shell options): http://johnmeister.com/linux/Microsoft/Win10-BASH/W10-shopt-list-details.html

 HAPPY SYS ADMIN DAY - July 28, 2017!!! 
Sys Admin song (sung to monty python's lumberjack tune):

[spoken]
I never really wanted to be a scientist.
I wanted to be...a...a SYSADMIN!

[system engineer choir and shift supervisor enter, music strikes up]

Oh, I'm a sysadmin and I'm OK,
I grep all night and I chown all day.

[choir]

He's a sysadmin and he's OK,
He greps all night and he chowns all day.

I ping the nodes, I do PM,
I awk and perl and sed.
I've got a Star Wars lunchbox,
And Tron sheets on my bed!

[choir]

He pings the nodes, he does PM,
He awks and perls and seds.
He's got a Star Wars lunchbox,
And Tron sheets on his bed!

I ping the nodes, I change the rates,
I fork the processes.
I wish that all my lusers
would catch some rare disease!

[choir, growing slightly uncomfortable]

He pings the nodes, he changes rates,
He forks the processes.
He wishes all his lusers
would catch some rare disease!

[choir brightens as they repeat chorus]

I ping the nodes, I lock the /home partition and umounts....
[choir, very uncomfortable and trailing off]

...HAPPY SYS ADMIN DAY...

tightening up sshd instead of using fail2ban

tweaks to /etc/ssh/sshd_config  - 
typical ssh files:  http://johnmeister.com/linux/Notes/SSH/sample-etc-ssh-config-files.html

    make sure root couldn't log in directly.:   PermitRootLogin no
    specify which users can ssh in:             AllowUsers luser
    force only one version of ssh:              Protocol 2 
    give the person ONE try:                    MaxAuthTries 1
    reduce the number of starting connections:  MaxStartups 2:50:5
    reduce the login grace time from 2 min:     LoginGraceTime 30
    restart sshd                                # service sshd restart 
                
                (remember:    man sshd | col -b > man.sshd.txt - RTM... YMMV!) 

system logs ssh: journalctl -f | grep ssh, or journalctl -f | grep -i Error | grep ssh
--> journalctl -f | grep ssh

Jun 21 23:18:09 the-linux-system sshd[23682]: error: PAM: User not known to the underlying authentication module for illegal user admin from 103.79.142.31
Jun 21 23:18:09 the-linux-system sshd[23682]: error: maximum authentication attempts exceeded for invalid user admin from 103.79.142.31 port 52295 ssh2 [preauth]
Jun 21 23:24:06 the-linux-system sshd[23694]: error: maximum authentication attempts exceeded for invalid user root from 190.49.255.143 port 42263 ssh2 [preauth]
Jun 21 23:27:53 the-linux-system sshd[23703]: error: maximum authentication attempts exceeded for invalid user oracle from 201.20.73.3 port 56130 ssh2 [preauth]


 if an IP is very persistent, use iptables to block
e.g.:  iptables -A INPUT -s 43.255.190.xx -j DROP

apache logs:  --> ll /var/log/apache2/*error_log
if an error log has content, view, and see if you can find the error.  

After fixing, make a copy of the log file then empty it as follows:
cp the-error_log  /var/log/apache2/FIXED/the-error_log-clear-21jun2017 ; cat /dev/null > the-error_log


 
 SESSION #105 - September 15, 2017 - FRI - Let's USE BASH on Windows 10!  
 
 note on html: color for cell above is "1242ab" (rrggbb), font color is "white"




 working through the book on using the BASH tool in Windows 10.
Follow along in Linux or MacOSX... it's mostly the same. (you can preview the book up to about page 19.)
 TABLE OF CONTENTS for "Let's USE BASH on Windows 10
  
 Introduction to BASH on Windows 10 
 How to INSTALL BASH on WIN10
 How to Configure the BASH window
 Chapter 1_Linux_Overview
 Chapter 2_the_5_essential_commands
 Chapter 3_the_vi_Editor
 Chapter 4_shell_config_History_script
 Chapter 5_File_Management:_cp_mv_rm
 Chapter 6_Files, Processes and Special Characters
 Chapter 7_Regular_Expressions_and_Filters
 Chapter 8_ssh_setup_and_use - will not cover for Win10
 Chapter 9_Network_Basics - will not cover for Win10
 Appendix: How to INSTALL_BASH_WIN10 - will not address in these sessions,
but this link might help, YMMV:   http://johnmeister.com/linux/Microsoft/Win10-BASH/Win10-BASH-install.html
 Appendix: The LINUX Philosophy
 Appendix: Exercise #1 - the 5 basic commands
 Appendix: Exercise #2 - chmod, chown, file permissions
 Appendix: Additional info on BASH
 Appendix: Additional info from Chapter 2
 Appendix: SCRIPT: fix filenames
 Appendix: SCRIPT: use wget, date, and a while loop
 Appendix: REAL WORLD LINUX COMMANDS
  NOTE: The book "Simply Linux" follows the same format,
but that version has more Linux and MacOSX detail, the Win10 version was scaled back slightly:
https://www.smashwords.com/books/view/705084




 
 SESSION #106 - September 22, 2017 - FRI - Let's USE BASH on Windows 10! - Pt 2  
 
 note on html: color for cell above is "00ff00" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  




 "Let's use BASH on Windows 10!" - Part 2 - Ch1, Ch2 
  
- Finish up Chapter One - discuss .bashrc and .History directory.  http://johnmeister.com/linux/bashrc-basic.html
- Discuss Chapter Two -  Five Essential Commands.


HOMEWORK EXERCISES:  

 RECOMMEND REVIEWING: 
http://johnmeister.com/linux/Notes/Philosophy_of_Linux.html
  - there are significant differences between Microsoft and Linux that need to be understood.
 http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-1.html


 TABLE OF CONTENTS for "Let's USE BASH on Windows 10
 Introduction to BASH on Windows 10
 How to INSTALL BASH on WIN10
 How to Configure the BASH window
 Chapter 1_Linux_Overview
 Chapter 2_the_5_essential_commands
 Chapter 3_the_vi_Editor
 Chapter 4_shell_config_History_script
 Chapter 5_File_Management:_cp_mv_rm
 Chapter 6_Files, Processes and Special Characters
 Chapter 7_Regular_Expressions_and_Filters
 Chapter 8_ssh_setup_and_use - will not cover for Win10
 Chapter 9_Network_Basics - will not cover for Win10
 Appendix: How to INSTALL_BASH_WIN10
 Appendix: The LINUX Philosophy
 Appendix: Exercise #1 - the 5 basic commands
 Appendix: Exercise #2 - chmod, chown, file permissions
 Appendix: Additional info on BASH
 Appendix: Additional info from Chapter 2
 Appendix: SCRIPT: fix filenames
 Appendix: SCRIPT: use wget, date, and a while loop
 Appendix: REAL WORLD LINUX COMMANDS
See session #105 for links to table of contents on smashwords.

OTHER RESOURCES:
 http://johnmeister.com/linux/Overview/LinuxOverview.pdf (print out and use as a guide)

 http://johnmeister.com/linux/Overview/Linux-PowerPoint-2004-overview.pdf

 http://johnmeister.com/linux/Notes/Real-world-Linux-Commands.

 http://johnmeister.com/linux/Intro-to-Linux/Special-Characters.pdf (print out and use as a guide)

http://johnmeister.com/linux/bashrc-basic.html


 
 SESSION #107 - September 29, 2017 - FRI - Let's USE BASH on Windows 10! - Pt 3  
 
 note on html: color for cell above is "0000FF" (rrggbb), font color is "white"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  


   Will transition FROM: "Let's use BASH on Windows 10!" - TO: --->   "Simply Linux: Basics" 
We will focus more on the actual Linux commands, less on the Win10 environment as host.
We'll talk more about Regular Expressions and the vi editor.  The commands, the core tools.

Not everyone will be able to use Ubuntu BASH in W10,
some will chose cygwin or other tools such as a VM or putty to a Linux system.

We will still discuss issues across platforms as we proceed.
It's important to note differences even between UNIX, Linux and other tools.
More will be shared during the session.


finish discussion of .bashrc and .History directory.  

http://johnmeister.com/linux/bashrc-basic.html
and then - Discuss Chapter Two -  Five Essential Commands.

Continuing in Chapter 2 of "Let's Use BASH on Windows 10", but shifting to the other book, same format...

Introduction to Linux

Chapter 1_Linux_Overview
Chapter 2_the_5_essential_commands
Chapter 3_the_vi_Editor
Chapter 4_shell_config_History_script
Chapter 5_File_Management:_cp_mv_rm
Chapter 6_Files, Processes and Special Characters
Chapter 7_Regular_Expressions_and_Filters
Chapter 8_ssh_setup_and_use
Chapter 9_Network_Basics
Chapter 10_Files_Systems:_df_du
...

Showing purpose with examples of settings in .bashrc:
    PATH; PROMPT; PERMISSIONS; HISTORY; EDITOR; ALIAS; SHELL OPTIONS; FUNCTIONS 

demonstrating "minor" issues and differences in paths, symlinks and command execution in BASH on Win10.
Will show how to create a copy of files in /mnt/c/


HOMEWORK EXERCISES:  

 RECOMMEND REVIEWING: 
http://johnmeister.com/linux/Notes/Philosophy_of_Linux.html
- there are significant differences between Microsoft and Linux that need to be understood.
 http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-1.html



 
 SESSION #108 - October 6, 2017 - FRIDAY - Adventures in BASH  
 
 note on html: color for cell above is "00FFFF" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  

  
Last week we discussed in detail the .bashrc file, and demonstrated some of the "five essential" commands:  

  man, ls, cd, pwd and more



This week: we will use regular expressions to replace special characters, 

deal with bad file-naming conventions, and rename long file names with AWK.



FIRST - what are special characters?  

  Special characters are powerful representations of shell functions and values. 

In the picture below there is a listing of the special characters and examples of their use.  

Special characters must be understood.

 






PROBLEM:  Someone created files and directories with spaces in the names, or worse, special characters.  

In UNIX-like systems spaces at the command line represent various commands or options.  

If a filename or directory has spaces, 

then those spaces must be "escaped" with a "\" or encapsulating the entire filename in quotes.

This gets complicated in a hurry.



With this script, we will remove spaces and replace special characters: 

 http://johnmeister.com/linux/Scripts/fix-filenames.sh.html 

 
#!/bin/bash
###############################################################################################
# 2014_05May_09 - john meister copyright © 2014 http://LinuxMeister
# script to remove spaces in file names and replace with underscore
# cleans up web pages by eliminating special characters and allows sorts to work properly
#  to use place this script in your path and make it executable, then cd to the directory
#    and execute by the saved name (fix-filename)  - works on Linux and MacOS
###############################################################################################
#             removes spaces, special characters, slashes, and _-_  but NOT caps
###############################################################################################
for f in *
    do
 mv -v "$f" `echo $f | tr ' ' '_' | tr -d '[{}(),\!]' | tr -d "\'" | sed 's/&/-n-/g' | sed 's/_-_/_/g'`
    done


NOTE:  we'll cover more loops as we go along, here's some examples of loops: 

 http://johnmeister.com/linux/Scripts/LOOPS-shell-examples.html


ISSUE:  created a bunch of files with really long stupid names, however, 

the good news is I had a "delimiter" that would work with AWK.

By using the "-" in the really long stupid file names I was able to use AWK 

to simplify the filenames.  This was a real world problem that was fixed.



Example of using AWK in a loop to move large numbers of files on a Linux web server:
 http://johnmeister.com/linux/Scripts/awk-to-rename-files.sh.html  



 create temporary script, e.g.   "vi fsj-rename"  

(if you have more than one directory create a more structured script and place it in your ~/bin).

 
#!/bin/bash
# GOAL:  rename all files in a directory from "fsj-digest-v1-????.txt" to "fsj-????.txt"
# DATE: 4/27/2004 - Author John Meister
#  TEST:  --> ls fsj-digest-v1-2123.txt | awk -F - '{print "mv "$1"-"$2"-"$3"-"$4" " $1"-"$4}'
#  PRODUCED:   mv fsj-digest-v1-2123.txt fsj-2123.txt
#
# --> ls fsj-digest-v1-????.txt | awk -F - '{print "mv "$1"-"$2"-"$3"-"$4" " $1"-"$4}' > mv-digests.sh
#   edit file to include header info & comments
#
mv fsj-digest-v1-1800.txt fsj-1800.txt
mv fsj-digest-v1-1801.txt fsj-1801.txt
mv fsj-digest-v1-1802.txt fsj-1802.txt
mv fsj-digest-v1-1803.txt fsj-1803.txt
mv fsj-digest-v1-1804.txt fsj-1804.txt
...
-------------------------
 THEN... execute the script by typing:   sh ./fsj-rename
 check directory and delete temporary script:   ls ; rm -f fsj-rename



If we have time we will use AWK to rename the history files discussed last week.  

I noticed that I did not use the date command in a way that makes sorting work well. 

Current format is: 2017_Mar_27_08:32.history   

It would be better for sorting if it were 2017_03_Mar_27_08:32.history



The good news is that I used "_" in the name; thereore, I can modify the script above to change the file names.



Previously the date variable was set to: 

HISTFILE="/home/luser/.History/`/bin/date '+%Y_%b_%d_%H:%M'.history`" ; export HISTFILE 

Last week we modified .bashrc to fix this variable:  

HISTFILE="/home/luser/.History/`/bin/date '+%Y_%m_%b_%d_%H:%M'.history`" ; export HISTFILE

THE DATE command option for alphabetical representation of the month is %b, while %m is the numeric value.  



I like to see the Month in letters, but like to sort things properly.

By sorting YEAR, MONTH and then DAY it displays in chronological order.

If I wanted to keep the filename shorter I could replace the hour and minutes with just the PID,

or use the seconds, the actual time of the file is not important.




 
 SESSION #109 - October 13, 2017 - FRIDAY - Adventures with Special Characters
 
 note on html: color for cell above is "F0F0F0" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  

  


 we've been covering the elements of the .bashrc:  

 Path, Prompt, Permissions, History, Editor, Alias, Shell options and Functions
 we've been reviewing the "five essential" commands:  man, ls, cd, pwd and more
 we've been working our way through the SPECIAL CHARACTERS - and will continue to this session...
 we'll discuss the scripts below - including the use of sed, awk, tr and even some perl
 used find, exiftool and reg exp tools to gather lens info from NEF files.


  Special characters are powerful representations of shell functions and values. 

 








Remove spaces and replace special characters: 

http://johnmeister.com/linux/Scripts/fix-filenames.sh.html 

 
#!/bin/bash
###############################################################################################
#             removes spaces, special characters, slashes, and _-_  but NOT caps
###############################################################################################
for f in *
    do
    mv -v "$f" `echo $f | tr ' ' '_' | tr -d '[{}(),\!]' | tr -d "\'" | sed 's/&/-n-/g' | sed 's/_-_/_/g'`
    done

examples of loops: 


http://johnmeister.com/linux/Scripts/LOOPS-shell-examples.html


Use AWK in a loop to move large numbers of files on a Linux web server:

http://johnmeister.com/linux/Scripts/awk-to-rename-files.sh.html  


 create temporary script, e.g.   "vi fsj-rename"  

(if you have more than one directory create a more structured script and place it in your ~/bin).
 
#!/bin/bash
# GOAL:  rename all files in a directory from "fsj-digest-v1-????.txt" to "fsj-????.txt"
# DATE: 4/27/2004 - Author John Meister
#  TEST:  --> ls fsj-digest-v1-2123.txt | awk -F - '{print "mv "$1"-"$2"-"$3"-"$4" " $1"-"$4}'
#  PRODUCED:   mv fsj-digest-v1-2123.txt fsj-2123.txt
#
# PROCESS:  --> ls fsj-digest-v1-????.txt | \
#            awk -F - '{print "mv "$1"-"$2"-"$3"-"$4" " $1"-"$4}' > mv-digests.sh
# PROCESS:    edit file to include header info & comments
#
mv fsj-digest-v1-1800.txt fsj-1800.txt
mv fsj-digest-v1-1801.txt fsj-1801.txt
...
-------------------------
 THEN... execute the script by typing:   sh ./fsj-rename
 check directory and delete temporary script:   ls ; rm -f fsj-rename







 
 SESSION #110 - October 20, 2017 - FRIDAY - wget, advanced vi editing
 
 note on html: color for cell above is "0F0F0F" (rrggbb), font color is "white"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  

  


 we've covered .bashrc:
    Path, Prompt, Permissions, History, Editor, Alias, Shell options and Functions
 we've reviewed the "five essential" commands:  man, ls, cd, pwd and more
 we've worked our way through the SPECIAL CHARACTERS - and will continue to...
 will discuss some issues of interoperability with different BASH environments,
    e.g. MacOSx, different versions of BASH
 we'll work on building some web pages - include the use of sed, awk, tr and even some perl





 
 SESSION #109 - October 13, 2017 - FRIDAY - Adventures with Special Characters
 
 note on html: color for cell above is "F0F0F0" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  

  


 we've been covering the elements of the .bashrc:  

 Path, Prompt, Permissions, History, Editor, Alias, Shell options and Functions
 we've been reviewing the "five essential" commands:  man, ls, cd, pwd and more
 we've been working our way through the SPECIAL CHARACTERS - and will continue to this session...
 we'll discuss the scripts below - including the use of sed, awk, tr and even some perl
 used find, exiftool and reg exp tools to gather lens info from NEF files.


  Special characters are powerful representations of shell functions and values. 

 








Remove spaces and replace special characters: 

http://johnmeister.com/linux/Scripts/fix-filenames.sh.html 

 
#!/bin/bash
###############################################################################################
#             removes spaces, special characters, slashes, and _-_  but NOT caps
###############################################################################################
for f in *
    do
    mv -v "$f" `echo $f | tr ' ' '_' | tr -d '[{}(),\!]' | tr -d "\'" | sed 's/&/-n-/g' | sed 's/_-_/_/g'`
    done

examples of loops: 


http://johnmeister.com/linux/Scripts/LOOPS-shell-examples.html


Use AWK in a loop to move large numbers of files on a Linux web server:

http://johnmeister.com/linux/Scripts/awk-to-rename-files.sh.html  


 create temporary script, e.g.   "vi fsj-rename"  

(if you have more than one directory create a more structured script and place it in your ~/bin).
 
#!/bin/bash
# GOAL:  rename all files in a directory from "fsj-digest-v1-????.txt" to "fsj-????.txt"
# DATE: 4/27/2004 - Author John Meister
#  TEST:  --> ls fsj-digest-v1-2123.txt | awk -F - '{print "mv "$1"-"$2"-"$3"-"$4" " $1"-"$4}'
#  PRODUCED:   mv fsj-digest-v1-2123.txt fsj-2123.txt
#
# PROCESS:  --> ls fsj-digest-v1-????.txt | \
#            awk -F - '{print "mv "$1"-"$2"-"$3"-"$4" " $1"-"$4}' > mv-digests.sh
# PROCESS:    edit file to include header info & comments
#
mv fsj-digest-v1-1800.txt fsj-1800.txt
mv fsj-digest-v1-1801.txt fsj-1801.txt
...
-------------------------
 THEN... execute the script by typing:   sh ./fsj-rename
 check directory and delete temporary script:   ls ; rm -f fsj-rename








 
 SESSION #111 - October 27, 2017 - FRIDAY - two scripts
 
 note on html: color for cell above is "tan" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  (can use names too...) 

  
 NOTE:  no session NEXT week, Friday 3 Nov 2017

 two scripts 
 work flow process for digital images and script to create web page with thumbnails
workflow process:
 
 download images from SD card to computer via scripts
 review original images; minor edits - resize, rename with dates - keeping RAW impages untouched
 resized/renamed images go up one directory.  review which ones will go on web page
 move resized/renamed images to web directory
 scp -r web directory to server
 ssh to server, run "mkpg" script http://johnmeister.com/linux/Scripts/mk-webpage-2015-05-08.html
 add "exiftool" info with extract into directories:  (can be done on Mac or either Linux system)
  --> more exif-strip.sh
    #!/bin/bash
    #  uses exiftool to extract EXIF info in original images
    #
    for x in `ls *`
        do
        exiftool $x > exifinfo-$x.txt
        cat exifinfo-$x.txt| grep -E -v '(Brightness|Timezone|ISO2|Zone|Bracket|Image|Artist|Crop|Sensitivity|\
    Preview|Difference|Shift|MIME|Tuning|Clarity|Serial|Mode|Modify|Distance|Confusion|Gain|Tune|Phase|CFA|Depth|\
    Hyper|Light|Mega|Red|Blue|GPS|Normal|TIFF|Jpg|Binary|Sub|Memory|Original|File|Directory|Sensing|Size|Power|\
    Multi|Raw|Contrast|Vari|Retouch|Max|Compression|Exit|Comment|Primary|MCU|Version|Shot|Position|Picture|Black|\
    Control|Points|Expansion|Zoom|Toning|Hue|Filter|Format|Min|Color|Quality|Levels|Flash|Scene|Other|Planar|Resolution|\
    Copyright|Strip|Balance|Bits|Stops|Samples|Software|Orientation|Byte|Make|Savings|Scale)'  > base-exif-$x.txt
     done

    mkdir ../EXIFINFO
    mv exifinfo-*.txt ../EXIFINFO
    mv base-exif-*.txt ../EXIFINFO
 created and add thumbnails
    --> convert -thumbnail 400 somepicture.jpg tn-somepicture.jpg
     Man page says "-thumbnail geometry" - substitute geometry for value...
     --> man convert | col -b > man.convert.txt
        http://johnmeister.com/linux/Commands/man.convert.txt
 Current thought is to incorporate thumbnails into the "mkpg" script,  

    and also incorporate select parts from the exiftool and exif data.
 will likely use tables to put each thumbnail with the title/link below 

    along with the exif data for exposure info.
 Each thumbnail would be linked to open in a new tab or window, 

    an alternate would be to use javascript to do a mouse hover to show image.


It's "simple" to use a tool like WordPress or other apps to do this...

but those tools have security issues, sometimes costs and become
obsolete or not installed...
KNOWING HOW to do something is best, even if it doesn't look as good.


2) Creating a script to extract Proper Names and Places from text files
     - using sed, grep, tr, perl, sort and uniq
  From prior sessions there were plain text files created from two translations.
  The 66 books have been separated into 8 sections and have two plain text translations
    to work with, or 132 text files to parse.

The goal is to find the proper names and places which begin with an upper case character
    and extract those names from the text into a file.

That file of names will have to be manually edited to remove other upper case words,
such as "The", "Also", "Before", and so on.

I've tried to use grep to remove them. However if a word on a line contains the
first letters of another word the grep -v will also remove those words.

So manually editing 132 files is less trouble than working an automated solution.

If this were an ongoing effort with continous data coming in from a regular test or email,
then I would take the time to determine a solution.

In prior sessions we've used wget to gather these files and worked them into one line per verse text files.
The project continues to work with those files to extract specific information.  Key names and places.

current flow:

 Have gathered the text files for the 66 books and put them into the 8 major sections:
# 1-LAW  - 5 books - 10 files  - identify and list proper names showing number of times and verses
# 2-HISTORY - 12 books - 24 files
# 3-WISDOM - 5 books - 10 files
# 4-PROPHETS - 17 books - 34 files
# 5-GOSPEL - 5 books - 10 files
# 6-PAUL - 13 books - 26 files
# 7-LTRS - 8 books - 16 files
# 8-REV - 1 book - 2 files

 take each of the two files per book and create a copy:  --> cp 02n_Exo.txt 2N ; cp 02k_Exo.txt 2K (do not edit 02?_Exo.txt)
 optional:  count the lines in the copy:   --> cat 02n_Exo.txt | wc -l ; cat 2N | wc -l ; cat 02k_Exo.txt | wc -l ; cat 2K | wc -l
 use perl and sed to put each word in the file on one line:   --> perl -pi -e 's/ /\n/g' 2N 
 repeat for the other file for each book, e.g. Exodus:  --> perl -pi -e 's/ /\n/g' 2K 
 clean 2N and 2K to remove blank lines, words starting with lower case, special characters and verse numbers:
 -->  cat 2N | sort | uniq | grep -v ^[a-z] | grep -v ^[0-9] | tr -d '[{}()\;\":,\!\.\?]' | sort | uniq | grep -v ^\' > 2Na
   note: found that sort and uniq repeated was needed after removing special characters... will test with sorts at end.  
 Keep 2N original to grep for the KEYWORDS with the short list found in 2Na.
 created script with for loop to extract keywords:
    #!/bin/bash
    OUT="V-2N.txt"  #  output file
    for x in `cat 2Na`         #  create variable x from the keyword list created in "2Na"
          do
          echo $x    | tee -a $OUT                              #  list keyword
          echo "------------------------" | tee -a $OUT
          grep $x 2N | tee -a $OUT                              # display each line with the keyword
          echo "------------------------" | tee -a $OUT
          done
    ls $OUT
 Then added the count and a better separator:
--> more find-names-count-in-N2
#!/bin/bash
OUT="V-2N.txt"
cat /dev/null > $OUT
for x in `cat 2Na`
          do
          echo $x    | tee -a $OUT
          echo "------------------------" | tee -a $OUT
          echo $x | grep $x 2N | wc -l | tee -a $OUT
          echo "------------------------" | tee -a $OUT
          grep $x 2N | tee -a $OUT
          echo "------------------------" | tee -a $OUT
          echo "## KEY WORD, times found, verse(s) ##################################" | tee -a $OUT
          done
ls $OUT
---------------------------
## KEY WORD, times found, verses ################################
Naphtali
------------------------
       1
------------------------
 Exo 1:4 Dan and Naphtali, Gad and Asher.




  


 
 SESSION #112 - November 10, 2017 - FRIDAY - KEY WORD script
 
 note on html: color for cell above is "ABCDEF" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  (can use names too...) 

  
 NOTE:  no session Friday 3 Nov 2017
SESSION #112 - Nov 10, 2017 - FRIDAY - Linux at Lunch -  KEY WORD SCRIPT walk through!

The Key Word Script  uses regular expressions:
grep, sed, awk, cut, uniq, sort, perl, tr and vi to extract KEY WORDS from a text document!
Key Words are those of proper names and places; words beginning with capital letters.

The output of this script produces two Key-Word variants; one that shows only the line where the word appears,
and the other shows the context, grep -C 1, listing the line before and the line after.

It's possible to insert HTML coding into this script to create a web page, e.g.:
with:  
with:  http://johnmeister.com/linux/Scripts/mk-webpage-2015-05-08.html    

There are GUI based tools like WordPress that might create nicer pages...
but these tools have security issues, require a database, are not trivial to install, learn, use or manage.

    SIMPLE ALWAYS WORKS.


http://johnmeister.com/bible/BibleSummary/KEY-WORDs/GET-KEY.sh.txt
(will create a version of this on johnmeister.com/linux to work with generic text files.)

------------------------------------------------------------
#!/bin/bash
### john meister 29 october 2017 - copyright 2017
### script to find Key words - upper case - Who and Where
USAGE="GET-KEY.sh.txt , e.g.  GET-KEY.sh.txt 02n_Exo.txt"
#########################################################################
cat $1 | tr -d '[{}()\;\":,\!\.\?]' > T2
       ### remove most special characters cat T1, filter, save as T2
perl -pi -e 's/ /\n/g' T2
       ### put all words on one line
grep -v "^'" T2  >  T3
       ### remove leading ticks from list save as T3
cat T3 | sort | uniq | grep -v ^[a-z] | grep -v ^[0-9] > T4
       ###  sort, remove lc and numbers, save as T4
rm -f T1 T2 T3
       ### deleting temporary files
##########################################################################
echo "edit T4 in vi to remove non-keywords; :wq will resume; hit any key to start"
read
vi T4                                   # T4 will be moved to ZLIST-names when done
#########################
OUT="KEY-names-with-VS-$1"              # Key names with counts and verses
OUT2="TIMES-names-$1"                   # Key names with counts only
cat /dev/null > $OUT
  for Z in `cat T4`
     do
     echo "----------------------------------------------------------------------" | tee -a $OUT
     grep $Z $1 | wc -l > T1
        ####################
     echo "The word \"$Z\" in \"$1\" is found `cat T1| cut -c 5-9` times." | tee -a $OUT2
     echo "======================================================================" | tee -a $OUT2
        ####################
     echo "The word \"$Z\" in \"$1\" occurs in these verses `cat T1| cut -c 6-9` times:" | tee -a $OUT
     echo "----------------------------------------------------------------------" | tee -a $OUT
     grep -C 1 $Z $1 | tee -a $OUT
     echo "======================================================================" | tee -a $OUT
  done
###########################################################
mv T4 ZLIST-names-$1 ; rm -f T1
##########################################################

So, the extract files look like this for the following command(s):
-------------------------------------------------------
GET-KEY.sh.txt 01k_Gen.txt

  KEY-names-with-VS-01k_Gen.txt 2017-10-30 00:40  1.2M
  TIMES-names-01k_Gen.txt       2017-10-30 00:40   69K
  ZLIST-names-01k_Gen.txt       2017-10-30 00:40  3.8K
-------------------------------------------------------
and
-------------------------------------------------------
GET-KEY.sh.txt 01n_Gen.txt

  KEY-names-with-VS-01n_Gen.txt 2017-10-30 00:40  1.2M
  TIMES-names-01n_Gen.txt       2017-10-30 00:40   69K
  ZLIST-names-01n_Gen.txt       2017-10-30 00:40  3.9K
-------------------------------------------------------
It's interesting to see the names and places listed in just ONE of the 66 books:

http://johnmeister.com/bible/BibleSummary/KEY-WORDs/01-GENESIS/ZLIST-names-01k_Gen.txt

In addition, I want to modify the output to only show the references:

http://johnmeister.com/bible/BibleSummary/KEY-WORDs/01-GENESIS/KEY-names-with-VS-01k_Gen.txt 
e.g.
----------------------------------------------------------------------
The word "Achbor" in "01k_Gen.txt" occurs 2 times: Gen 36:38; Gen 36:39 
======================================================================
or:
----------------------------------------------------------------------
The word "Achbor" in "01k_Gen.txt" occurs 2 times:
Gen 36:38; Gen 36:39 
======================================================================

SIMPLE is portable, secure and easily modified; but it may take more effort and understanding,
and may include more "steps" and sometimes requires manual intervention; NOTICE the USE of "read" in the
script above.  I had to manually edit the "key word" list to remove unimportant words like
"The", "And", "Because", etc.  And a few other items that didn't filter out properly.

It is possible that I could create a "heredoc" to filter those words, but after converting the strings
to single words with new lines any grep -v filters would take out the rest of any words starting out
with the same beginning (e.g And would remove Andover, etc.).

That would push the filtering into the text prior to the new line insertion,
and since one has no idea which words are "important" or of interest, better to remove them manually.

One other thing, when running the script and trying to make it work, you might want to save the "key word"
list by a different name, then when editing the list you can delete the contents and read in what you
already edited.  Keep the copy file name simple and easy to remember.  It'll take a few iterations to dial it in.






 
 SESSION #113 - December 1, 2017 - FRIDAY - RECAP of 3 years of Linux at lunch!
 
 note on html: color for cell above is "fedcba" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  (can use names too...) 


 recap of 3 years worth of sessions
 http://johnmeister.com/linux/Intro-to-Linux/session-list-2017.html
  

  


 
 SESSION #114 - December 8, 2017 - FRIDAY - Oracle VirtualBox and LVM 
 
 note on html: color for cell above is "yellow" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  (can use names too...) 


 Expanding the capactity of a dynamic vdi disk in Oracle's VirtualBox
Built a default Linux 64bit VM with Centos, however, needed more space,
   had to increase the capacity from the default of 8GB to about 14GB.
https://www.virtualbox.org/

http://johnmeister.com/linux/SysAdmin/Virtualization/VM_setup_SuSE_13.2/ALL.html

Steps to increase capacity of a vdi drive in VirtualBox (Linux or MacOSx host) at the COMMAND LINE:

-->sudo VBoxManage modifyhd /home/luser/VirtualBox_VMs/Centos-Lab/Centos_Lab.vdi --resize 14480
         0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Once that was done, checked settings in VirtualBox;  was good... but local drive in VM still not expanded.

FROM INSIDE THE VM, at the command line:  
    pvs ; pvdisplay ; vgs ; vgdisplay; lvs ; lvdisplay  # displays system values
    fdisk /dev/sda      #  create a partition in the "expanded" virtual drive
        p, n, p, 3, t, 82e, w
    pvcreate /dev/sda3    ; pvs; vgs    # create the physical volume, check it
   vgextend centos_centos7-vm /dev/sda3  # extend the existing volume group into the expande physical drive
   vgs ; pvscan ; lvdisplay ; df -h; vgdisplay
   lvextend /dev/mapper/centos_centos7--vm-root /dev/sda3 # extend the logical volume into the expanded volume group
   df -h . ; vgs ; lvs
   resize2fs /dev/mapper/centos_centos7--vm-root  ## FAILURE - bug in RH LVM and/or XFS  (should have extended the file system)
## FAILURE - bug in RH LVM and/or XFS
--> resize2fs /dev/centos_centos7-vm/root   (FAILURE IN RH-type system and xfs)
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: Bad magic number in super-block while trying to open /dev/centos_centos7-vm/root
Couldn't find valid filesystem superblock.

FIX:
--> xfs_growfs  /dev/centos_centos7-vm/root

http://johnmeister.com/linux/FileSystems/ADD-6GB-to-vdi-CentOS-VM.html

process:  Order of precendence and logic:   - PHYSICAL   - VOLUME  - LOGICAL 
PHYSICAL VOLUME(s): pvs; pvdisplay; pvresize
VOLUME GROUP(s): vgs; vgdisplay; vgextend        Add physical volumes to a volume group
LOGICAL VOLUME(s): lvs ; lvdisplay; lvextend        Add space to a logical volume
http://johnmeister.com/linux/FileSystems/setup-LVM.html
http://johnmeister.com/linux/FileSystems/lvm-commands.html

  

  


 
 SESSION #115 - December 15, 2017 - FRIDAY - using find and perl
 
 note on html: color for cell above is "ivory" (rrggbb), font color is "black"

(red, green, and blue): colors between 0 and 255 (hex values= ff).  255,0,0 = red ;  0,255,0 = green  ;  0,0,255 = blue  (can use names too...) 

  NOTE:  LAST SESSION OF 2017
 using find and perl to update HTML pages for URL correction and consolidation of domains

http://johnmeister.com/linux/SysAdmin/Using-find-n-perl-to-manage-URLs.html

 also:  time, grep, col, and Apache: /etc/apache2/errors.conf  and creating a missing.html page




 
 Session #116 - January 12, 2018 - FRIDAY - using command line tools to build a web page
 
 
###############################################
1)  FILES TO MERGE INTO PARALLEL COLUMNS: used excel to create list
      (moved columns in excel), copy/paste

    http://johnmeister.com/bible/ReadInOneYear/one-year-plan.txt
  - Added leading zeros so all numbers were 3 digits for sorting, underscores for spaces.
  - Global replace in vi "touch" at the front of the line, save and: sh ./listofnames
   created empty files, ls > Daily-Passages.html, edited and:
 
    http://johnmeister.com/bible/ReadInOneYear/Daily-Passages.html

Took the list and extracted only the chapters using cut to get the ranges needed.
   ls Day_212_Jul_07_31_Isaiah_57-59.html | cut -c 25-29
        # tested placement of verses... off by 1
   ls Day_212_Jul_07_31_Isaiah_57-59.html | cut -c 26-30
   ls | cut -c 26-30 > create-files.sh
        #  then I took the list of 365 days and created the list of chapters
--> echo Day_212_Jul_07_31_Isaiah_57-59.html | cut -c 26-30
57-59       #  so, VS="57-59" and would be created for each section.
###############################################
2)  created copies: kjv, nas, greek
   cp gnt.txt g$VS.txt ; cp KJVb.txt k$VS.txt; cp NAS-NT.txt n$VS.txt
###############################################
3) vi - edit out all but chapters for reading:
    vi file, /Mark 15, k, 1G, /Mark 17,dG,:wn... last,:wq
###############################################
3) check lines on all three files (using cat and wc -l)
cat g$VS.txt | wc -l ; cat k$VS.txt | wc -l ; cat n$VS.txt | wc -l
####  time saver:  cat g*.txt | wc -l ; cat k*.txt | wc -l ; cat n*.txt | wc -l
###############################################
4) copy empty file from web directory  (would do this as the last step in the script)
   DAY1 would be file in edit, DAYN is the next day.
###############################################
5) paste tags and text together into file
  paste -d '\n' tags1.txt k$VS.txt tags2.txt n$VS.txt tags3.txt g$VS.txt tags4.txt > $DAY1
#### manual / time saver mode (grabbed from history, command line complete of Day...):
#  paste -d '\n' tags1.txt k*.txt tags2.txt n*.txt tags3.txt g*.txt tags4.txt \
#                    > Day_294_Oct_10_21_Luke_1-2.html
###############################################
6) vi $DAY1 - deleted empty rows at end;  :r FILE-BASE.txt ; type day and verses; save
###############################################
7) scp file to server and mv to web directory
###############################################
8) repeat for all 365 days: 1,189 chapters; 31,102 verses and 781,621 words (YMMV)

##########################################
   script for part of the process:
##########################################
#!/bin/bash
#############
DAY1="Day_293_Oct_10_20_Mark_15-16.html" ; VS="15-16"
DAYN="Day_294_Oct_10_21_Luke_1-2.html"
#############
cp gnt.txt g$VS.txt ; cp KJVb.txt k$VS.txt; cp NAS-NT.txt n$VS.txt
  vi ?$VS.txt
cat g$VS.txt | wc -l ; cat k$VS.txt | wc -l ; cat n$VS.txt | wc -l
read  #  safety net - edit files in a different shell - hit enter when ready
####  THE MAIN COURSE: tags have html formatting including colors
  paste -d '\n' tags1.txt k$VS.txt tags2.txt n$VS.txt tags3.txt g$VS.txt tags4.txt > $DAY1
###########
  cat FILE-BASE.txt >>  $DAY1
        #  I found that trimming the file of empty lines first was better
vi $DAY1 ; ls -al ; echo $DAY
scp $DAY1 server:/home/luser/website/ReadInOneYear/
mv $DAY1 ../ReadInOneYear/
mv ?$VS.txt hold-tmp   # a safety net - just in case... (used it once so far)
#############
echo "next day"
cp ../10_Oct/$DAYN .
# cp ../07_Jul/Day_191_Jul_07_10_Ecclesiastes_1-4.html . # manual steps for smaller sections
echo "next $DAYN" ; echo "===================="
######################################################  could queue up more day
    # DAY1="Day_286_Oct_10_13_Mark_1-3.html" ; VS="1-3"
    # DAYN="Day_287_Oct_10_14_Mark_4-5.html"
######################################################






 
 Session #117 - January 19, 2018 - FRIDAY - BASH command precedence
 
 
Discusson on command precedence and useful HTML features as demonstrated in his page:

  There are two major lessons:
1) path considerations, special characters, built-ins, commands and aliases.
 see prior material at: bashrc-n-history-details.html 
 see prior material at: basic-setup-for-BASH.html
 see example .bashrc file: bashrc-basic.html
2) practical use of HTML features, e.g. code for buttons used to hide material until expanded.






 
 Session #118 - January 26, 2018 - FRIDAY - dry run of Engineering Week presentation
 
 
 During the Engineering week presentation will provide an overview of Linux with links to more information,
and then describe a time savings of over 182 manhours using Regular Expressions in an engineering analysis.

Presentation is scheduled for Tuesday, 20 February 2018 from 1300-1400 hrs at the Bomarc Complex.



How to use Regular Expressions to save 13 hours of Microsoft Excel sorting time 
with 5 SECONDS in a Linux Shell. Seriously... there are pictures... kind of...
 (btw, I love spreadsheets... but... FIVE SECONDS!!!)







 
 Session #119 - February 9, 2018 - FRIDAY - updated Engineering Week presentation and adventures with AWK
 
 
UPDATED the material - will go through the new powerpoint


 During the Engineering week presentation will provide an overview of Linux with links to more information,
and then describe a time savings of over 182 manhours using Regular Expressions in an engineering analysis.
Presentation is scheduled for Tuesday, 20 February 2018 from 1300-1400 hrs at the Bomarc Complex.



How to use Regular Expressions to save 13 hours of Microsoft Excel sorting time 
with 5 SECONDS in a Linux Shell. Seriously... there are pictures... kind of...
 (btw, I love spreadsheets... but... less than FIVE SECONDS!!!) 

(using cygwin on Win7, pure Linux about 0.2 seconds)

Then we will discuss some simple features of AWK; the ones we can remember. 






 
 Session #120 - February 16, 2018 - FRIDAY - scripting environments
 
 

 discussion of scripts


 

a couple of scripts and PERFORMANCE




 
 Session #121 - February 23, 2018 - FRIDAY - cygwin scripting environments
 

Linux at Lunch - Session #121 - Feb 23, 2018 - cygwin, Linux 101, CSV, exiftool
Session #121 February 23, 2018 - FRIDAY - Linux at Lunch -  session opens 15 minutes before...
1) cygwin  - basic steps to setup
            (for home, BASH Win10 - older info:
            http://johnmeister.com/linux/Microsoft/Win10-BASH/Win10-BASH-install.html)
2) Linux 101 - over view of commands
3) CSV - script showing conversion of system info to CSV to Excel
    part of the Video series:  http://johnmeister.com/linux/The_Art_of_Linux_System_Administration.html
    Script To Create System Info Spreadsheet 06m23s 0808-SA-tools managing the system resources
    using a spreadsheet
4) exiftool and CSV
-------------------------------------------
This week we'll talk about installing Cygwin, show the basic setup with .bashrc and .vimrc,
 http://johnmeister.com/linux/basic-setup-for-BASH.html
 and then discuss commands briefly from the Overview page linked below, kind of a Linux 101,
basically just talking about the various grouping of commands and then moving into some scripts
to show the commands in action.  http://johnmeister.com/linux/Overview/
http://johnmeister.com/linux/Overview/LinuxOverview.pdf

ADDITIONAL INFO:
http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-1.html
http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-2.html
http://johnmeister.com/linux/Commands/vi-examples-of-sed-strings.html
http://johnmeister.com/linux/Commands/MAN/LPI-commands.html
http://johnmeister.com/linux/Intro-to-Linux/Special-Characters.pdf
http://johnmeister.com/linux/Intro-to-Linux/Slides/ALL.html
http://johnmeister.com/linux/Intro-to-Linux/OLDER-NOTES/

Then will walk through the CSV script, and will show how to read in and out of excel with
    CSV. (comma separated values).
http://johnmeister.com/linux/SysAdmin/system-info-to-CSV-sysadmin.sh.html
from :  http://johnmeister.com/linux/The_Art_of_Linux_System_Administration.html

Script To Create System Info Spreadsheet 06m23s 0808-SA-tools managing the system resources
using a spreadsheet.  Have an idea for a useful script for my photography, will run the exiftool,
filter out specific details and place them in CSV format with the server link and add them to Excel.

http://johnmeister.com/linux/Scripts/extract-base-EXIF-info.sh.html

  I can then open that file in OpenOffice and create a PDF or an HTML page for navigation.
Always interested in which lens I used, exposures and so on.
Session #121 February 23, 2018 - FRIDAY - Linux at Lunch -  session opens 15 minutes before...




 
 Session #122 - March 9, 2018 - wget, paste and reg ex tools 
 
Linux at Lunch - Session #122 - March 9, 2018 - wget, paste, sed, and other useful tools in practice
          Linux at Lunch sessions EVERY OTHER WEEK during 2018:
            March 9 (#122), March 23 (#123); April 6 (#124), April 20 (#125);
            May 4 (#126), May 18 (#127); June 8 (#128); July 27 (#129);
            August 10 (#130), Aug 24 (#131); Sep 7 (#132), Sep 21 (#133);
            Oct 19 (#134); Nov 16 (#135) - END OF SERIES (? mmv...).

Updated a long term project, you can use "wget" to copy this entire directory to a local drive,
allowing you to view it without a network.  TO DOWNLOAD:

   wget -r -np -nd -P RKNG http://johnmeister.com/tech/RKNG

alternate method:
mkdir RKNG ; cd RKNG ; wget -r -np -nd  http://johnmeister.com/tech/RKNG

the -r is for recursive (goes down the path listed,
    can be limited with a "-l # " , e.g. "  -l 2 " for two levels down

the  -np is for "no parent"  (doesn't go back up to the root file system - only gets the path specified)

the  -nd is to prevent copying the entire directory structure from the top down... FQDN and path...

the  -P RKNG - is the designated path to copy to.
        use whatever name you prefer, or mkdir "name" ; cd "name" 

That project made use of a lot of sed, grep, awk, vi, perl and vi edits.
It is a self-contained directory with no external links,
using only internal html references using name and href designations.

Useful links for learning more about Linux:
 http://johnmeister.com/linux/Overview/
http://johnmeister.com/linux/Commands/commands-bash.txt
http://johnmeister.com/linux/vi/ - seriously vi is the only editor worth
        really learning if you're going to work at the command line in Linux/UNIX
 http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-1.html
http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-2.html
http://johnmeister.com/linux/Commands/Man-Pages-Lab/create-man-txt.sh.txt
http://johnmeister.com/linux/Commands/vi-examples-of-sed-strings.html
http://johnmeister.com/linux/Commands/MAN/LPI-commands.html
http://johnmeister.com/linux/Intro-to-Linux/Special-Characters.pdf
http://johnmeister.com/linux/Intro-to-Linux/Slides/ALL.html
http://johnmeister.com/linux/Intro-to-Linux/OLDER-NOTES/
http://johnmeister.com/linux/Scripts/Engineering/
for home, BASH Win10 - older info: http://johnmeister.com/linux/Microsoft/Win10-BASH/

https://www.smashwords.com/books/search?query=John+Meister  link to e-books
http://johnmeister.com/linux/The_Art_of_Linux_System_Administration.html
 http://www.oreilly.com/pub/au/6963 

giving some thought to the following option, not completely sure yet:
 < esc >:wq!  




 
 Session #123 - March 23, 2018 - /bin vs. /usr/bin; systemd (systemctl and journalctl) 
 
remember:  SIMPLE ALWAYS WORKS!  Find the lowest common denominator in your setup and tools, master them.

Two topics this week:
1) FHS (Filesystem Hierarchy Standard) info related to setting up .bashrc and related VIM issues:

http://johnmeister.com/linux/FileSystems/Linux-bashrc-vim-differences.jpg

2) "very" quick refresher on systemd, logs, and services:
http://johnmeister.com/linux/Notes/systemd-info.html

3) open talk on future sessions - thinking going through very basic regular expression use might be helpful... thoughts?



 
 Session #124 - March 30, 2018 - building interoperability 
 
remember:  SIMPLE ALWAYS WORKS!  Find the lowest common denominator in your setup and tools, master them.

March 30, FRIDAY - Linux at Lunch
Linux at Lunch - Session #124 - building a base of interoperability with Linux

Two topics this week:
Discuss a variety of file management practices to ensure interoperability.
Using Cygwin on Windows and working with Linux or Mac with the same files.
Moving files between systems on a network, via email or via thumbdrive.
Setting up a thumbdrive to work on Mac, Linux and Microsoft.
Using rsync, winscp, putty, Exceed, VNC, VM's, ssh, scp, config mgt and backup tools.
Linux commands involving  ssh setup, rysnc, scp, and creating a function or alias to make backup copies of files.

Useful links for learning more about Linux:
http://johnmeister.com/linux/Overview/
http://johnmeister.com/linux/Commands/commands-bash.html
http://johnmeister.com/linux/vi/ - the only editor worth learning if you're going to work at the command line in Linux/UNIX
 http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-1.html
http://johnmeister.com/linux/Intro-to-Linux/lab-exercise-2.html
http://johnmeister.com/linux/Commands/Man-Pages-Lab/create-man-txt.sh.html
http://johnmeister.com/linux/Commands/vi-examples-of-sed-strings.html
http://johnmeister.com/linux/Commands/MAN/LPI-commands.html
http://johnmeister.com/linux/Intro-to-Linux/Special-Characters.pdf
http://johnmeister.com/linux/Intro-to-Linux/Slides/ALL.html
http://johnmeister.com/linux/Intro-to-Linux/OLDER-NOTES/
http://johnmeister.com/linux/Scripts/Engineering/
for home, BASH Win10 - older info: http://johnmeister.com/linux/Microsoft/Win10-BASH/
https://www.smashwords.com/books/search?query=John+Meister  link to e-books
http://johnmeister.com/linux/The_Art_of_Linux_System_Administration.html
http://www.oreilly.com/pub/au/6963 videos:  http://shop.oreilly.com/product/0636920050209.do



 
 Session #125 - April 6, 2018 - BASH Commands in CYGWIN 
 
remember:  SIMPLE ALWAYS WORKS!  Find the lowest common denominator in your setup and tools, master them.

April 6, FRIDAY - Linux at Lunch
Linux at Lunch - Session #125 - Real World Linux Commands - in CYGWIN


http://johnmeister.com/linux/Notes/Real-world-Linux-Commands.html





 
 Session #126 - April 13, 2018 - BASH Commands in CYGWIN - continued 
 
remember:  SIMPLE ALWAYS WORKS!  Find the lowest common denominator in your setup and tools, master them.

April 13, FRIDAY - Linux at Lunch
Linux at Lunch - Session #126 - Real World Linux Commands - in CYGWIN continued.
Working through the list of commands to see which are built-in, which are commands and which are not there.
 http://johnmeister.com/linux/Notes/Real-world-Linux-Commands.html





 
 Session #127 - April 27, 2018 - BASH Commands in CYGWIN - continued 
 
remember:  SIMPLE ALWAYS WORKS!  Find the lowest common denominator in your setup and tools, master them.

April 27, FRIDAY - Linux at Lunch
Linux at Lunch - Session #127 - Real World Linux Commands - in CYGWIN continued.
Working through the list of commands to see which are built-in, which are commands and which are not there.

THIS WEEK: parsing info from the Microsoft "systeminfo" command, and also some CSV data from Active Directory.
    Will identify details about a system and create a CSV file to input into Excel, or OpenOffice, or LibreOffice.

 http://johnmeister.com/linux/SysAdmin/system-info-to-CSV-sysadmin.sh.html 

 http://johnmeister.com/linux/Notes/Real-world-Linux-Commands.html






 
 Session #128 - May 11, 2018 - BASH Commands in CYGWIN - continued 
 
Session #128- May 11, FRIDAY: SIMPLE ALWAYS WORKS!    just not always as fast as we'd like...

Plan to continue working with CygWin and using it for parsing information;
                while not as fast as native Linux, it's cost-effective (as in free and no additional hardware required) and convenient.
I recognize that this tool is the best way of getting power users familiar with regular expressions and Linux!

I have been testing various configurations and working with others to learn more about this tool.  I was able to get ssh working in Cygwin and am
researching it further.  The plan is to see if we can ssh between Microsoft systems!   This will greatly help me manage my computer lab!!!!
Also considering making X-windows work in Cygwin...  lots to explore with this tool, one computer, one OS, but many tools!

The focus on the sessions will be about the commands and scripts,
    and I will comment on how these commands relate to the Linux environment where they differ.

Using CYGWIN for these sessions provides the LOWEST COMMON DENOMINATOR to help folks learn regular expressions and Linux commands.





 
 Session #129 - June 8, 2018 - GPARTED & SuSE 42.3 update; CygWin/BASH continued
 
Session #129- June 1, FRIDAY: SIMPLE ALWAYS WORKS!

SITUATION:  Toshiba R600 Portege laptop, 500GB drive - SuSE 42.2 - EOL.  Needed to update, and low on space.
RESOURCES:  DVD with SuSE 42.3 Leap; 1TB SSD from recently deceased laptop; and Gparted on a USB stick.

Thought I would need to use EZGig or Clonezilla... EZgig was version 2... out of date, failed... downloaded v6, but
it required the hardware device... long ago deceased and disposed... Tried Clonezilla... bad disk... dug out my USB stick with
Gparted that we've used in prior session to rescue systems, and dd for cloning:
http://johnmeister.com/linux/Notes/Gparted-for-Recovery/ALL.html
http://johnmeister.com/linux/FileSystems/rescuing-failing-drive-w-dd.html

http://johnmeister.com/linux/FileSystems/Gparted-clone-expand-partition-n-upgrade-SuSE.html

But I forgot that one can "copy" partitions with Gparted...  so I tested it, and it worked.

Getting the system to boot to the correct drive was a challenge... had to be careful of /dev/sda as it had the good
SuSE 42.2 with MATE... attached to a USB SATA dual drive unit was the target 1TB SSD (which had a full up MINT/MATE
environment on it with a corrupted MATE Menu, the files had already been copied to another 1TB SATA drive). And
then of course the USB stick with Gparted (Debian Linux), which was what I needed to boot to.  It took a few tries
to get the correct boot device, but once Gparted started it was all good.  What I ended up doing was removing
the 1TB drive from the USB/SATA "toaster" and that allowed the system to find the USB stick, once the Gparted screen
appeared and BEFORE it loaded its kernel, I reinserted the 1TB drive.  If you boot into the Gparted kernel without
both drives (source and target) in place it won't work, it needs to be registered in the kernel when started.

The process steps were:

1) boot into Gparted (disk or thumbdrive) - make sure all disks are attached
2) identify drives, e.g.

    /dev/sda1  swap 10GB
    /dev/sda2  root 138GB  (SuSE 42.2)
    /dev/sda3  home 318GB

    /dev/sdb1  swap 8GB
    /dev/sdb2  root 989GB  (Mint/Mate)

    /dev/sdc1  root 512M  (Gparted/Debian)

3) select /dev/sdb - create partition (gpt)
    - yes, delete all partitions ; apply

4) select /dev/sda1 - copy ; select /dev/sdb, paste
    (leave the same size, 10GB for 8GB of memory is adequate for swap)

5) select /dev/sda2 - copy ; select /dev/sdb, paste
    (leave the same size, 138GB for the OS is adequate)

6) select /dev/sda3 - copy ; select /dev/sdb, paste
    - AND expand to fill the drive 998GB or whatever.

7) after being completely certain you did this right, click APPLY and wait...
    for what may be a long time...

8) once complete reboot the system verify it's ok...
    this is where the real work begins.

9) disassemble the laptop and swap the drives, making sure you keep them straight.
    remember, the original drive is still good and correct.

10) swap drives (YMMV), boot up to the cloned drive... good luck, you'll need it.
    - at this point things can get very complicated...
    - if your system used UUID instead of KERNEL devices (i.e. some cryptic string, vs. /dev/sda1)
       you will have a problem... the UUID of the device you cloned will be different...
       that means you'll have to edit /etc/fstab and likely go into /boot and grub.cfg and so on.
    - if you can't boot at this point, and you're smart enough to use OpenSuSE, you have options.

11) Assuming you were bright enough to use OpenSuSE, you have options.
    - you can fiddle with files, or boot up to the install media.
    - when booting to the SuSE Install disk, select upgrade (if you select install the disk
        is smart enough to realize that it's already installed, and allow you to upgrade,
        which isn't something that often happens with other distros... of course, YMMV.)
    - when doing the upgrade I always select at least one or two additional packages,
        by doing this I make sure the system is changed so that the boot features are also changed.

12) the system will do it's upgrade and usually fix the booting issue.
    - if not either dig into grub or put the old drive back in.
    - once the old drive is in, modify grub.cfg and /etc/fstab to show KERNEL drivers, not UUID,
      or you can get fancy and fake out the UUID on your system to make the new drive appear to be
      the old drive... confused?  yeah, me too... so I ignore the warning about kernel names and
      go that way... however, if you swap drives around on your bus by accident, that will BITE you! be careful.
    once the system boots up you can restore your old repos (non-oss) and make sure everything is working.
    (I forgot to save my old repos... needed non-oss for virtualbox and VLC... not hard to add back in.).

This was way more fun than CygWin...



 
 
 Session #130 - June 15, 2018 - "home brew" revision control
 
 
Session #130- June 15, FRIDAY: SIMPLE ALWAYS WORKS! - "home brew" revision control and file management

Talking with a few of the regular attenders I feel inspired to discuss the creation of a "home brew" revision control system.

I've "cooked" up a few things over the years to manage system files and notes.
Keeping it simple keeps it portable.

Simple is what you can remember... YMMV.


http://johnmeister.com/linux/Scripts/wrapper-copy-w-date.sh.html 

http://johnmeister.com/linux/Scripts/wrapper-KSH-experiment.html 

http://johnmeister.com/linux/Scripts/wrapper-script.html 

then, after you get a wrapper script or PROCESS in place, you can expand it across systems
and do some things with ssh/scp and rsync:


http://johnmeister.com/linux/Scripts/rsync-directories.sh.html 

http://johnmeister.com/linux/Scripts/run-now-on-all-hosts.sh.html 


#!/bin/bash
# vicp.sh - created 2015-09-01 jm - wrapper script
#  updated 2018-06-12 to add a "BAK" directory in local path, be sure to chmod 755
##################################################################################
#  if desired, add an archive Directory path to copied file name
#  /usr/bin/cp $1 $1.before-`date +'%Y%m%d-%H%M'`  # BEFORE
#  /usr/bin/cp $1 BAK/$1.after-`date +'%Y%m%d-%H%M'`  # AFTER
##################################################################################
if [ ! -d BAK ]                 # tests for directory, if not, then mkdir...
#  echo in-loop   # commented to test loop - had a problem with special characters
  then
  mkdir BAK
#  echo dirbak # comment to check process.. a space, a tick and an apostrophe...
  echo "history directory made"
fi
cp $1 BAK/$1.before-`date +'%Y%m%d-%H%M'`
#  ls -al BAK    # added this to see that the loop worked (cutting and pasting has issues!)
vi $1
cp $1 BAK/$1.after-`date +'%Y%m%d-%H%M'`
##################################################################################


orking as a Systems Administrator you do NOT always have the luxury of advanced tools and applications...
while there are good tools available in UNIX and Linux and likely Cygwin, they are NOT installed by default,
and if they are they may not be configured. You can't trust them.

So, yeah, I know there are tools, but they're not always in YOUR tool box when you're dealing with bare metal
or on "other" systems. My focus in training and sharing is on what I've used and can remember from the real world.
If you're using a system all the time in a stable environment you may be able to add the "better" tools...
or do what I did and "grow" my scripts to do more things.

These scripts work because I've used them to manage enterprises.

I make them work, then build a web page to share them... and...
cut and paste into any environment I might find myself...
because trying to remember some of these things is a challenge...
syntax can be complex. Simple Always Works.
I'm pretty simple, and always seem to be working... ...



 
 
 Session #131 - July 27, 2018 - 
 
 
Session #131- July 27, FRIDAY: SIMPLE ALWAYS WORKS!

1)migrated from 4TB to 8TB -
    used parted, http://johnmeister.com/linux/FileSystems/usingParted8TB.html

2) then used rsync and dd - rsync from 4TB to 8TB took over a day... dd has been almost a week... not done...

3) using script to rsync and update files between computers -
      strategy for use including using external drives kept in air tight metal box.

4) modifying a script to sequentially number files:
  http://johnmeister.com/linux/Scripts/number-file-names.sh.html number files from 1 to 1,189
  http://bibletech.net/BY-CHAPTER/ and then create an index that matches what's already encoded!
  Will be using awk, sed, and likely perl.
 Some of the scripts will be on cygwin, others on SuSE.

other scripts useful for archiving:
http://johnmeister.com/linux/Scripts/wrapper-copy-w-date.sh.html
http://johnmeister.com/linux/Scripts/wrapper-KSH-experiment.html
http://johnmeister.com/linux/Scripts/wrapper-script.html
after you get a wrapper script or PROCESS in place, you can expand it across systems
     and do some things with ssh/scp and rsync:
http://johnmeister.com/linux/Scripts/rsync-directories.sh.html
http://johnmeister.com/linux/Scripts/run-now-on-all-hosts.sh.html




 
 
 Session #132 - August 17, 2018 - 
 
 
Session #132- August 17, FRIDAY: SIMPLE ALWAYS WORKS!   

1) dd of 8TB - took 16.36 days... not doing THAT again. :)
http://johnmeister.com/linux/FileSystems/usingParted8TB.html
http://johnmeister.com/linux/FileSystems/rescuing-failing-drive-w-dd.html
------------------------------------------------
root@LOVING-LINUX -root- [/root]
------------------------------------------------
--> dd if=/dev/sdc of=/dev/sdd
------------------------------------------------
dd: reading `/dev/sdc': Input/output error
43184776+0 records in
43184776+0 records out
22110605312 bytes (22 GB) copied, 4404.01 s, 5.0 MB/s
------------------------------------------------

2)   rsync script:

3) sequentially number files - had to do a bit of manual processing...
        http://johnmeister.com/linux/Scripts/number-file-names.sh.html,
had to  number files from 1 to 1,189 to get:  http://bibletech.net from http://bibletech.net/READ

4) used paste, head, cut, cat, and a few other commands to merge several separate files into 1,189 html files.

5)  Reordered the daily reading to follow the Hebrew version's ordering of these texts...
http://bibletech.net/Mikrah  - looking at it I need to create an index file for it...
that will involve:   ls > index.html ; vi index.html  ;  :%s$.*$ & $g  ;
and then will need to put them into various sections, add headings and insert the table format associated with my base README.html template.
Which is mostly laid out in this script:
http://johnmeister.com/linux/Scripts/mk-webpage-2015-05-08.html  this creates an "ALL.html" page,




 
 
 Session #133 - September 7, 2018 - 
 
 

Session #133- September 7, FRIDAY: SIMPLE ALWAYS WORKS!   
1) using grep, find and perl...
   or how to fix 3,359 web pages in 8 minutes and 40 seconds.
- 
http://johnmeister.com/linux/Notes/Perl-to-fix-incomplete-URL.html 
- 
http://johnmeister.com/linux/Commands/PERL-examples-2018-09-Sep-06.txt
- will fix some entries on all the pages in real time, e.g.:
   a) grep for suspect string, pick a file:
     grep "Cor rinthian" *   # 56 files were found with this error.
     1091_47_2_Corinthians_13_nas.html:<th colspan="3" bgcolor="aaccff">  2 Cor rinthians 13 <!-- DAY and VERSE info here --> </th>
   b)  WHAT is a good idea at this point is to make a COPY of the file in tmp as a precaution!
     cp 1091_47_2_Corinthians_13_nas.html /tmp/save-for-a-bit   # (history will help you remember the real name)
   c) select one file, as noted above, try perl string, grep again or edit file to validate,
     perl -pi -e 's/Cor rinthians /Corinthians /g' 1091_47_2_Corinthians_13_nas.html
   d) test: grep "Cor rinthian" 1091_47_2_Corinthians_13_nas.html    # expected response is none... nothing... "silence is golden"
   e) now, recall history and change the file name 1091... to a wildcard, or restricted wildcard, e.g. *Cori*
   f) perl -pi -e 's/Cor rinthians /Corinthians /g' *
   g) at this point there are two other directories that have related files, would cd ../OTHERDIR1, recall command via history, then cd again
   h) while not completely necessary, removing the file from /tmp (or where ever you parked it) is a good idea.

2) combine the use of find to fix or change various strings in the text, for example:
   # grep "" *     # two files came up in the subdirectory
   # perl -pi -e 's///g' bashrc-n-history-details.html    #  this fixed one file, there's one left
   # find . -type f -name 'setup-diverse-systems-prototype.sh.html'  -exec perl -pi -e 's///g' {} \;

grep "a string" *     # two files
perl -pi -e 's/a string//g' bashrc-n-history-details.html
grep "a string" *    # just one file
find . -type f -name 'setup-diverse-systems-prototype.sh.html'  -exec perl -pi -e 's/a string//g' {} \;
grep "a string" *    # none found

   At this point we change the string to hit all html files and go to the top of each directory where this might exist.

GOOD IDEA TO TEST FIRST:
find . -type f -name '*.html'  -exec grep "a string" {} \;
  <img src="http://johnmeister.com/pix/practical-suggestions-for-Microsoft-Windows-small.jpg" width="86%"><br></a>
THEN EXECUTE REPLACEMENT OF STRING:
find . -type f -name '*.html'  -exec perl -pi -e 's/a string//g' {} \;

--> time find . -type f -name '*.html'  -exec grep "even free" {} \;

real    4m17.062s
user    0m57.338s
sys 0m31.313s

 
   SEE ALSO:  http://johnmeister.com/linux/SysAdmin/How-to-update-18-thousand-pages.html  for use of xargs or just find... 

3) time permitting, review the work of a couple years' using wget, perl, sed, grep, awk, vi, html and find...
   to build a complete website that works with any browser or device and is portable and stands alone.




 
 
 Session #134 - September 14, 2018 - 
 
 

Session #134- September 14, FRIDAY: SIMPLE ALWAYS WORKS!   
TWO THINGS TO SORT OUT:
    1) ERROR:   xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
    2) difference in counts between xargs and -exec when using the same command
         (...and of course any differences with grep -i and escaping ".")

Once again, upon searching with this error I find myself once again in
     the WASTELAND of FORUMS with misinformation and unanswered questions.
Leaving me to setup a known situation and case and testing the variables myself.
Will also try to do this in Cygwin so we can also use find and perl on Windows...

HOWEVER, one of the lunch time linux folks suggested finding the rogue file, found it, fixed it and updated server...

------------------------------------------------
## using xargs gave an error, and a slight different count than using -exec
------------------------------------------------
--> time find . -type f -name "*.html" | xargs grep -i "johnmeister.com" | wc -l
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
52843
   real 0m10.360s
   user 0m5.439s
   sys 0m0.321s
------------------------------------------------
--> time find . -type f -name "*.html" -exec grep "wagoneers\.com" {} \; | wc -l
61947
   real 3m29.839s
   user 0m53.282s
   sys 0m25.615s
------------------------------------------------
Here's the steps used to sort out the single quote thing and update the server:

    
    http://johnmeister.com/linux/Scripts/Using-find-perl-to-fix-wagoneers.com.html

---------
UPDATE FROM SESSION:
find tmp -type f -name 'sed-xargs-test.txt' | xargs sed -ir 's$before$after$g'
    sed with a "-i" = in place editing
    sed with a "-r" = extended regex   (more perl compatible) (still must be carefull with parens!)

the find with parens destroyed thousands of files... must escape parens:  \(  \)

NOTE:
sed 's/[a-z]*\(-processing\)/post\1/' 
sed -r 's/[a-z]*(-processing)/post\1/' 




 
 
 Session #135 - September 21, 2018 - 
 
 

Session #135- September 21, FRIDAY: READING a file into another file...
SITUATION: Have 1,189 web pages that I'd like to add NAVIGATION in to.

However, there are 66 groupings of these pages with different navigation.

So I need to create a custom navigation for each of these 66 groupings -
have a basic set of internal vi sed commands.

Once that's created I need to insert that navigation, or more correctly,
a Table of Contents, into each of those like pages.

Then scp or rsync my local system with my server and test.


1) I have a manual process:
    1) vi *files-with-same-name*
    2) <esc> / key words
    3) dd ; <esc>:r toc-filename
    4) <esc>:wn
    5) repeat until fingers bleed then scp -rp <updated files> server:/path

2) someone said PYTHON, his name was NOT monty...  so... with his help I got it working:
        1) 1 created the toc (table of contents) file, then found the key word search string
        2) dummied up files with simple names and tested - worked
        3) attempted to insert a simple for loop for *files-with-same-name*, failed
        4) BRUTE FORCE SCRIPT worked:  http://johnmeister.com/linux/Scripts/Python-script-attempt-without-a-loop.txt
        5) the use of python without knowledge, training or a clear cheat sheet was MORE WORK

3) someone said SED... it wasn't Fred... but it was said by more one... so...
        1) got some basic stuff working...
        2) can find the pattern and read the toc file... however, it doesn't output the original file to stdout
        3) I could not get sed to edit in place with the pattern as stated
        4) am reading the sed&awk book by oreilly and hoping to write a sed script before Friday
        5) one possibility is to replace the complex with PERL, then...
        6) sed '/banana/r file2' file1 > newfile
        7) GOOD NEWS at 23:13:38 19 Sep - PROBLEM: needed to add ".bak" to the -i in MacOSX BASH / SED!!!
                  Once I escaped all the special characters I still got this error:
                sed: 1: "mal.html": invalid command code m
                SEARCH pointed out the problem was a "safety feature".
    sed -i.bak '/\<br\>\<center\>\<hr width=\"50%\"\>\<\/center\>\<\/td\>/r toc-mal' mal.html

       8) now I can write a for loop to get through all of the groupings!
       9) thanks to the folks who said to use SED.    :)
      10) one other note... had to add one more line to remove the string that was supposed to be replaced:
     sed -i.bak '/\<br\>\<center\>\<hr width=\"50%\"\>\<\/center\>\<\/td\>/d' mal.html

4) PERL still seems like a viable tool...
        1) however, my perl scripting class is long expired (1998)
        2) but... I can use my perl -pi -e 's/complex string for/ banana/g'  files
            - and then use the sed line from #6 above.
        3) I'm going to have my nose in the O'Reilly book on sed & awk and see what h and H and r really do in sed.
        4) Considering the damage I did with perl on prior sessions, I'm stopping at SED this trip.
        5) although perl is still in my tool box, just fixed an error where I had "Revelation" in the title of Isaiah!

        --> grep Revelation *Isa*     #  oops...  all 66 books had it!  didn't edit my template properly
        --> perl -pi -e 's/Revelation/Isaiah/g' *Isa*
        --> grep Revelation *Isa*     #  all gone...



ONE THING I'VE LEARNED is that a bunch of my really smart friends will say, oh, just use this or that...
most of them just leave you with that, or say RTM...
    or they could do it with this or that tool... and let you research it.

I've heard and experienced this for decades... it's very common for a developer or advanced sys admin to
say these things, it's another for those helpful and knowledgeable folks to actually help with the exact
syntax and make it work.... generally it is never as easy as any of us thought.

But every little bit of info, advice and suggestions helps.

Have a friend back east who has been helping me with Python...
he's run into the same issues I have, but worked it through... he's promised to show me
how to setup a for loop tonight (Wed)...

I'm trying to make a generic script that does this and just change the variables.  I have
manually edited those 1,189 files several times...
and have ideas for more enhancements.  :)
Despite my lack of coding skills, the web site is coming along nicely.





 
 
 Session #136 - September 28, 2018 - 
 
 

Session #136- September 28, FRIDAY: READING a file into another file...
SITUATION RESOLVED: 1,189 web pages have been updated.
Linux at Lunch - Session #136 - 9/28/2018 - completed file insertion using SED and SHELL

FRIDAY'S SESSION:

READ a file into another file... COMPLETED...
1,189 web pages NAVIGATION added (by Monday night!)...

review and discuss the process and tools used.
Show how using and editing the history allowed for faster turnaround by adding commands via semicolon on one line.

http://johnmeister.com/linux/Scripts/SED-used-to-insert-Table-of-Content-in-HTML-pages.txt

http://johnmeister.com/BIBLE/STUDY



 
 
 Session #137 - October 12, 2018 - 
 
 

Session #137 Linux at Lunch - 10/12/2018 - review environments and scripts, using cygwin for windows commands
FRIDAY'S SESSION:
    - discuss environments (.bashrc)   http://johnmeister.com/linux/basic-setup-for-BASH.html
    - review scripts and discuss:   http://johnmeister.com/linux/Scripts/
    - discuss cmd in cygwin:   http://johnmeister.com/linux/Microsoft/CygWin-when-penguins-are-not-allowed-to-roam-free.html




 
 
 Session #138 - October 26, 2018 - 
 
 
Session #138 Linux at Lunch - 10/26/2018 -
Basics of RAID - (redundant array of inexpensive disks):
 
 http://johnmeister.com/linux/FileSystems/RAID-info.html

logical volume manager on Linux will allow you to use JBOD for RAID:
 
 http://johnmeister.com/linux/FileSystems/setup-LVM.html 

using more than one computer at home and with external drives connected via USB is a reasonable backup method:
will discuss different scenarios, and using rsync:

http://johnmeister.com/linux/Scripts/rsync-directories.sh.htm



 
 
 Session #139 - November 9, 2018 - 
 
 
Session #139 Session #139 - November 9 - Linux at Lunch
This SESSION: crontab, wrapper scripts and backup techniques

- wrapper script to create a copy of a file with a sortable date prefixed to file name
    http://johnmeister.com/linux/Scripts/using-DATE-prefix-or-suffix.html
    http://johnmeister.com/linux/Scripts/wrapper-script.html
    http://johnmeister.com/linux/Scripts/wrapper-copy-w-date.sh.html
    http://johnmeister.com/linux/Scripts/filedate.sh.html
- identify a list of key files: e.g. .bashrc, .vimrc, .viminfo, .ssh/authorized_keys, /etc/hosts, /etc/passwd, etc.
        http://johnmeister.com/linux/Scripts/Replace-README-using-find.html
        http://johnmeister.com/linux/SysAdmin/Editing-Apache2-etc-default-server.conf.html
- create script to check key files and create a dated backup copy of updates in ~/bin/KEY
        http://johnmeister.com/linux/Scripts/LOOPS-shell-examples.html
        http://johnmeister.com/linux/SysAdmin/system-check-script-July-2012.sh.html
- create rsync script to sync directory of backup copies with another system
        http://johnmeister.com/linux/Scripts/rsync-directories.sh.html
        http://johnmeister.com/linux/SysAdmin/Using-rsync.html
        http://johnmeister.com/linux/Notes/RSYNC-cmd.html
        http://johnmeister.com/linux/SysAdmin/SysAdmin-scripts-n-tools.html
- incorporate scripts to check key files and rysnc into crontab
        http://johnmeister.com/linux/Notes/crontab-email-setup.html
        http://johnmeister.com/linux/Notes/wget-n-crontab.html
        http://johnmeister.com/linux/SysAdmin/Cron-Rsync-sync-up-drives.sh.html
        http://johnmeister.com/linux/FileSystems/man.rsync.txt
- demonstrate how to use tar and compress (etc.)  to create an additional backup copy to be archived
        http://johnmeister.com/linux/FileSystems/creating-TAR-n-ZIP.html
        http://johnmeister.com/linux/Commands/MAN/man.tar.html
        http://johnmeister.com/linux/Notes/how-to-TAR-home.html
        http://johnmeister.com/linux/SysAdmin/UNIX/OldTapeBackup-ufsdump-restore/



 
 
 sessions continue at:

 http://johnmeister.com/linux/Intro-to-Linux/One-Hour-Linux-Sessions-2019.html  


 
 
 Session #140 - November 30, 2018 - 
 
 
Session #140 Linux at Lunch - 11/30/2018 - LAST SESSION 
 SESSION #1 was held on November 26, 2014 
this will be the last session, at least for 2018...
    ending this series at #140 on Nov 30, 2018  - 4 years and 4 days...

THIS SESSION:
- using cygwin on windows for file management and regular expressions
             
             http://johnmeister.com/linux/Scripts/Engineering/Get-a-CLUE-w-Linux.pdf
- using time to see how well cygwin performs (cf. w/linux if possible)   (time )
- fix-filenames:  a script to undo bad habits
    
    http://johnmeister.com/linux/Scripts/fix-filenames.sh.html
- determine disk hogs  (du -sh) and other file management tricks of sys admins (that create more problems later! :)
- find large files and move, remove, etc.  (find ....) (page with examples is AWOL... !!!)
       
       http://johnmeister.com/linux/Commands/find-copy.html
- using script to capture tasks you'd like to make into a script  (script name.RAW; cat name.RAW | col -b > name.txt )
- using windows command in cygwin



 
 
 Session #141 - January 11, 2019 - 
 
 
Session #141 Linux at Lunch - 01/11/2019  ... was going to stop but...  will explain during the session.
 SESSION #1 was held on November 26, 2014 

THIS SESSION:

- setup a new laptop using SUSE LINUX LEAP 15.0 - did an update and stuff broke... so I nuked it...
    installed LINUX MINT MATE 19.0 Tara... and... recorded (most of) the steps:

    http://johnmeister.com/linux/SysAdmin/Install-n-Configure-Linux-MINT-MATE.html 
    
... found that DNS was broken...  so... I googled the issue and found the villian:  systemd-resolved.
    
why???  how hard is it to enter a line in /etc/resolv.conf?
    
    http://johnmeister.com/linux/SysAdmin/Undo-Systemd-Resolved-a-script-and-results.html
    
- created a script to clean up my "bin" directories on all of my Linux systems... the issue is that when I
rsync I'm picking up subdirectories that have source s/w... so rather than trying to identify which directories
to rsync, I will only rsync the top level of bin.  So my archived or out of date scripts that I use for reference
will need to be moved out of "ARCHIVED" into the main directory. The plan is to write a script to prepend the file
name with "z_" and then add a suffix with the date.  This way I'll have backup copies and a library of old scripts
from which to recover the "perls of wisdom", especially important because of CRS (can't remember stuff).
I looked through old scripts and these class notes to find the technique...   Worked first try.
    
    http://johnmeister.com/linux/Scripts/z-archive-it.html
                
- using cygwin on windows for file management and regular expressions -
    this will include using "du", find and even rsync to manage backups of local files to thumbdrives or shares.
             
             http://johnmeister.com/linux/Scripts/Engineering/Get-a-CLUE-w-Linux.pdf

- using time to see how well cygwin performs (cf. w/linux if possible)   (time )

- fix-filenames:  a script to undo bad habits
    
    http://johnmeister.com/linux/Scripts/fix-filenames.sh.html

- determine disk hogs  (du -sh) and other file management tricks of sys admins (that create more problems later! :)

- find large files and move, remove, etc.  (find ....) (page with examples is AWOL... !!!)
       
       http://johnmeister.com/linux/Commands/find-copy.html

- using script to capture tasks you'd like to make into a script  (script name.RAW; cat name.RAW | col -b > name.txt )

- using windows command in cygwin  - one such interoperable script would be to clean windows:
 
    http://johnmeister.com/linux/Microsoft/batchfile-to-clean-your-windows.html
    (of course this script/batch file needs to have documents and settings changed to user...)



 
 
 Session #142 - January 18, 2019 - 
 
 
Session #142 Linux at Lunch - 01/18/2019

THIS SESSION:

- follow up on performance improvements on Linux laptop after a few minor hardware changes (Memory and SSD upgrade)
     - on HP Elitebook 840/G3 started with a 256GB SSD with "adequate" performance and 8gb of memory.
  LINUX MINT 19.0 with MATE desktop results:
        - first test:  baseline 256GB ssd/8gb memory:                       13.33 seconds - base performance
        - first step was doubling the memory, 256gb ssd/16gb memory:         4.67 seconds - 65% improvement! (2x memory!)
        - second step was to replace ssd with faster 1TB ssd/16gb:           3.33 seconds - an additional 28.57% improvement.

  OVERALL PERFORMANCE IMPROVEMENT on LINUX was 75%!!!  with a $175 ssd and $40 memory chip on a $299 used laptop.
    - performance test script is:  http://johnmeister.com/linux/Microsoft/LOADTEST-OS-COMPARISON.html

- created a script to create an index page, thumbnail index page and the ALL.html for my photography:
    
    http://johnmeister.com/linux/Scripts/make-page-index-with-thumbnails.txt

    INDEX:  http://johnmeister.com/2019/01-Jan-13-Skagit-Birds/index.html
    THUMBNAIL: http://johnmeister.com/2019/01-Jan-13-Skagit-Birds/tindex.html
    ALL.html: http://johnmeister.com/2019/01-Jan-13-Skagit-Birds/ALL.html

-  will walk through this script as well:    
    
    http://johnmeister.com/linux/Scripts/z-archive-it.html

- fix-filenames:  a script to undo bad habits - created a quick for loop to fix a number of directories
    
    http://johnmeister.com/linux/Scripts/for-loop-for-fix-filenames.txt

    
    http://johnmeister.com/linux/Scripts/fix-filenames.sh.html




 
 
 Session #143 - February 1, 2019 - 
 
 
Session #143 Linux at Lunch - 02/01/2019

THIS SESSION:
 1) improved the script for thumbnails:

    http://johnmeister.com/linux/Scripts/make-page-index-with-thumbnails.txt
output example: http://johnmeister.com/2019/01-Jan-26-Mountains/tindex.html

key featured: "convert"
    The convert program is a member of the ImageMagick(1) suite of tools.

 2) cygwin updating in the lab:
    The process to update cygwin has been to delete everything in LAB_USERS and cygwin64 prior to a Microsoft copy.
    FIRST:   copy files from a system that is configured, either "push" or "pull"
         xcopy /E  c:\LAB_USERS \\LAB-K2\C$\LAB_USERS
            xcopy /E  \\LAB-E3\C$\LAB_USERS c:\LAB_USERS
    THEN,  xcopy the cygwin files into the cygwin64 directory.
            xcopy /E c:\LAB_USERS\...  C:\cygwin64
    (making sure cygwin64 is empty or it'll have issues even if switched to overwrite)

I ran into a NUMBER of issues with Microsoft hanging up, or not copying reliably, permissions were wrong...
    despite setting switches properly, but generally xcopy worked...
Also had to update the desktop link to cygwin
    - adding this to the "start in" directory:  c:\cygwin64\home\%USERNAME%

By copying the entire cygwin64 directory I was hoping to not have to do an install on the workstation.
I tried tar files, zip files, but the only thing that seemed to work "reliably" to replicate the
    install throughout the lab was to xcopy into an empty directory.

HOWEVER, in trying to make ssh work I have since learned that using Microsoft's xcopy across the network
    did NOT preserve permissions and ssh has failed. There are complexities with cygwin on Microsoft.

4) using cygwin to clean some files and directories globally using find, xargs, du and integrating Microsoft commands.

5) had a long term plan, incorporate scripts like this into the lab:
    http://johnmeister.com/linux/Scripts/chksys.sh.html
        http://johnmeister.com/linux/Scripts/run-now-on-all-hosts.sh.html
    (launch the first script via this one... on a cron job...)
  HOWEVER, unless I can find a way of installing cygwin via a template, tar file or other means
        so as not to install on each system... it'll take a long time.

6) also working on id3tools for MP3's  and exiftools for digital photography,
    and using "dvdbackup -i /dev/sr0 -M"   which works splendidly with VLC.
The dvdbackup tool creates a directory that is 4.4G in size and VLC reads it just like the DVD.
A great tool if you don't want to wear down your battery spinning a DVD, or if you don't have an optical drive,
or don't want to travel with your DVD library.  I've used handbrake to convert chapters on the dvd to mkv files.
  future session will discuss loops and scripting using id3 and exif... have been distracted with other scripts.




 
 
 Session #144 - February 8, 2019 - 
 
 
Session #144 Linux at Lunch - 02/08/2019

THIS SESSION:

1) basics of disk and removable media and file systems:
        - fdisk / parted (partitions)
            http://johnmeister.com/linux/FileSystems/newdisk-setup.html
            http://johnmeister.com/linux/FileSystems/usingParted8TB.html
        - df (disk free)
                 alias: alias dfa='df -h | grep sda'
                 alias dfh='df -h | grep -v fs | grep -v boot | grep -v udev'
        - du (disk usage)    du -sh *
        - mount/umount ( /etc/fstab, mounting and unmounting filesystems)
               http://johnmeister.com/linux/FileSystems/mount-external-drive-rysnc.html
               http://johnmeister.com/linux/FileSystems/NFS-info.html
        - gvfs (gnome virtual file system)
                http://johnmeister.com/linux/SysAdmin/Undo-Systemd-Resolved-a-script-and-results.html
        - using /proc
                http://johnmeister.com/linux/FileSystems/RAID-info.html
                --> cat /proc/diskstats
                    ...
                    8       0 sda 201275 864 56505586 678268 14613 5748 1758344 1076088 0 736088 1754128
                    8       1 sda1 201249 864 56503490 678248 14612 5748 1758344 1076088 0 736068 1754104
                --> cat /proc/devices
                Character devices:
                  1 mem
                  3 cons
                  5 /dev/tty
                  5 /dev/console
                  5 /dev/ptmx
                  9 st
                13 misc
                14 sound
                117 ttyS
                136 tty
                Block devices:
                  2 fd
                  8 sd
                11 sr
                65 sd
                66 sd
                67 sd
                68 sd
                69 sd
                70 sd
                71 sd
        - using /dmesg and /var/messages (YMMV)
                --> dmesg | grep usb | wc -l
                    67
        - using lsusb
            --> lsusb
                    Bus 001 Device 001: ID 8086:8d26 Intel Corp. \ Bus 002 Device 001: ID 8086:8d2d Intel Corp.
                    Bus 003 Device 005: ID 0424:2744 Standard Microsystems Corp.
                    Bus 003 Device 006: ID 0424:2744 Standard Microsystems Corp.
                    Bus 003 Device 003: ID 045e:0039 Microsoft Corp. IntelliMouse Optical
                    Bus 003 Device 004: ID 045e:00db Microsoft Corp. Natural Ergonomic Keyboard 4000 V1.0
                    Bus 003 Device 002: ID 08e6:3437 Gemalto (was Gemplus) GemPC Twin SmartCard Reader
                    Bus 001 Device 002: ID 8087:8002 Intel Corp. \  Bus 002 Device 002: ID 8087:800a Intel Corp.
                    Bus 003 Device 001: ID 8086:8d31 Intel Corp.

2) read removable media
        - using SD and microSD card readers in Linux, Mac, VirtualBox and cygwin
        - using USB devices in Linux, Mac, VirtualBox and cygwin

3)   ran into a problem with one of my SD card scripts over the weekend - had to rewrite it. - trace through scripts
    http://johnmeister.com/linux/Scripts/get-images-from-SD-card.sh.html

4) quick mention of lsusb and dmesg - helped a friend back east with an industrial equipment problem - USB to RS232
    searched on:  linux u232-p9 driver download
    https://ubuntuforums.org/showthread.php?t=1716756
    boot up; plug in device ;  lsusb  ; dmesg | tail

NOTE:  from the discussion wanted to share a couple of Mike's inputs, however he can't recommend the installation
    completely...  https://yourunclemike.github.io/lunch-n-learn_2016-09/index.html  there are some issues... HOWEVER,
    the issues seem to be related to the distribution and version, and I'll leave this here because the steps are still helpful.
    He also suggested a website to try out various distros, as he points out it's less trouble than setting up a VM, or
    a thumbdrive for a trial run:  https://distrotest.net/    to see all the distros out there:   http://distrowatch.com
    Also... there's this:  https://manjaro.org/download/

If you're going to setup a system for personal use my personal recommendation stands with Linux Mint with the MATE desktop.
If you're possibly working in IT and will support an Oracle or Apache system I'd recommend SuSE Linux.
If you're stuck with supporting RedHat, I suggest Centos for your personal system, in an Oracle VirtualBox VM... hosted on Mint or SuSE. ;)




 
 
 Session #145 - February 22, 2019 - 
 
 
Session #145 Linux at Lunch - 02/22/2019

THIS SESSION:
1) cal - calendar - using it with julian and mdays for calculating time and scheduling
    cal 02 2019
    cal 2019
    cal -j 2019
2) find - using find to locate particular types of files
     http://johnmeister.com/linux/Commands/find-examples.html
     http://johnmeister.com/linux/Commands/find-copy.html 
3) xargs - using xargs along with find to manage files found with find.
     http://johnmeister.com/linux/Commands/xargs-some-examples.html
4) du - using disk usage to find directories with large files
    --> find /home/luser/ -type d -exec du -sh {} \;  | grep -v ^0 | grep -v ^[0-9].0K
5) id3tool - tool to update metadata in MP3 and related files: titles, albumns, year, author, etc.
         http://johnmeister.com/linux/Commands/id3tool-command-info.html




 
 
 Session #146 - March 15, 2019 - 
 
 
Session #146 Linux at Lunch - 03/15/2019

THIS SESSION:
Let's use BASH Windows 10!
Will go through the book on Using BASH on Windows 10.

 https://www.smashwords.com/books/view/703463

Ubuntu BASH on Windows 10 performs much better than Cygwin.

Comparing BASH on various systems, almost everything works the same, as I note in this table:

http://johnmeister.com/linux/FileSystems/Linux-bashrc-vim-differences.jpg




 
 
 Session #147 - March 22, 2019 - 
 
 
Session #147 Linux at Lunch - 03/22/2019

THIS SESSION:
After I wrote the Win10/BASH book we talked about last week, I continued on to finish the Linux book that was the starting point.
 http://johnmeister.com/linux/Overview/ 
So, this week we will go through that book, "Simply Linux: BASICS" 

https://www.smashwords.com/books/view/705084
• Published: Feb. 19, 2017
• Words: 48,820
• Language: English
• ISBN: 9781370361489 

 just as a note... the smashword royalties this month for all five books came out to $7.43...   :)




 
 
 Session #148 - March 29, 2019 - 
 
 
Session #148 Linux at Lunch - 03/29/2019

THIS SESSION:
 
 update on id3tool - http://johnmeister.com/linux/Notes/id3tool-info.html
 migrating from LibreOffice-->OpenOffice at the command line
editing colors in HTML using vi - http://johnmeister.com/linux/colors.html 

id3tool process:
1) copy original mp3's into new directories, create  ##_AAA, cd ##_AAA; ls >> idAAA ; vi idAAA

 cp -r ../ORIGINAL-MP3s/58_Hebrews/ 58_HEB ;
    cd 58_HEB ;
    ls >> id3heb

2) edit to  "mv" from old long name to new short name

    vi id3heb
    :%s/.*/mv & 58_00_Heb_&/g
   #    edit by removing the long previous name and add number for each track - using searches and replaces a tedious manual process...

3) execute mv ; ls >> idAAA ; edit id3tool info saving old info behind #

     sh ./id3heb    #  (this renames all the files to the new, shorter file name)
     ls >> id3heb   #  appending to the file
     vi id3heb
    :24,$s/.*/#  &/g          # first select all the original "mv" lines to the bottom, then comment them out

    :1,23s/.*/id3tool -t "&" -c Z -y 2018 -G "Speech" -a "Hebrews - Letters" -n "bridgechristianfellowshipcom" -r "Rick Crawford" &/g

4) execute id3tool info

     sh ./id3heb    #  (this updates all the id3tag information in files)
     id3tool *.mp3    #  check to see if it updated and didn't truncate, if it did, edit "id3heb" and execute again

5) mv id3tool script idAAA to  ../../ORIGINAL-BCF-MP3s/NOTES/

    mv id3heb ../../ORIGINAL-MP3s/NOTES/
        #  save a copy of each script for the 66 directories to help identify proper time and sequence

6) scp -r NEW_DIR/ server:/web/bible/mp3 ; scp NEW_DIR/*.mp3 server:/web/BIBLE/MP3-ALL/    #  one method was to ssh and cp -r

7) repeat for all 66 books:  finished work:    http://johnmeister.com/BIBLE/BCF-MP3-ALL/ 

--> id3tool 58_11_Heb_9-10v18.mp3
Filename: 58_11_Heb_9-10v18.mp3
Song Title:     58_11_Heb_9-10v18
Artist:         Rick Crawford
Album:          Hebrews - Letters
Note:           bridgechristianfellowshipcom
Track:          11
Year:           2018
Genre:          Speech (0x65)


 migrating from LibreOffice to Apache's OpenOffice.org suite
NOTE:  these steps are for any debian based Linux distro: Ubuntu, Mint or Debian, but minor tweaks and it'll work for all distros

BEFORE:     --> ls -al /usr/bin/soffice
            # lrwxrwxrwx 1 root root 34 Nov 27 02:13 /usr/bin/soffice -> ../lib/libreoffice/program/soffice

AFTER:     --> ls -al /usr/bin/soffice
           # lrwxrwxrwx 1 root root 32 Oct 23 04:31 /usr/bin/soffice -> /opt/openoffice4/program/soffice


sudo apt-get update
sudo apt-get -y remove --purge libreoffice* libexttextcat-data* && sudo apt-get -y autoremove
cp ~/bin/Apache_OpenOffice_4.1.6_Linux_x86-64_install-deb_en-US.tar.gz /tmp
    # NOTE:  instructions called out an older version; found the current version; it was simple to download and copy to /tmp
    # file downloaded and copied to /tmp:    Apache_OpenOffice_4.1.6_Linux_x86-64_install-deb_en-US.tar.gz
using wget:
    # cd /tmp && wget http://downloads.sourceforge.net/project/openofficeorg.mirror/ \
            4.1.6/binaries/en-US/Apache_OpenOffice_4.1.6_Linux_x86-64_install-deb_en-US.tar.gz
cd /tmp
tar -xvf Apache_OpenOffice*.tar.gz     # wildcards save typing...
sudo dpkg -i en-US/DEBS/*.deb
    # using the debian packaging tool, if using SuSE, use zypper, Rh/Centos, yum, etc.  Process is the same.
sudo dpkg -i en-US/DEBS/desktop-integration/*.deb   #  this put the icons in the menu and made it all happy.



editing colors in HTML using vi
  http://johnmeister.com/linux/colors.html
    for info on RGB/Hexadecimal values:  https://www.rapidtables.com/web/color/RGB_Color.html
   Use sRGB rather than Adobe's proprietary colors for most applications.   (standard Red Green Blue color space)
    sRGB is recommended by most photographers, especially for the web and monitors.
   see:  https://photographylife.com/srgb-vs-adobe-rgb-vs-prophoto-rgb  and https://kenrockwell.com/tech/adobe-rgb.htm

    RGB = Red Green Blue - the three primary colors.
    The red, green and blue use 8 bits each, which have integer values from 0 to 255.   256*256*256=16,777,216 colors.

 wiki on shades of white 

the color   Platinum     is presented by this value: #E5E4E2     R=229    G=228    B=226    (values range from 0 through 255)

    the color Red is FF0000
    the color Green is 00FF00 (actually comes out lime green in html!)
    the color Blue is 0000FF

 this is the html code that displays the table below 

<table border="5" bgcolor="FFFFFF">

<tr> <td bgcolor="#FFCCCC"> Law </td> <td bgcolor="#FFCCCC"> #FFE5CC </td>
    <td bgcolor="#FFCCCC"> 255 </td> <td bgcolor="#FFCCCC"> 204 </td> <td bgcolor="#FFCCCC"> 204 </td></tr>

<tr> <td bgcolor="#FFFFCC"> History </td> <td bgcolor="#FFFFCC"> #FFFFCC </td>
    <td bgcolor="#FFFFCC"> 255 </td> <td bgcolor="#FFFFCC"> 255 </td> <td bgcolor="#FFFFCC"> 204 </td></tr>

<tr> <td bgcolor="#E5FFCC"> Wisdom </td> <td bgcolor="#E5FFCC"> #E5FFCC </td>
    <td bgcolor="#E5FFCC"> 229 </td> <td bgcolor="#E5FFCC"> 255 </td> <td bgcolor="#E5FFCC"> 204 </td></tr>

<tr> <td bgcolor="#E9FFDB"> Prophets </td> <td bgcolor="#E9FFDB"> #E9FFDB </td>
    <td bgcolor="#E9FFDB"> 233 </td> <td bgcolor="#E9FFDB"> 255 </td> <td bgcolor="#E9FFDB"> 219 </td></tr>

<tr> <td bgcolor="#CCFFE5"> Gospels </td> <td bgcolor="#CCFFE5"> #CCFFE5 </td>
    <td bgcolor="#CCFFE5"> 204 </td> <td bgcolor="#CCFFE5"> 255 </td> <td bgcolor="#CCFFE5"> 255 </td></tr>

<tr> <td bgcolor="#CCFFFF"> Paul </td> <td bgcolor="#CCFFFF"> #CCFFFF </td>
    <td bgcolor="#CCFFFF"> 204 </td> <td bgcolor="#CCFFFF"> 255 </td> <td bgcolor="#CCFFFF"> 255 </td></tr>

<tr> <td bgcolor="#CCE5FF"> Letters </td> <td bgcolor="#CCE5FF"> #CCE5FF </td>
    <td bgcolor="#CCE5FF"> 204 </td> <td bgcolor="#CCE5FF"> 229 </td> <td bgcolor="#CCE5FF"> 255 </td></tr>

<tr> <td bgcolor="#EFCCFF"> Revelation </td> <td bgcolor="#EFCCFF"> #EFCCFF </td>
    <td bgcolor="#EFCCFF"> 229 </td> <td bgcolor="#EFCCFF"> 204 </td> <td bgcolor="#EFCCFF"> 255 </td></tr>

</table>



    
  Law   
   
  #FFE5CC 
   
  255 
   
  204 
   
  204 
    
  History   
   
  #FFFFCC 
   
  255 
   
  255 
   
  204 
    
  Wisdom 
   
  #E5FFCC 
   
  229 
   
  255 
   
  204 
    
  Prophets   
   
  #E9FFDB 
   
  233 
   
  255 
   
  219 
    
  Gospels 
   
  #CCFFE5 
   
  204 
   
  255 
   
  255 
    
  Paul   
   
  #CCFFFF 
   
  204 
   
  255 
   
  255 
    
  Letters   
   
  #CCE5FF 
   
  204 
   
  229 
   
  255 
    
  Revelation   
   
  #EFCCFF 
   
  229 
   
  204 
   
  255 

 seeking colors that are easy to read, have a logical flow and are pleasing to the eye... current colors were picked in excel:
 




 
 
 Session #149 - April 12, 2019 - 
 
 
Session #149 Linux at Lunch - 04/12/2019

THIS SESSION:
 Look through content on http://LinuxMeister.net
  working on a new "booklet": "Using Linux:BASICS" - basically a non-computer user guide to using Linux
    - install Linux (dual boot Linux Mint with Mate desktop on Windows 10)
    - show the MATE desktop settings, menus and customization
    - show the basic tools like Thunderbird, Firefox, LibreOffice or OpenOffice, Stellarium, DarkTable, and other free applications
    - will likely make the book "free" or set the price, which can be "free".
    - plan to use "Simply Linux:BASICS" as the starting point, removing a lot of content to keep it focused on use.

 
 
 Session #150 - May 17, 2019 - 
 
 
Session #150 Linux at Lunch - RANDOMIZATION - 05/17/2019

THIS SESSION:
Randomization tools: creating random numbers for adding a prefix to files and updating id3tags using BASH and Perl.

PROBLEM: MP3 files were playing in sequence on a couple different players that had no random play feature...
 using $RANDOM at the command line worked, but not in a shell script... leading me to a Perl solution:
http://johnmeister.com/linux/Scripts/rename-files-with-random-number-suffix.sh.txt 
http://johnmeister.com/linux/Scripts/id3tool-change-title.html



 
 
 Session #151 - May 31, 2019 - 
 
 
Session #150 Linux at Lunch - the joys of vim - 05/31/2019
THIS SESSION:
"vim info"  - presented by one  of our regulars.

One of the resources listed is a "free" vim scripting book, I've included the HTML version on my website,
it is NOT a how-to book, but has a lot of useful information in it:
 learn vim script the hardway . by steve losh
other vim info at:
  vim info