Bash: Script to archive files into YYYY/MM based on creation date

 I had over 10 years of photos that I wanted to organize by YYYY/MM.  Created this Bash script to do it.

#!/bin/sh
#script:       archiver.sh
#
#

sourceDir="/f/from_kims_ext/"
#sourceDir="/f/test/"
archiveDir="/f/image_archive/"
ARCHIVEFILE="/f/move_archive.txt"

function getfilelist {
`ls -Rp $sourceDir | awk '
/:$/&&f{s=$0;f=0}
/:$/&&!f{sub(/:$/,"");s=$0;f=1;next}
NF&&f{ print s"/"$0 }' | sed "/\/$/d" > $ARCHIVEFILE`
}

function parsefilename {
  sourcefilename=`basename $1`
  extension="${sourcefilename##*.}"
  filename="${sourcefilename%.*}"
}

function updatefilename {
  typeset counter
  counter=0
  
  #check if ok to use original filename
  updatecheck="$archiveDir$datepath/$filename.$extension"
  echo "checking if target $updatecheck exists"
 # if [[ -f $updatecheck ]]; then
   if [ ! -f ${updatecheck} ]; then
    echo "$updatecheck does not exist"
    return
  else
    echo "$updatecheck exists"
  fi

  while [[ $counter -lt 200 ]]
  do
    counter=$((counter+1))
    updatecheckinc="$archiveDir$datepath/$filename"
    updatecheckinc+="_"
    updatecheckinc+="$counter.$extension"
    echo "checking for new $updatecheckinc"
    if [[ ! -f "${updatecheckinc}" ]]; then
      break
    fi
  done
 # changing name because it already exists, so increment to new file name
  filename+="_"
  filename+="$counter"
  echo "updating filename to $filename"
  return
}

set -u
#LOG_TIME=`date |cut -c12-13,15-16,21-23| sed s/" "//g`
LOG_TIME=`date`
 thisyear=`date +"%Y"`
 thismonth=`date +"%m"`
 
echo "archive starting..."
echo "Starting archive at $LOG_TIME"

getfilelist

# Setting delimiter to an empty string since we want to read the entire line of the file
# and be able to handle files with spaces since the default is a space delimiter
SAVEIFS=$IFS
IFS=$''

((archived_moved=0))
while read file
do
  parsefilename $file
  echo "processing $filename with extension $extension"
  fDate=`stat -c %y "${file}"`
  fDate=$(echo "$fDate" | cut -d' ' -f1)
  #example  2018-01-12
    year=$(echo ${fDate} | cut -d'-' -f1 )
    month=$(echo ${fDate} | cut -d'-' -f2)
    day=$(echo ${fDate} | cut -d'-' -f3)
    filedate=${year}${month}
   if [ $(echo "$filedate" | grep -c '^[0-9]\{6\}$') -eq 1 ]
    then
        if [ ! -d ${archiveDir}${year}/${month} ]
        then
          echo "echo making directory $archiveDir$year/$month"
          mkdir -p ${archiveDir}${year}/${month}
        fi
        datepath="${year}/${month}"
        updatefilename
        echo "Archiving ${file} to ${archiveDir}${datepath}/$filename.$extension"
        mv "${file}" "${archiveDir}${datepath}/${filename}.${extension}"
        ((archived_moved=archived_moved+1))
        
    else
        echo "Couldn't get stat on ${reportBakDir}${file}"
    fi
done <"$ARCHIVEFILE"
IFS=$SAVEIFS
#END_TIME=`date |cut -c12-13,15-16,21-23| /usr/bin/sed s/" "//g`
END_TIME=`date`
echo "Completed archive at $END_TIME"
exit 0

Python 3: RegEx Back References

Regual Expressions is one of those things that isn't completely regular when switching between languages. This is how you use a back reference in Python.

 

import re

print( re.sub(r'(foo)(bar)',r'\g<2>\g<1>','foobar'))

More info here https://www.regular-expressions.info/replacebackref.html

Python 3: solving 'utf-8' codec can't decode byte 0x92

In Python 3, UTF-8 is the default encoding. In windows you may run into issues where characters will not decode such as 'utf-8' codec can't decode byte 0x92. Use a header with either cp1252 or windows-1252

 

# coding=CP1252
import re

text = """
LADY CAPULET
    Alack the day. She’s dead, she’s dead, she’s dead!
CAPULET
    Ha! Let me see her. Out, alas! She’s cold.
    Her blood is settled, and her joints are stiff.
    Life and these lips have long been separated.
    Death lies on her like an untimely frost
    Upon the sweetest flower of all the field.
Nurse
    O lamentable day!
"""

Autosys: Changing output of autosys -j JOB_NAME into Markdown

When generating output using autosys -j command line, it produces an output using space delimited typical console output.  The following command will output the result into a markdown table.

autorep -j YOUR_JOB_OR_BOX_NAME | sed -e 's/ \{2,\}/|/g'| sed -e 's/_\{2,\}/ /g' | sed 's/\(.*\)/|\1/g' | sed -e 's/\(Job Name\)/|\1/g' | sed -e 's/\(Last Start\)/|\1/g' | sed -e 's/\(Last End\)/|\1|/g'| sed -e 's/ST Run/ST||Run/g' | sed -e 's/Ntry Pri\/Xit/Ntry||Pri\/Xit||/g'| sed -e 's/\(.*\)\([0-9]\{8\}\/[0-9]*\)/\1|\2|/g'

Python: Performing XSLT recursively on a directory of XML files

I've been using https://github.com/ajryan/RptToXml to extract out xml information from Crystal Reports. I then rip through the created xml files with xlst to provide detail information on how the reports are created.  This little python 3.5+ script uses https://www.saxonica.com/download/download_page.xml and will create html files from a directory tree containing xml files.  

TODO: Write an xxl file to convert Crystal Reports to Jasper Reports.

import os
import subprocess
import glob
from pathlib import Path
root_dir = r"C:\crystalxml"
xslFile = r"C:\Saxon\pretty.xsl"

for filename in glob.iglob(root_dir + '/**/*.xml', recursive=True):

 my_parts = filename.split("\\")
 justname = my_parts[-1].split('.')[0]
 my_dir = os.path.dirname(filename)
 newname = fr'{my_dir}\{justname}.html'
 cmd = fr'java -cp C:\Saxon\saxon-he-10.1.jar net.sf.saxon.Transform -t -s:"{filename}" -xsl:{xslFile} -o:"{newname}"'
 subprocess.call(cmd)

Python: Learning by starting with the basics.

Just a simple mind exercise in learning a language.  Created a console Hangman game that uses a word api with a hint command that fills in the first available blank.

hangman.py

Guess a Letter:hint
------
|     |
|     0
|    /|\
|    /
t y p h u _ e _ 
Wrong Guesses:lmaio
Guess a Letter:s
------
|     |
|     0
|    /|\
|    /
t y p h u s e s 
Wrong Guesses:lmaio
stay of execution

Oracle: Capitalizing and removing underscores from a column name

Oracle table names are all caps with underscores.  So if you're consuming them from something like all_tab_columns, this quick little function makes them a little prettier. 

select initcap(replace(lower('SOME_COLUMN_NAME'),'_',' ')) from dual;

VBA: A little function to Capitalize and Split a String

Here's a subroutine for converting a range of strings into a space separated Capitalization (i.e.) "FOO_BAR" becomes "Foo Bar"

Sub Capitalization_Split()
'Declaring variables
Dim Cell As Range
Dim Source As Range
Dim Capitalized As String
Dim Humps As Variant
Dim Hump As String
Dim i As Long
'Initializing source range
Set Source = Application.Selection
'Looping through each cell in the source range
For Each Cell In Source
    Capitalized = ""
    Hump = Cell.Value
    Humps = Split(Hump, "_")
    For i = LBound(Humps, 1) To UBound(Humps, 1)
        Hump = UCase(Left(Humps(i), 1)) & LCase(Right(Humps(i), Len(Humps(i)) - 1))
        Capitalized = Capitalized & Hump & " "
    Next
    Cell.Value = Capitalized 

Next
End Sub

GoLang: Issues Installing Delv Analysis Tool

I've recently just started setting up my development environment to play around with GO.  My editor of choice is Eclipse as it 1) suits my development efforts, 2) It's pretty much free,  though I did opt to pay for CodeMix which has a very reasonable price of $29 per year as opposed to the competition.

I mostly use windows and the initial setup went fairly well except I was getting an error that all the analysis tools were not there.  I was able to "Go Get" some of the ones that were missing, but dlv just wouldn't install in that manner.  So I ended up checking out the https://github.com/go-delve/ repo, performing a go build and then manually copying over repo to my $GOPATH/pkg/mod/github.com/ folder, put the compiled dlv.exe into the $GOPATH/bin folder and then launched Eclipse.   No more warnings of analytic tools being missing.

No off to do something other than the obligatory "Hello World".

Why Lucee is Better Than ColdFusion #1

Lucee's Query of Queries is better than Adobe.  Try this little snippet on Lucee 5 and then CF2018 or earlier

<cfscript>
       myQuery=QueryNew("Country,Capital City,Amount##","varchar,varchar,numeric");
       /*Add array values in column*/
       myAreaValues=ArrayNew(1);
       myAreaValues[1]=100000;
       myAreaValues[2]=50000;
       myQuery.addColumn("Area","integer",myAreaValues);
       WriteDump(myQuery);
</cfscript>

XSLT with carriage returns and line breaks

I needed to translate carriage returns and line feeds to <BR/> elements in an XSLT process.  Sometimes you get hung up how you normally use functions and then hit a language wall that you think should work,

In this case, the solution was to iterate over the selection and output the elements with an appended tag.

<tr><td><xsl:for-each select="tokenize(customerinfo/@comments, '&#xD;&#xA;')">
                <xsl:value-of select="." /><br/>
</xsl:for-each></td></tr>