Wednesday, July 31, 2019

PERL: Utility to sort and remove duplicates from a csv file.

Here is a little PERL Script that will take a file and sort the data lines and remove duplicates.


#!/usr/bin/perl

$ifile=$ARGV[0];
$ofile=$ARGV[1];

$header=`sed -n '1p' $ifile` ;
$data=`sed '1d' $ifile | sort -u` ;
open(my $fh, '>', $ofile) or die "Could not open file '$ofile' $!";
print $fh $header;
print $fh $data;

close $fh;
exit 0