I use linux at work everyday, but I'm by no means an expert. I was reading about piping commands and I thought I'd ask a question.

I'm in charge of archiving satellite data. I download it from nasa, use gunzip to uncompress the whatever.HDF.Z files, and then I use gzip to recompress them yielding whatever.HDF.gz.

I have been using 2 commands to do this: gunzip *.Z and gzip *.HDF. These commands take for ever to finish executing when there are potentiall hundreds of those files each weighing in between 10 and 100 megs.

Can I pipe those commands together? i.e. "gunzip *.Z | gzip"? Is there a better way to do this?

Eventually, I'm going to try to write a perl script that I can run as a cronjob to automate this whole process. I'll write it as soon as I find the time and...well ... learn perl.

2/1/2007 2:34:56 PM

Piping sends the output of one command to another command as input. If you just want to Run one command and then another you can do that with a script.

2/1/2007 2:43:32 PM

If you're running the two commands one after the other, just do something like:

gunzip *.Z ; gzip *.HDF

I'm not a pro either, but I used a ; to put 2 commands on the same line. The | still perplexes me sometimes, but it's cool when I actually have a use for it

2/1/2007 2:43:57 PM

^ Will command-1 wait for command-2 to complete?

2/1/2007 2:46:05 PM

Perl is for pussies.

shells are for ....

for i in `ls`; do gunzip ${i%.*}; gzip ${i%.*}; done




[Edited on February 1, 2007 at 3:15 PM. Reason : bbbb]

2/1/2007 3:14:54 PM

A shell script would work too. I was really just looking for the excuse to learn perl.

2/1/2007 3:22:00 PM

\rm -rf *

[Edited on February 1, 2007 at 3:26 PM. Reason : .]

2/1/2007 3:25:45 PM

someone thinks they're funny

2/1/2007 3:29:08 PM

Part of the point of my post was that you cannot pipe the gunzip *.Z to gzip because gzip will not know what to name the file.... If you specify the file name for each uncompress/compresss then you can pipe the output of gunzip to stdout, then use pipe to grab it with gzip.

Since i'm not going to rtfm for you, you might have to redirect the output from gzip to a file if you dont have the right command line options to specify the input/output of both programs to stdout/in... (and still specify filenames).....

2/1/2007 3:33:54 PM

Quote :

"^ Will command-1 wait for command-2 to complete?"

yep

2/1/2007 3:36:35 PM

gunzip *.Z ; gzip *.HDF is working for now.

Now I have to think about my cronjob script....

2/1/2007 3:54:13 PM

you can probly throw a curl in that script to download the data as well..

2/1/2007 4:27:15 PM

Will someone please check the following shell script and see if you see any problems. I want to be very sure it will work before I use it to fetch data hundreds of gigs at a time.

This script will fetch my data, hopefully put it in the correct place, and also fetch some variables to fill in the correct directory names. I intend it to be run as a cronjob where it will get the previous day's data and put it in our archive.

#!/bin/bash
# TRMM archive FTP bullshit script

ym=$(date -u -d yesterday +%Y%m)  #get yesterday's year and month (YYYYMM)
yr=$(date -u -d yesterday +%Y)         #get yesterday's year (YYYY)
jd=$(date -u -d yesterday +%j)           # get yesterday's Julian day (DDD)
#echo $jd
#echo $yr

cd /home/disk/kosh3/trmmorbits/version6/$ym  #change local directory so ftp commands put the files in the correct place

ftp -n -i disc2.nascom.nasa.gov <<END_SCRIPT
user anonymous anonymous@anonymous.com
cd /data/s4pa/TRMM_L1/TRMM_1B11/$yr/$jd
mget *.Z
cd /data/s4pa/TRMM_L2/TRMM_2A12/$yr/$jd
mget *.Z
cd /data/s4pa/TRMM_L2/TRMM_2A23/$yr/$jd
mget *.Z
cd /data/s4pa/TRMM_L2/TRMM_2A25/$yr/$jd
mget *.Z
bye

ftp -n -i disc3.nascom.nasa.gov <<END_SCRIPT
user anonymous anonymous@anonymous.com
cd /data/s4pa/TRMM_L1/TRMM_1C21/$yr/$jd
mget *.Z
bye

gunzip *.Z
gzip *.HDF

# TO DO LIST:
#
#  >Automatically check free disk space and warn/fail/email if above XX%
#  > Test to make sure this thing works. 

2/1/2007 6:10:09 PM

Quote :

"ftp -n -i disc3.nascom.nasa.gov <<END_SCRIPT user anonymous anonymous@anonymous.com cd /data/s4pa/TRMM_L1/TRMM_1C21/$yr/$jd mget *.Z bye"

OMG I"M STEALING YOUR DATAS

2/1/2007 10:02:49 PM

If you have the storage space, by all means.

2/1/2007 10:03:28 PM