A Quickstart Tutorial

Introduction
Filename extensions and starting SPIDER
WEB
Creating another test image in SPIDER
Filtering
Geometric and Algebraic operations
Correlation
Registers
Peak Search
Shift Alignment
Volumes
Document Files
Running procedure files
Debugging
Inline files
Control Structures: DO Loops
The IF Statement
Run-time Designation of File Names
Reading Document files
Procedure files

Introduction

Below is a brief introduction to SPIDER and WEB. The SPIDER Tutorial is a more detailed tour of many SPIDER operations. The SPIDER Manual of operations provides information on the specific options and choices of parameters for each SPIDER operation. The User's guide covers issues relating to the syntax and conventions of SPIDER, including a longer section on writing and using procedures.

Filename extensions and starting SPIDER

SPIDER expects all files within a session to have the same filename extension.

For this lab, we will use the extension "dat". In a Unix command window, start SPIDER by typing at the Unix prompt:

    spider dat

    .OPERATION:

    .INPUT FILE:

Create a simple image using the MO operation:

 .OPERATION: MO

 .OUTPUT FILE: circ001

 .ENTER DIMENSIONS (NX,NY): 75,75
 
 .B/C/G/R/S/T/W:  C
 
 .RADIUS (FLOATING POINT): 16

 .OPERATION: 

WEB

Web is often run simultaneously with a SPIDER session, allowing the user to view the output files. We have two different Webs available X-window Web and Java based JWeb. We will usually use JWeb in this workshop but if you prefer you can use Web which is described below. In another Unix window, type 'web dat' or 'jweb'

  web dat 

Options listed under the main menu headings will be written as: Menu/Command

For example, when Web is first started, you may have to clear the screen with the option Commands/Clear

To view an image, use Commands/Image, and select the file to be viewed (circ001.dat). Web displays the image, and prints out some information (image size and range of the gray levels) in the status window.

In Web The size of the displayed image may be adjusted: select Options/Image, and enter -2 in the SIZE REDUCTION entry box. Positive numbers will decrease the size by the specified factor, while negative numbers will enlarge the displayed image. Use Commands/Image again to view circ001.dat. It should appear larger this time. All subsequently displayed images will also be enlarged, until SIZE REDUCTION is changed again. Remember, Web only displays a picture - it does not actually enlarge the original data file.

In JWeb The size of all subsequently displayed images may be adjusted: select Options/Settings, and use the 'Resize' slider.

As you try out the SPIDER operations below, you will go back and forth between SPIDER and Web/JWeb, performing image processing in SPIDER, and using Web to view the results.

Creating another test image in SPIDER

The 'PT' operation allows you to create geometric shapes. Here is another test image you can make using PT:

 .OPERATION: PT

 .OUTPUT FILE: test001

 .ENTER DIMENSIONS (NX,NY): 75, 75

 .ENTER CODE (P,L,C,T,R,S): r

 .ENTER COORDINATES OF UPPER LEFT POINT: 18, 12

 .ENTER  X & Y OFFSETS: 38, 16

 .CONTINUE? (Y/N): Y 

 .ENTER CODE (P,L,C,T,R,S): T

 .ENTER COORDINATES OF FIRST  POINT: 18, 33

 .ENTER COORDINATES OF SECOND POINT: 56, 36

 .ENTER COORDINATES OF THIRD  POINT: 32, 66

 .CONTINUE? (Y/N): N

Importing an image into SPIDER

The 'CP FROM RAW' operation allows you to import various non-SPIDER images:

 .OPERATION: CP FROM RAW

 .EXISTING RAW FILE: img/esp_raw.dat

 .ENTER BITS / PIXEL IN INPUT IMAGE (8, 16 OR 32): 8 
          
 .ENTER COLUMNS, ROWS, & SLICES: 800, 526
       
 .ENTER HEADER BYTES TO BE SKIPPED: 0
         
 .NEW SPIDER IMAGE FILE: esp000

 .OPERATION:   MR

 .INPUT FILE:  esp000

 .OUTPUT FILE: esp001

 .MIRROR AT (Y) OR (X) AXIS (DEF=Y): x

Filtering

SPIDER has a variety if filtering operations. The FQ NP operation provides a set of filters to choose from. Filter your test image with 'FQ NP'. NOTE: text after a semicolon (;) is a SPIDER comment. You DO NOT have to type the comments upon input!

 .OPERATION:   FQ NP

 .INPUT FILE:  esp001

 .OUTPUT FILE: filt001

 .FILTER TYPE (1-8): 3      ; Selects Gaussian low-pass filter

 .FILTER RADIUS: 0.1 

If your input data is noisy there are various filters including a median filter 'CE MED' operation that may help. Filter your test image with 'CE MED'.

 .OPERATION: CE MED

 .INPUT FILE:    img/espnoise

 .OUTPUT FILE:   filt002

 .FILTER LENGTH: 5       ; Selects filter dimension

 .BOX OR CROSS(B/C): C 

Geometric and Algebraic operations

These include rotating (RT), shifting (SH), or resizing (IP) the image. Apply a shift operation to an image:

 .OPERATION: SH

 .INPUT FILE:   esp001

 .OUTPUT FILE: shift001

 .SHIFT COMPONENTS IN X-Y: 16.7, -7.2            

Images may be added (AD), subtracted (SU), or multipiled (MU) together. Subtract the image you just shifted from the original test image:

 .OPERATION: SU

 .INPUT FILE:      esp001

 .SUBTRACTED FILE: shift001

 .OUTPUT FILE:     sub001

 .SUBTRACTED FILE: *

Correlation

Cross correlation (CC) is a way to measure the similarity of 2 images. The 2 images are multiplied together, pixel by pixel, and the sum of these multiplications is stored at a location representing the position of image1 relative to image2. Then the images are shifted by one pixel, and multiplied again, with the resultant value stored at the next pixel. This is carried out by shifting image1 across all positions of image2 (the reference image). Pixels with high values ('peaks') in the cross correlation result show where the best matches were, while the amplitude of the peaks indicates the degree of similarity.

 .OPERATION: CC

 .INPUT FILE:     shift001

 .REFERENCE FILE: esp001

 .OUTPUT FILE:    cc001

Registers

SPIDER provides register variables to hold floating point values. Register variables can be named by the user and the variables are denoted with enclosing square brackets: e.g. [reg-var]. (There is an older preset-name syntax for registers which you may encounter sometimes: x11....x99) . You can use registers anywhere you would normally enter a number in SPIDER. Registers can be used as a form of pocket calculator. At the prompt, type:

 .OPERATION: [loc] = 25

 .OPERATION: [newloc] = [loc] + 3.5

 .OPERATION: [loc]
   25.0000    

 .OPERATION: [newloc]
   28.5000    

1. By being explicitly set equal to a value or a value from an arithmetic expression (as above).
2. As output resulting from an operation.
3. When used as an index in a do-loop.
4. Using the RR operation.
5. When received as a procedure argument.

A peak search is commonly applied after a correlation, to find the location of the best match - a 'peak' in the pixel intensities. The PK operation prints its results to the screen, but it can also store these values in registers on the operation line. In the following example, registers [ix] and [iy] are set to the X and Y coordinates of the highest peak found. Apply a peak search to the cross correlation result to find the coordinates of the bright spot:

 .OPERATION:  PK [ix],[iy]

 .INPUT FILE: cc001

 .ENTER # OF PEAKS, CENTER ORIGIN OVERRIDE (0/1): 1,0

 .OPERATION: [ix]
   17.0000    

 .OPERATION: [iy]
   -7.00000

 .OPERATION: [ixn] = -[ix] 

 .OPERATION: [iyn] = -[iy]

Use the results from the peak search to align the shifted image to the original test image. For the X,Y shift components, use the negative of the numbers obtained from the peak search (i.e., -17 and 7). Since we have these numbers in registers, use registers [ix] and [iy] in place of the actual numbers.

 .OPERATION:   sh

 .INPUT FILE:  shift001

 .OUTPUT FILE: align001

 .SHIFT COMPONENTS IN X-Y: [ixn], [iyn]     ; Instead of -17,7 

 .OPERATION: SU

 .INPUT FILE:      filt001

 .SUBTRACTED FILE: align001

 .OUTPUT FILE:     sub002

 .SUBTRACTED FILE: *

Volumes

A volume may be visualized in Web by viewing it slice-by-slice, or by surface rendering. Create a model volume with SPIDER's MO 3 operation:

 .OPERATION: MO 3

 .OUTPUT FILE: sphere

 .ENTER DIMENSIONS (NX,NY,NZ): 75,75,75

 .B/C/G/H/HA/NUM/R/S/SP/SPA/SPV/T/W: sp    ; Selects sphere from the menu

 .GET COORDINATES FROM DOCUMENT FILE (Y/N): n

 .ENTER MASS VALUE PER VOXEL (DEFAULT = 2.0): 2
   
 .OUTER AND INNER RADII OF SPHERES: 19, 14
   
 .X,Y,Z: 38,38,38                         ; The sphere's center
    
 .X,Y,Z: 0,0,0                            ; Terminates the operation

View the ribosome volume 'img/ribosome.dat' in Web:

Commands/Clear
Commands/Image
       Select img/ribosome.dat, click 'Accept'

You will see slices of the ribosome volume, with a dark intersubunit space in some of the central slices.

Now create a surface rendering of the volume.

Commands/Clear
Commands/Surface
       Select: img/ribosome.dat
       Set Scale to 5.0, click 'Accept'

In this view, the small subunit is on the left, the large subunit on the right.

Click the right mouse button to end the surface session.

Document files

Information obtained during a SPIDER session can be saved to document files (called doc files). Data is written to a doc file with the SD operation, which writes one line at a time. The numbers to be stored are placed on the SD operation line. The first number (here 1), is the key. This doc file can be read by other SPIDER operations; the key tells SPIDER which line to read.

 .OPERATION: SD 1, [ix], [iy], [ixn], [iyn]

 .DOCUMENT FILE: doc001

 .OPERATION: VM

 .SYSTEM COMMAND: cat doc001.$DATEXT

The doc file has two lines:

 ;dat/dat   01-MAY-2007 AT 20:29:05   doc001.dat
    1 4  17.000     -7.0000     -17.000      7.0000    

<key> <number of data items in this line> <data> <data> <data> <data>

End your interactive SPIDER session now:

.OPERATION: EN

Certain operations may have written data in the SPIDER results file: results.dat.0. A different results file will be created with each SPIDER run by incrementing the final digit(s) of the file name.

Running Procedure files

For this part, you will not be running SPIDER in an interactive mode. Instead, you will run SPIDER repeatedly from the Unix prompt. Also, a text editor is required.

SPIDER's interactive mode is good for trying out operations and ideas. But more sophisticated processing can require dozens or even hundreds of operations. The operations can be put into a text file and executed in procedure mode. Open up an editor at the Unix prompt, and type the following (you don't have to type the comments following the ;):

 RT
   test001   ; Image            (input)
   rot001    ; Rotated image    (output)
   60        ; Rotation angle (in degrees)

 IP
   rot001    ; Image            (input)
   big001    ; Interpolated image    (output)
   150, 150  ; New dimensions

 WI
   big001    ; Image            (input)
   win001    ; Windowed image   (output)
   75, 75    ; Size of window
   1, 75     ; Top left window coordinates

 EN D  

Procedure files should have an extension different than the data extension. To run SPIDER in procedure mode, use the following format at the Unix prompt:

    spider spf/dat @bat01  

Run the procedure file you created, and view the files rot001.dat, big001.dat, win001.dat in Web. The original file has been rotated, enlarged, and a subimage cut out, all in one call to SPIDER.

Debugging

In procedure mode, SPIDER still prints out the banner, version number, and the name of the results file:

    Results file: results.spf.0  

    STOP **** SPIDER NORMAL STOP ****

    STOP *** FATAL ERROR ENCOUNTERED IN BATCH MODE

   tail results.spf.0

 .OPERATION:      RT
 .INPUT FILE:     test999
 *** FILE NOT FOUND: test999.dat
 TERMINATED ON ERROR IN BATCH MODE  22-OCT-2002 at 16:42:39

To obtain the final result (win001.dat) in the example above, two intermediate files were created (rot001.dat and big001.dat). If the images are large, and there are many operations, you can clutter your disk drive with numerous unnecessary files. SPIDER allows special variables to be substituted for filenames. These are called inline files, and are denoted by _N, where N is a digit from 1..99, preceded by one to three underscore symbols. Therefore, up to nine inline files are allowed at any time. inline files may be used wherever a filename is used in SPIDER (even in interactive mode).

Copy bat01.spf to bat02.spf, and, in the editor, make the following changes to bat02.spf at the lines marked *****:

 RT
   test001 
   _1        ; ***** Replace the output filename with _1
   60        

 IP
   _1        ; ***** Use the above result
   _2        ; ***** Replace the output filename with _2
   150, 150 

 WI
   _2        ; ***** Use the above result
   win002    ; ***** Give the final result a different number
   75, 75     
   1, 1      ; ***** Window out a different region 

 EN D        ; End and delete results file

This file created a new output (win002.dat) without writing the intermediate rotated and enlarged images. Those files were created in memory, but not written to disk. Not only does this save disk space, but it saves time - multiple disk accesses will slow down a procedure file. If you are processing thousands of image files, inline files can save much time and disk space.

Control Structures: DO Loops

DO loops enable operations to be repeated. The DO loop syntax is:

 DO  <loop register> = <start_value>, <end_value> <optional increment_value>
    < Any SPIDER operations can be inside the DO loop >
 ENDDO

 DO <Label> <loop register variable > = <start_value>, <end_value>, <optional increment_value>
    < Any SPIDER operations can be inside the DO loop >
 <Label>

The start and end values set the number of times the operations inside the DO loop will be executed (in this case twenty times). Start and end values may be registers. The number of iterations, K, equals (end_value - start_value) + 1.

The loop register variable takes on a new value for each iteration. Here, [index] will have the values 1 to 20. (Never change the loop register inside a loop. You will be sorry.)

The DO construct executes all SPIDER operations between the lines DO ... and ENDDO, K times.

Create the following procedure file:

 [unused]= 0                ; Required for RAN operation
 DO [index] = 1,20          ; Set up the DO loop

   [ran] = RAN([unused])   ; These operations...

   SD [index],[ran]        ;    will be executed...
     randoc                ;    20 times.
 ENDDO                      ; End of the DO loop

 EN D

This procedure file uses the arithmetic operator RAN to generate random numbers in the range 0..1 (see Mathematical Functions in the User's Guide). The 'SD' operation writes a single line to a doc file, randoc.dat. Look at the contents of randoc.dat with the Unix cat command:

 cat randoc.dat

Run this procedure file again, and look at randoc.dat. What happened? Whereas SPIDER image and volume files are overwritten if you use the same filename, the 'SD' operation appends to the end of the doc file. This can be a problem: if you run a procedure file repeatedly, the resulting doc file will grow and grow. You should delete the old doc files first if you need to rerun the procedure file.

The IF Statement

The IF operation is a control structure that provides conditional execution. The IF syntax is:

 IF (TEST_CONDITION) THEN
    <any SPIDER operations>
 ENDIF

You can delete a file if it already exists, using an IF statement. Copy bat03.spf to bat04.spf, and add the following lines above the DO loop. In addition, make one change to the RAN expression in the DO loop:

 ; The new lines

 IQ FI [exists]           ; If the following file exists, set [exists]=1, else [exists]=0
    randoc

 IF ([exists].EQ.1) THEN  ; IF [exists] is equal to 1, THEN do the following...
    DE
    randoc
 ENDIF

 ; The DO loop :
 [unused]= 0                ; Required for RAN operation

 DO  [index] = 1,20

   [ran] = 360 * RAN([unused])   ; ***** change this line too

   SD [index],[ran]        
     randoc                  
 ENDDO                     

 EN D

Look at the new contents of randoc.dat with the Unix cat command:

 cat randoc.dat

In addition to the IF..THEN operation, there is a one-liner version of IF, as well as IF..GOTO, which conditionally jumps to any line in the procedure file. See the Manual of Operations for details.

Multiplying the output of RAN by 360 changes the range of output numbers. When you run bat04.spf, you should be back to a single set of numbers in randoc.dat, this time in the range 0..360.

 [unused]= 0                     ; Required for RAN operation
 [ran]   = 10 * RAN([unused])    ; Random number 0..10
 [iran]  = INT([ran])   
 [iran]                          ; Echo: iran

 IF([iran] < 2) THEN
    [xsize]=100 
 ELSEIF ([iran] < 4) THEN
    [xsize]=400 
 ELSEIF ([iran] < 6) THEN
    [xsize]=600 
 ELSE
    [xsize]=800 
 ENDIF

 SYS
   echo ' 'Xsize: {****[xsize]}

EN

The value of Xsize reported by the procedure should vary with invocations as the random number changes.

Run-time Designation of File Names

If a series of files is created in a loop, how do you name them, especially if you don't know how many there will be? SPIDER enables filenames to be created at run time with the following syntax:

   template{****[register-variable]}

Create a new file, bat06.spf:

 DO [idx] = 1,10

    [ang] = [idx] * 20  ; Use this as the rotation angle

    RT
      test001           ; File (input)
      rot{***[idx]}     ; This will produce rot001, rot002, ...
      [ang]             ; Rotation angle

 ENDDO

 EN D

Commands/Montage
       Select rot001.dat (this is the first image in the series), click 'Ok'
       Enter 10 in the LAST FILE NUMBER box, click 'Accept'

You should see the test image going through a series of rotations.

Reading Document files

Next, use the random numbers in the doc file as rotation angles, using the UD operation to read the doc file:

 ; bat07.spf

 ; UD N counts the number of lines in the doc file, puts result in [n]
 UD N [n]   
   randoc

 DO [idx] = 1,[n]   ; Loop for each line in the doc file

   UD [idx],[rotang]
     randoc

   RT
     filt001
     rot{***[idx]} 
     [rotang]
 ENDDO

 EN D

Commands/Clear

Options/Image
       Check the SHOW FILE NAMES box, Click 'Accept'

Commands/Montage
       The selected file will still be rot010.dat. Reselect rot001.dat as the first image.
       Set LAST FILE NUMBER to 20.

You should see 20 copies of the test image, at random orientations.

Procedure files

Procedure files are subroutines that can be called from other procedure files. They can have variable arguments that change with every call to the procedure.

Copy bat07.spf to proc.spf, and make the following changes:

• Add 3 'FR' operations to the top of the file. These File-Read operations set the name of a file to a symbolic parameter, such as [symvar]. These parameters can become any file name used by the procedure file that calls this procedure. The text inside the ?question marks? is used as a prompt in an interactive session; here, they are simply used as a reminder of the type of file the procedure is expecting.
  
• Change all instances of "randoc" to "[angsdocfile]", "filt001" to "[img]", and "rot" to "[output]".
  
• Change the last operation, EN, to RE. When all the lines in a procedure file have been executed, control is returned to the calling procedure file.

 FR
   ?angles?[angsdocfile]      ; This sets [angsdocfile] = "randoc"

 FR
   ?input file?[img]          ; This sets [img] = "filt001"

 FR
   ?output template?[output]  ; Sets [output] = "rot"

 UD N  [n]                    ;   
   [angsdocfile]              ; *** Change randoc to angsdocfile

 DO [num] = 1,[n]

   UD [num],[ang]
     [angsdocfile]           ; *** Change randoc to angsdocfile
 
   RT
     [img]                   ; *** Change filt001 to [img]
     [output]{***[num]}      ; *** This becomes rot001, rot002, ...
     [ang]

 ENDDO

RE                           ; ***** IMPORTANT - CHANGE "EN D" TO "RE"  *****

 ; bat08.spf  : rotate images using a procedure call

 @proc          ; Call to the procedure (named proc)
   randoc       ; Doc file of rotation angles
   test001      ; Input file
   rot          ; Output template (will be expanded to rot001, rot002, ...)

                ; Any SPIDER operations may go here

 EN D

Run the procedure file: spider spf/dat @bat08 . SPIDER runs the procedure file, which in turn, calls the procedure.

The results are the same as for bat07.spf. Change bat08.spf so that the 3rd line after the procedure is "root", instead of "rot". When you run the procedure file, it will create a new set of images. Procedures provide reusable code with variable filenames, that can be called by procedure files, or even other procedures.

Passing Arguments to Procedures

Register variables can be passed as arguments to procedures.

 ; bat09.spf  : Fourier transform images using a procedure call

 [n] = 5        ; Number of images
 @proc-ft([n])  ; Call to the procedure (named proc-ft)

 EN D

 ([numimgs])

 DO [img]=1,[numimgs]
   FT                 ; Fourier transform
     sam{***[img]}    ; Input image
     four{***[img]}   ; Output Fourier image

   PW                 ; Power spectrum of Fourier image
     four{***[img]}   ; Input image
     pow{***[img]}    ; Output Power spectrum
 ENDDO

 RE

Run the procedure file: spider spf/dat @bat09
SPIDER runs the procedure file: bat09 which in turn, calls the procedure: proc-ft with argument [numimgs] set to the value of: 5.

Examine the Power Spectra in Web. Adjust the contrast with: Options - Contrast to see more detail.

Updated: Mar 31, 2009,     Authors: Bill Baxter & ArDean Leith