Refinement

Running Serial Refinement

1. Access your refinement_dir, e.g.:
   cd   /mydir/project/Refinement
2. If you are using OPTIONAL "amplitude enhancement" you will have to add the following initial files to input

   ¤	input/scattering	X-ray scattering power spectrum
   ¤	input/mask	Mask created from ribosome-translocon

3. Edit refine_settings.pam to set necessary values for refinement parameters and and any abnormal input file names, e.g.
   nedit refine_settings.pam
4. Start refinement using refine.pam, e.g.
   spider pam/ftf @refine 0 &

1. Login to master node of cluster, where PubSub has been installed and is running, e.g.:
   ssh   radha
2. Access your refinement_dir, e.g.:
   cd   /mydir/project/Refinement
3. If you are using OPTIONAL "amplitude enhancement" you will have to add the following initial files to input

   input/scattering	X-ray scattering power spectrum
   input/mask	Mask created from ribosome-translocon

4. Remove any existing output files currently in the refinement directory, e.g.:
   \rm f    pubsub.log    results.*    LOG.*    fort.*    work/*    local/*    final/*    core*   
5. Copy latest SPIDER executable into the current directory and rename it: spider (ensures that same SPIDER executable is used throughout whole refinement), e.g.:
   cp /usr/local/spider/bin/spider_linux_mp_opt64.18.06    spider
6. Edit refine_settings.pam to set necessary values for refinement parameters and input file names, e.g.
   xedit    refine_settings.pam
7. Start Master SPIDER refinement process pub_refine.pam, e.g.
   spider pam/ftf    @pub_refine 0 &
8. Master SPIDER refinement process will create and publish new SPIDER jobs as necessary.
9. If the refinement fails.
   
   1. Use 'killspi' to stop any remaining parallel jobs that belong to you.
   2. Use 'wherespi' to find any remaining parallel jobs. If any remain, use 'killspi' again until they are all gone.
   3. Clean any un-needed output files (you may want to save final/* and results.* if you are restarting at a later iteration): e.g..
      \rm -rf    pubsub.log    results.*    LOG.*    work/    local/   final/    core*    fort.*
   4. To restart while saving some defocus group results for this iteration, alter your: input/order_select,* and input/order_select_sort.* files to contain only the desired defocus groups and edit: pub_refine.pam to stop at end of iteration. Restart, then commence again at next iteration with step: 7.
   5. Restart at step: 8.

1. Set initial parameters & file names
2. Prepare input files.
3. Loop over all defocus groups
• Create group CTF's weighted by number of particles.

  TF C3	Compute phase contrast transfer function
  AR	Weight current CTF by number of particles

4. Loop over all iterations creating iteration volumes
   
   • Set parameters for this iteration
   • Create Fourier and document files used for this iteration
     

     FT	Fourier on current volume produced by previous iter.
     VO EA	Create reference angles for this iteration
     DOC CREATE	Create selection file for reference angles

   • Loop over all defocus groups creating group volumes
     
     • Create CTF corrected volumes

       MU	Multiply Fourier of current volume by CTF file for this group
       FT	Fourier back transform the CTF corrected volume
       PJ 3Q	Create stack holding reference angular projections from CTF corrected volume (reference projections)
       CP	Copy current aligned images to inline stack for speed
       AP REF or AP SH	Find reference projections matching each current aligned image. Find alignment parameters to match selected ref. projection.

     • Loop over all images in current group
     • Align original images for back-projection

       RT SQ

       Rotate, mirror, and shift unaligned original image to create current aligned image

     • Back-project original images into new group volume

       BP 32F	Calculate refined group volume from new aligned images and angle data
       CG PH	Calculate center of gravity of refined group volume
       SH F	Shift refined group volume to center
       RF 3	Calculate phase residual for refined group volume

   • Merge group volumes into iteration volume

     TF CTS	Create CTF corrected volume from refined group volumes
     TF CTS	Create 1/2 CTF corrected volume from refined group volumes
     TF CTS	Create 1/2 CTF corrected volume from refined group volumes
     RF 3	Calculate resolution for group volume
     @enhance	Optional amplitude enhancement
     FQ	Filter final corrected volume to resolution

5. End iterations when angular displacements are small in all groups.
6. Loop over all defocus groups
• Create final group volumes using BP CG

  BP CG	Create final group volume from aligned images and angle data
  BP CG	Create final 1/2 group volume from aligned images and angle data
  BP CG	Create final 1/2 group volume from aligned images and angle data
  RF 3	Calculate resolution for final group volume

7. Continue with small-angle refinement or GO TO FINAL VOLUME MERGE.
   • Set parameters for small-angle refinement
   • Loop over all small-angle iterations creating iteration volumes
     
     • Set parameters for this iteration
     • Create Fourier and document files used for this iteration
       

       FT	Fourier on current volume produced by previous iter.
       VO EA	Create relative offset for reference angles for this iteration
       DOC CREATE	Create selection file for reference angles

     • Loop over all defocus groups creating group volumes
       
       • Create CTF corrected volumes

         MU	Multiply Fourier of current volume by CTF file for this group
         FT	Fourier back transform the CTF corrected volume

       • Loop over all images in current group
         • Align original images for back-projection

           UD IC	Get current angle for this image
           VO RAS	Get angular range around current angle
           PJ 3Q	Project thru volume at these angles to create set of angular reference projections
           AP REF	Find nearest reference projection matching each current image.
           RT SQ	Rotate, and shift unaligned original image to create aligned image

       • Back-project original images into new group volume

         BP 32F	Calculate refined group volume from new aligned images and angle data
         RF 3	Calculate phase residual for refined group volume

     • Merge group volumes into iteration volume

       TF CTS	Create CTF corrected iteration volume from refined group volumes
       TF CTS	Create 1/2 CTF corrected iteration volume from refined group volumes
       TF CTS	Create 1/2 CTF corrected iteration volume from refined group volumes
       RF 3	Calculate resolution for iteration volume
       FQ	Filter final corrected volume to resolution
       CG PH	Calculate center of gravity of refined volume
       SH F	Shift refined volume to center

   • End iterations when angular displacements are small in all groups.

8. Loop over all defocus groups
   • Create final group volumes using BP CG

     BP CG	Create final group volume from aligned images and angle data
     BP CG	Create final 1/2 group volume from aligned images and angle data
     BP CG	Create final 1/2 group volume from aligned images and angle data
     RF 3	Calculate resolution for final group volume

9. Merge group volumes into final iteration volume and determine final resolution

   TF CTS	Create final CTF corrected iteration volume from final group volumes
   TF CTS	Create final 1/2 CTF corrected iteration volume from final group volumes
   TF CTS	Create final 1/2 CTF corrected iteration volume from final group volumes
   RF 3	Calculate resolution for final iteration volume

10. Refinement finished.

Tree of Procedure calls

refine refine_settings prepare grploop smangloop mergegroups enhance endmerge endrefine

pub_refine refine_settings prepare pub_refine_start refine_settings pub_ref_loop_clone grploop smangloop endmerge pub_refine_doc_sync pub_ref_loop_declone pub_refine_wait pub_refine_start pub_ref_merge_clone mergegroups enhance pub_ref_merge_declone pub_refine_wait endrefine