++++++++++++++++++++++++++++++++++++++++
Processing Sequences with Missing Images
++++++++++++++++++++++++++++++++++++++++

DIALS treats sequences as a contiguous set of rotation images, and in some circumstances deviations from this will cause problems. A clear example of this is having one or more "bad" images in your data set - simply removing them *will not* be enough to allow processing to complete smoothly.

Importing
=========

Importing the images with ``template=blah_####.cbf`` will not work, as this checks for continuous images, and it will be necessary to use ``allow_multiple_sequences=true``. After this, finding spots and indexing work as usual, but with multiple imagesets.

Refinement
==========

Refinement *without* ``scan_varying=true`` will work fine, but following that, scan varying refinement will fail with an error::

  Sorry: A single scan-varying crystal model cannot be refined when associated with more than one scan or goniometer

The issue here is that scan-varying refinement requires that each crystal being refined is associated with a single scan. In our case, we have a single crystal model from indexing, but this is associated with multiple scans. To proceed, we can split the experiment list into individual files. This breaks the sharing of models between experiments by making copies of each model for each file::

  dials.split_experiments indexed.refl indexed.expt

From this point, we could process each block as per the usual tutorial instructions (ideally in separate directories). However, this will refine the beam and the detector, which *should* be shared, separately for each process. A better way to proceed would be to recombine the experiments as follows::

  dials.combine_experiments experiments_*.expt reflections_*.refl \
    reference_from_experiment.goniometer=0 \
    reference_from_experiment.detector=0 \
    reference_from_experiment.beam=0

The ``reference_from_experiment`` options tells ``dials.combine_experiments`` to take the goniometer, detector and beam models only from the first experiment (although any would have done as these are merely copies of each other). The resulting ``combined.expt`` file has a separate (but identical) crystal model for each scan, alongside shared goniometer, detector and beam models. Scan-varying refinement and integration can now proceed as usual::

  dials.refine combined.expt combined.refl scan_varying=true
  dials.integrate refined.expt refined.refl

Export
======

Currently, ``dials.export`` will not allow MTZ export of the multiple-experiment integration, but we can rely on ``dials.split_experiments`` again::

  dials.split_experiments integrated.expt integrated.refl
  i=0
  for e in $(ls experiments_*.expt)
  do
    r=reflections_$i.refl
    echo "dials.export $e $r mtz.hklout=integrated_$i.mtz"
    ((i++))
  done
