Implementing the DASD DIAG Driver for Debian GNU/Linux on the s390x Architecture

Disclaimer

This is not an official Debian site.  The author is not a member of the Debian kernel team.  The author is not a member of the Debian s390x port team.  The author is not a member of the Debian Installer team.  This is not an official Linux kernel site.  This is not an official IBM site.  This document details the author's experiences and recommendations in implementing the DASD DIAG driver for the Debian s390x port running in a virtual machine under z/VM®.  All opinions expressed are those of the author and do not necessarily represent the opinions or the official positions of the staff or management of Debian, the Linux kernel organization, IBM, or anyone else.  This information is presented in the hope that it will be useful, but without any warranty or guarantee of any kind.  This information is presented free of charge, free of support, free of service, and free of liability.  Take this information with as many grains of salt as you think it's worth; and use it, if you choose to do so, entirely at your own risk.  The author hereby explicitly places this material in the public domain.  All trademarks, registered trademarks, service marks, etc. are the property of their respective owners.

Maintenance Log

Purpose

This document explains how to configure Debian GNU/Linux running in a virtual machine under z/VM to use the high performance DIAG driver (dasd_diag_mod) to access its DASD devices. 

Who can use the DIAG driver?

In order to use the DIAG driver, you must be running Debian GNU/Linux in a virtual machine under IBM's z/VM® operating system.  The DIAG driver is supported for both FBA and CKD devices.

What is the DIAG driver?

The DIAG driver makes use of the DIAGNOSE instruction in a virtual machine, with DIAGNOSE code X'250', to make use of CP's block I/O service to do I/O to DASD devices, rather than using the traditional SSCH (START SUBCHANNEL) instruction to do DASD I/O.  With DIAG X'250' the I/O can be synchronous or asynchronous, it can exploit CP's minidisk caching, and completion of asynchronous I/O is signaled by an external interruption, rather than a traditional I/O interruption.  The DIAG X'250' instruction provides the virtual machine with a device-independent interface for accessing data on a DASD device.  Data is accessed by a relative block number, even on a CKD device, rather than by cylinder, head, and record, which may vary from one device type to another depending on drive geometry.  Error recovery actions and channel program retry, if needed, are all done automatically by CP and transparently to the guest virtual machine.  The guest operating system need not concern itself with such matters.  Only permanent I/O errors are reflected to the guest virtual machine.

Why would I want to use the DIAG driver?

Performance benchmarks often show improvement in throughput with the DIAG X'250' interface as compared with SSCH in a virtual machine, particularly with minidisk caching on and particularly if multiple servers share the same (read-only) minidisks.  The performance improvement is even more pronounced if you are using FBA devices (real or emulated) instead of CKD devices. 

Are there any kernel requirements for using the DIAG driver?

Yes.  There must be support in the kernel for "DIAG access to disks".  In your kernel config file, "CONFIG_DASD_DIAG" must be set to "m" (module) or "y" (built-in to the kernel).  All Debian-supplied stock kernels have CONFIG_DASD_DIAG=m set in their config files.  All examples in this document assume that CONFIG_DASD_DIAG is set to "m" for support as a module.

Are there any other restrictions on using the DIAG driver?

Yes.  The Linux kernel for s390x currently supports four DASD formats: Linux disk layout (ldl), compatibility disk layout (cdl), CMS non-reserved minidisk, and CMS reserved minidisk.  ldl and cdl formats are supported only on CKD DASD devices.  CMS non-reserved and CMS reserved formats are supported on both CKD and FBA DASD devices.  Currently, cdl format is the only format supported by the Debian Installer.  In other words, if you ask the Debian Installer to format a disk during installation, it will always use cdl format.  Unfortunately, cdl format is the only format that cannot be used with the DIAG driver!  (Don't worry about that.  There are ways around this problem, which I will illustrate.)

If you pre-format a disk in the ldl format, CMS non-reserved format, or CMS reserved format prior to installation (a procedure not explained in this document) and tell the installer not to format the disk, then when you get to the "create partitions" step, parted will recognize the implicit partition present on such a device and will allow you to assign a mount point to it or designate it as a swap partition.  However, the Debian installer has no support for the DIAG driver.

If you are installing to CKD devices, you can safely turn on DIAG support after installation, provided you did not use cdl format.  However, if you are installing to FBA DASD devices, DIAG support can only be safely turned on after installation if you formatted the disks in CMS with a block size of 512.  Since this destroys much of the performance gain possible with the DIAG driver, I don't recommend this.  The method illustrated in this document will be an installation using the Debian installer to CKD DASD devices in cdl format, followed by a manual copying to CMS reserved disks and conversion to the DIAG driver.  This technique works whether your final target disks are CKD devices or FBA devices, but the example uses CKD devices.

There is one further restriction: the s390x boot loader, zIPL, does not support writing out IPL records on a device controlled by the DIAG driver.  It can be booted from such a device, but it cannot write out IPL records to such a device.  Therefore, I recommend creating a small /boot partition, just large enough to hold the necessary boot files, and have this minidisk use the ECKD driver (or FBA driver, as appropriate).  All other minidisks, when suitably configured, can use the DIAG driver, including the / partition.

OK, how do I do this?

Step 1

Use the Debian installer and install in the normal way.  (I suggest that you use CKD DASD and allow the installer to use cdl format.)  In my example, I assume a configuration of four minidisks: 0200 as the / partition, 0201 as the /boot partition, 0202 as the /home partition, and 0203 as a swap partition.  All minidisks are in cdl format, and all minidisks contain exactly one partition which occupies all available space on the disk.  In my sample server, all minidisks are allocated on a 3390 DASD, which is a CKD device.  The 0200 minidisk is 2000 cylinders, the 0201 minidisk is 75 cylinders, the 0202 minidisk is 500 cylinders, and the 0203 minidisk is 500 cylinders.  In fact, in my sample scenario, I will postulate the existence of two identical servers so configured, which I will give the very imaginative names of DEBIAN1 and DEBIAN2.  These servers are booted by means of "IPL 0201".  I assume that all minidisks use a block size of 4096.

Step 2

Now, create a duplicate set of minidisks for DEBIAN1: 0400, 0401, 0402, and 0403, with approximately the same sizes as 0200, 0201, 0202, and 0203, respectively.  Here you may use FBA DASD, if you have it and wish to use it.  Since there are eight 512-byte blocks in a 4096-byte block, make the number of 512-byte blocks for your FBA DASD device eight times the number of 4096-byte blocks on the corresponding CKD device.  Also, for best performance, make sure that your FBA minidisk starts on a block number which is a multiple of eight.

Use whatever method you normally use in your shop for allocating minidisks in the CP directory, but do not format them.  Now take DEBIAN1 down.  (In other words, do a "shutdown -h now", then LOGOFF the virtual machine once it is down.)  Link to these four new minidisks from a userid running CMS, such as MAINT, and do a FORMAT and RESERVE on them.  The CMS RESERVEd minidisk is the only format of minidisk which can be used by both the Linux DIAG driver and CMS; therefore I recommend it.  (A CMS non-reserved minidisk can be accessed by CMS after formatting and before Linux builds a file system or swap space on the implicit partition; but once Linux builds a file system or swap space on the implicit partition, the CMS file system structures are overwritten; and CMS can no longer access the minidisk.)

The FORMAT command does the low-level formatting and builds a CMS file system on the minidisk.  The RESERVE command allocates all available space on the minidisk to a single CMS file, reserving only enough blocks at the beginning of the minidisk for the required CMS file system structures.  In detail, the structure of the minidisk for a CKD device is as follows:
 

     Start   No. of      Description
     Block   Blocks
     =====   ======      ===========================================
       1       2         IPL records
       3       1         Volume label
       4       2         CMS file directory
       6       *         Pointer blocks for allocation map
       next    *         CMS allocation map
       next    *         Duplicate for CMS allocation map backup
       next    *         File pointer blocks
       next    *         File data blocks (the reserved file itself)

The single CMS file has a filemode number of 6, which means that the existing blocks must be updated in place rather than be re-allocated somewhere else when their contents are changed.  (This only means something when CMS is doing I/O to the minidisk, not Linux.) 

Linux treats the blocks of the CMS reserved file on such a minidisk (except the last block) as the single implicit partition on the minidisk.  When Linux builds a file system or a swap space on that partition, only the blocks of the CMS reserved file are altered.  The other structures on the minidisk which are outside the blocks of the file itself are not altered.  Thus, CMS still recognizes the minidisk as having a valid CMS file system on it; and CMS can access the minidisk.  Having the minidisk recognizable by CMS has many advantages, including being able to use VM/CMS tools to do backup, restore, copying, and other manipulations of the data.  Here is a sample console session from the CMS id which does the FORMAT and RESERVE commands for DEBIAN1's new minidisks.  For illustrative purposes I assume that the write password for all of the minidisks is WRITE.  For security reasons this is not recommended!  I hope you come up with a more imaginative (and hard to guess) password than this, but it will do for illustrative purposes.  The sample session assumes that filemode E is a free (unused) filemode.  It also assumes that device numbers 0400, 0401, 0402, and 0403 do not exist.
 

     cp link debian1 400 400 w
     ENTER WRITE PASSWORD:
     write   [display inhibited]
     Ready;
     format 400 e ( blksize 4096
     DMSFOR603R FORMAT will erase all files on disk E(400). Do you wish to continue?
     Enter 1 (YES) or 0 (NO).
     1
     DMSFOR605R Enter disk label:
     deb200
     DMSFOR733I Formatting disk E
     DMSFOR732I 2000 cylinders formatted on E(400)
     Ready;
     reserve debian1 mdsk0200 e
     DMSRSV603R RESERVE will erase all files on disk E(400). Do you wish to continue?
      Enter 1 (YES) or 0 (NO).
     1
     DMSRSV733I Reserving disk E
     Ready;
     release e ( det
     DASD 0400 DETACHED
     Ready;

     cp link debian1 401 401 w
     ENTER WRITE PASSWORD:
     write   [display inhibited]
     Ready;
     format 401 e ( blksize 4096
     DMSFOR603R FORMAT will erase all files on disk E(401). Do you wish to continue?
     Enter 1 (YES) or 0 (NO).
     1
     DMSFOR605R Enter disk label:
     deb201
     DMSFOR733I Formatting disk E
     DMSFOR732I 75 cylinders formatted on E(401)
     Ready;
     reserve debian1 mdsk0201 e
     DMSRSV603R RESERVE will erase all files on disk E(401). Do you wish to continue?
      Enter 1 (YES) or 0 (NO).
     1
     DMSRSV733I Reserving disk E
     Ready;
     release e ( det
     DASD 0401 DETACHED
     Ready;

     cp link debian1 402 402 w
     ENTER WRITE PASSWORD:
     write   [display inhibited]
     Ready;
     format 402 e ( blksize 4096
     DMSFOR603R FORMAT will erase all files on disk E(402). Do you wish to continue?
     Enter 1 (YES) or 0 (NO).
     1
     DMSFOR605R Enter disk label:
     deb202
     DMSFOR733I Formatting disk E
     DMSFOR732I 500 cylinders formatted on E(402)
     Ready;
     reserve debian1 mdsk0202 e
     DMSRSV603R RESERVE will erase all files on disk E(402). Do you wish to continue?
      Enter 1 (YES) or 0 (NO).
     1
     DMSRSV733I Reserving disk E
     Ready;
     release e ( det
     DASD 0402 DETACHED
     Ready;

     cp link debian1 403 403 w
     ENTER WRITE PASSWORD:
     write   [display inhibited]
     Ready;
     format 403 e ( blksize 4096
     DMSFOR603R FORMAT will erase all files on disk E(403). Do you wish to continue?
     Enter 1 (YES) or 0 (NO).
     1
     DMSFOR605R Enter disk label:
     deb203
     DMSFOR733I Formatting disk E
     DMSFOR732I 500 cylinders formatted on E(403)
     Ready;
     reserve debian1 mdsk0203 e
     DMSRSV603R RESERVE will erase all files on disk E(403). Do you wish to continue?
      Enter 1 (YES) or 0 (NO).
     1
     DMSRSV733I Reserving disk E
     Ready;
     release e ( det
     DASD 0403 DETACHED
     Ready;

I put a blank line between the sections for each minidisk for improved readability, but an actual console log would not contain these blank lines.  I also eliminated the timing information from all of the "Ready;" messages, as though "SET RDYMSG SMSG" were in effect.  Notice that we are linking to the 400-series minidisks of DEBIAN1, but we are using 200-series minidisk labels and reserved filenames.  We are doing this because these minidisks will eventually take the place of the current 200-series minidisks.  Note also that all the minidisks are formatted by CMS with a block size of 4096.  I recommend this even for FBA DASD devices.  For FBA DASD devices, the physical block size is always 512, but the CMS logical block size can still be 4096.  Unlike the FBA driver, which ignores the CMS logical block size and treats the minidisk as if had a block size of 512, the DIAG driver honors the CMS logical block size.  This generally results in further performance improvements.

Step 3

OK, now login to DEBIAN2 as root.  We are going to link to DEBIAN1's minidisks from DEBIAN2 and bring them online.  (Remember, we are reconfiguring DEBIAN1's minidisks, not DEBIAN2's minidisks.  DEBIAN1 is down and logged off right now.)  For illustrative purposes I am assuming that the read password for all of DEBIAN1's existing cdl-format disks is READ.  Again, this is not recommended for security reasons.  I do this for illustrative purposes only.  Here is a sample console session from DEBIAN2:
 

     debian2:~# vmcp LINK DEBIAN1 0200 0400 R READ
     debian2:~# vmcp LINK DEBIAN1 0201 0401 R READ
     debian2:~# vmcp LINK DEBIAN1 0202 0402 R READ
     debian2:~# vmcp LINK DEBIAN1 0400 0500 W WRITE
     debian2:~# vmcp LINK DEBIAN1 0401 0501 W WRITE
     debian2:~# vmcp LINK DEBIAN1 0402 0502 W WRITE
     debian2:~# vmcp LINK DEBIAN1 0403 0503 W WRITE
     debian2:~# cd /sys/bus/ccw/devices/0.0.0400
     debian2:/sys/bus/ccw/devices/0.0.0400# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0400# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0400# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0400# echo 1 >readonly
     debian2:/sys/bus/ccw/devices/0.0.0400# cat readonly
     1
     debian2:/sys/bus/ccw/devices/0.0.0400# echo 1 >online
     debian2:/sys/bus/ccw/devices/0.0.0400# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0400# cd ../0.0.0401
     debian2:/sys/bus/ccw/devices/0.0.0401# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0401# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0401# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0401# echo 1 >readonly
     debian2:/sys/bus/ccw/devices/0.0.0401# cat readonly
     1
     debian2:/sys/bus/ccw/devices/0.0.0401# echo 1 >online
     debian2:/sys/bus/ccw/devices/0.0.0401# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0401# cd ../0.0.0402
     debian2:/sys/bus/ccw/devices/0.0.0402# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0402# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0402# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0402# echo 1 >readonly
     debian2:/sys/bus/ccw/devices/0.0.0402# cat readonly
     1
     debian2:/sys/bus/ccw/devices/0.0.0402# echo 1 >online
     debian2:/sys/bus/ccw/devices/0.0.0402# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0402# cd ../0.0.0500
     debian2:/sys/bus/ccw/devices/0.0.0500# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0500# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0500# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0500# modprobe dasd_diag_mod   [if not already loaded]
     debian2:/sys/bus/ccw/devices/0.0.0500# echo 1 >use_diag
     debian2:/sys/bus/ccw/devices/0.0.0500# cat use_diag
     1
     debian2:/sys/bus/ccw/devices/0.0.0500# echo 1 >online
     debian2:/sys/bus/ccw/devices/0.0.0500# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0500# cd ../0.0.0501
     debian2:/sys/bus/ccw/devices/0.0.0501# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0501# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0501# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0501# echo 1 >online
     debian2:/sys/bus/ccw/devices/0.0.0501# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0501# cd ../0.0.0502
     debian2:/sys/bus/ccw/devices/0.0.0502# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0502# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0502# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0502# echo 1 >use_diag
     debian2:/sys/bus/ccw/devices/0.0.0502# cat use_diag
     1
     debian2:/sys/bus/ccw/devices/0.0.0502# echo 1 >online
     debian2:/sys/bus/ccw/devices/0.0.0502# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0502# cd ../0.0.0503
     debian2:/sys/bus/ccw/devices/0.0.0503# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0503# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0503# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0503# echo 1 >use_diag
     debian2:/sys/bus/ccw/devices/0.0.0503# cat use_diag
     1
     debian2:/sys/bus/ccw/devices/0.0.0503# echo 1 >online
     debian2:/sys/bus/ccw/devices/0.0.0503# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0503# cat /proc/dasd/devices
     0.0.0200(ECKD) at ( 94:     0) is dasda       : active at blocksize: 4096, 360000 blocks, 1406 MB
     0.0.0201(ECKD) at ( 94:     4) is dasdb       : active at blocksize: 4096, 13500 blocks, 52 MB
     0.0.0202(ECKD) at ( 94:     8) is dasdc       : active at blocksize: 4096, 90000 blocks, 351 MB
     0.0.0203(ECKD) at ( 94:    12) is dasdd       : active at blocksize: 4096, 90000 blocks, 351 MB
     0.0.0400(ECKD) at ( 94:    16) is dasde   (ro): active at blocksize: 4096, 360000 blocks, 1406 MB
     0.0.0401(ECKD) at ( 94:    20) is dasdf   (ro): active at blocksize: 4096, 13500 blocks, 52 MB
     0.0.0402(ECKD) at ( 94:    24) is dasdg   (ro): active at blocksize: 4096, 90000 blocks, 351 MB
     0.0.0500(DIAG) at ( 94:    28) is dasdh       : active at blocksize: 4096, 360000 blocks, 1406 MB
     0.0.0501(ECKD) at ( 94:    32) is dasdi       : active at blocksize: 4096, 13500 blocks, 52 MB
     0.0.0502(DIAG) at ( 94:    36) is dasdj       : active at blocksize: 4096, 90000 blocks, 351 MB
     0.0.0503(DIAG) at ( 94:    40) is dasdk       : active at blocksize: 4096, 90000 blocks, 351 MB
     debian2:/sys/bus/ccw/devices/0.0.0503#

In this case, the password is supplied on the command line.  Therefore, CP does not prompt for a password.  If it did, you would need to be logged on to the virtual machine console in order to answer the CP password prompts.  Since the password is supplied on the command line, this can be done, for example, in a remote SSH session.  Note that we did not link to DEBIAN1 0203.  That's the old cdl-format swap minidisk.  We don't need to copy any data off of a swap minidisk.  But we did link to the new swap minidisk, DEBIAN1 0403 (as 0503), because we need to create a swap space on it.  Also note that we did not set use_diag to 1 for the 501 minidisk (DEBIAN1 0401).  That is going to be our new /boot minidisk.  Although it is a CMS reserved minidisk, it must use the ECKD driver (or the FBA driver, as appropriate) so that zIPL can write IPL records to it.  We also set the "readonly" variable to 1 for the three minidisks that we are going to copy to let the driver know that these minidisks are to be treated as read-only minidisks.  They were linked read-only by CP, but the driver doesn't know that.

In this example, dasda, dasdb, dasdc, and dasdd are the device names of DEBIAN2's own minidisks: 0200, 0201, 0202, and 0203, respectively.  The device names of the seven dynamically linked minidisks are assigned in the order that they are brought online.  dasde is assigned to 0400 (DEBIAN1 0200), dasdf is assigned to 0401 (DEBIAN1 0201), and dasdg is assigned to 0402 (DEBIAN1 0202).  The console log above shows how they are manually brought online.  dasdh is assigned to 0500 (DEBIAN1 0400), dasdi is assigned to 0501 (DEBIAN1 0401), dasdj is assigned to 0502 (DEBIAN1 0402), and dasdk is assigned to 0503 (DEBIAN1 0403).

Step 4

OK, now that the CMS minidisks have been brought online, it's time to create the file systems and the swap space.  We continue the console log above where we left off.
 

     debian2:/sys/bus/ccw/devices/0.0.0503# cd
     debian2:~# mke2fs -L DEB200 -t ext3 -v /dev/dasdh1
     mke2fs 1.41.3 (12-Oct-2008)
     fs_types for mke2fs.conf resolution: 'ext3', 'default'
     Filesystem label=DEB200
     OS type: Linux
     Block size=4096 (log=2)
     Fragment size=4096 (log=2)
     89936 inodes, 359617 blocks
     17980 blocks (5.00%) reserved for the super user
     First data block=0
     Maximum filesystem blocks=369098752
     11 block groups
     32768 blocks per group, 32768 fragments per group
     8176 inodes per group
     Superblock backups stored on blocks:
             32768, 98304, 163840, 229376, 294912

     Writing inode tables: done
     Creating journal (8192 blocks): done
     Writing superblocks and filesystem accounting information: done

     This filesystem will be automatically checked every 35 mounts or
     180 days, whichever comes first.  Use tune2fs -c or -i to override.
     debian2:~# mke2fs -L DEB201 -t ext3 -v /dev/dasdi1
     mke2fs 1.41.3 (12-Oct-2008)
     fs_types for mke2fs.conf resolution: 'ext3', 'small'
     Filesystem label=DEB201
     OS type: Linux
     Block size=4096 (log=2)
     Fragment size=4096 (log=2)
     13504 inodes, 13477 blocks
     673 blocks (4.99%) reserved for the super user
     First data block=0
     Maximum filesystem blocks=16777216
     1 block group
     32768 blocks per group, 32768 fragments per group
     13504 inodes per group

     Writing inode tables: done
     Creating journal (1024 blocks): done
     Writing superblocks and filesystem accounting information: done

     This filesystem will be automatically checked every 20 mounts or
     180 days, whichever comes first.  Use tune2fs -c or -i to override.
     debian2:~# mke2fs -L DEB202 -t ext3 -v /dev/dasdj1
     mke2fs 1.41.3 (12-Oct-2008)
     fs_types for mke2fs.conf resolution: 'ext3', 'small'
     Filesystem label=DEB202
     OS type: Linux
     Block size=4096 (log=2)
     Fragment size=4096 (log=2)
     89952 inodes, 89897 blocks
     4494 blocks (5.00%) reserved for the super user
     First data block=0
     Maximum filesystem blocks=92274688
     3 block groups
     32768 blocks per group, 32768 fragments per group
     29984 inodes per group
     Superblock backups stored on blocks:
             32768

     Writing inode tables: done
     Creating journal (4096 blocks): done
     Writing superblocks and filesystem accounting information: done

     This filesystem will be automatically checked every 21 mounts or
     180 days, whichever comes first.  Use tune2fs -c or -i to override.
     debian2:~# mkswap -L DEB203 /dev/dasdk1
     Setting up swapspace version 1, size = 368214 kB
     LABEL=DEB203, UUID=00bd6c9f-b2a8-4f90-b2d6-744d53153f4d
     debian2:~#

Note two things.  First, we did not use dasdfmt or fdasd on these devices to prepare them.  The CMS FORMAT and RESERVE commands substituted for these commands.  Second, we created all file systems and swap spaces on a partition, not a device.  For example, we created a file system on /dev/dasdh1, the first (and only) partition of /dev/dasdh, not on /dev/dasdh itself.  Linux interprets the CMS reserved file (except the last block) as the single implicit partition on this device.

Step 5

OK, now that we have created file systems on the new CMS reserved minidisks, it's time to copy the data from the old cdl minidisks to the new CMS reserved minidisks.  I chose /media and /mnt as the mount points in the example below, but it doesn't have to be those two directories.  What is important is that both directories, prior to mounting a file system on them, are empty.  Also, neither mount point may be "under" the other in the overall file system structure.  We continue the console log above where we left off.
 

     debian2:~# mount -r /dev/dasde1 /media
     debian2:~# mount /dev/dasdh1 /mnt
     debian2:~# cp -a /media/. /mnt
     debian2:~# umount /mnt
     debian2:~# umount /media
     debian2:~# mount -r /dev/dasdf1 /media
     debian2:~# mount /dev/dasdi1 /mnt
     debian2:~# cp -a /media/. /mnt
     debian2:~# umount /mnt
     debian2:~# umount /media
     debian2:~# mount -r /dev/dasdg1 /media
     debian2:~# mount /dev/dasdj1 /mnt
     debian2:~# cp -a /media/. /mnt
     debian2:~# umount /mnt
     debian2:~# umount /media
     debian2:~#

There, that wasn't so bad, was it?  After all of the preparation, the actual copying process wasn't so bad.  Each partition is copied separately.  The input partition is mounted with the "-r" option to specify that the file system is read-only.  This is not the same as telling the driver that the device is read-only.  The "-a" option of cp is critical.  Make sure you specify it.  The trailing forward slash period (/.) sequence after /media in the cp commands is also critical.  Be sure you type the commands exactly as written above.

Note that a variation of this technique can be used to enlarge the size of a minidisk which is nearly full.  It's a good idea to have a spare Linux virtual machine around to be able to perform this kind of maintenance on another server while it is down.

Step 6

Now we will get rid of some minidisks that we don't need anymore.
 

     debian2:~# cd /sys/bus/ccw/devices/0.0.0402
     debian2:/sys/bus/ccw/devices/0.0.0402# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0402# echo 0 >online
     debian2:/sys/bus/ccw/devices/0.0.0402# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0402# cat readonly
     1
     debian2:/sys/bus/ccw/devices/0.0.0402# echo 0 >readonly
     debian2:/sys/bus/ccw/devices/0.0.0402# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0402# cd ../0.0.0401
     debian2:/sys/bus/ccw/devices/0.0.0401# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0401# echo 0 >online
     debian2:/sys/bus/ccw/devices/0.0.0401# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0401# cat readonly
     1
     debian2:/sys/bus/ccw/devices/0.0.0401# echo 0 >readonly
     debian2:/sys/bus/ccw/devices/0.0.0401# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0401# cd ../0.0.0400
     debian2:/sys/bus/ccw/devices/0.0.0400# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0400# echo 0 >online
     debian2:/sys/bus/ccw/devices/0.0.0400# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0400# cat readonly
     1
     debian2:/sys/bus/ccw/devices/0.0.0400# echo 0 >readonly
     debian2:/sys/bus/ccw/devices/0.0.0400# cat readonly
     0
     debian2:/sys/bus/ccw/devices/0.0.0400# cd
     debian2:~# vmcp DETACH VIRTUAL 0400-0402
     0400-0402 DETACHED
     debian2:~#

We just got the three old cdl minidisks offline, then detached them from the virtual machine.

Step 7

Now we need to connect the pieces together and tweak the file system for booting.
 

     debian2:~# mount /dev/dasdh1 /mnt
     debian2:~# mount /dev/dasdi1 /mnt/boot
     debian2:~# mount /dev/dasdj1 /mnt/home
     debian2:~# mount -o bind /sys /mnt/sys
     debian2:~# mount -o bind /proc /mnt/proc
     debian2:~# mount -o bind /dev /mnt/dev
     debian2:~# mount -o bind /dev/pts /mnt/dev/pts
     debian2:~# mount -o bind /run /mnt/run
     debian2:~# mount -o bind /run/lock /mnt/run/lock
     debian2:~# mount -o bind /run/shm /mnt/run/shm
     debian2:~# mount -o bind /var/lib/nfs/rpc_pipefs /mnt/var/lib/nfs/rpc_pipefs
     debian2:~# chroot /mnt
     debian2:/#

The first three mount commands mount the three new minidisks that we just copied in a file system structure that duplicates the one they will have when they are in production.  The mount commands with the "-o bind" option mount the various "special" file systems, those not associated with a real disk, and map them to the new structure we just created.  To determine what these are, issue the mount command with no operands prior to starting the above sequence of commands.  Finally, the "chroot" command puts us in a nested shell session that makes the root directory the same as the new root directory of the DEBIAN1 server.  (In order for this to work properly, the two servers should be at compatible maintenance levels.)  We now have some work to do.

Note: this procedure assumes that the latest stretch (9.x) version of the sysconfig-hardware package, or later, is in use.  Older versions of sysconfig-hardware have no support for the DASD options.

Create (or edit) configuration files for each of the DASD devices in /etc/sysconfig/hardware, specifying the DASD configuration options (DASD_USE_DIAG and DASD_READONLY) as appropriate for each device.  These files have names of the form config-ccw-0.0.xxxx, where xxxx is the four-digit hexadecimal virtual device number of the device as it will be known in the DEBIAN1 server when it is actually booting.  In this example configuration, all of them except config-ccw-0.0.0201 should contain
 

     DASD_USE_DIAG=1

To let sysconfig-hardware know that the DIAG driver should be used for these devices.  config-ccw-0.0.0201, since it is the file for the device which contains the partition which will be mounted as /boot, must not have this option.  Note that there is no space either before or after the equal sign.

Create or edit the file /etc/modprobe.d/dasd.conf.  Add the "soft dependency" lines for dasd_diag_mod, if they are not already present, as follows:
 

     softdep dasd_eckd_mod pre: dasd_diag_mod
     softdep dasd_fba_mod pre: dasd_diag_mod

These lines tell modprobe that dasd_eckd_mod has a "software dependency" on dasd_diag_mod and that dasd_fba_mod also has a "software dependency" on dasd_diag_mod.  In other words, modprobe is told that it must load dasd_diag_mod before it loads either dasd_eckd_mod or dasd_fba_mod.  In a virtual machine environment where the DIAG driver is going to be used, especially if it is going to be used for the root file system, these "software dependencies" must be declared.

Edit /etc/zipl.conf and change the root file specification to use a UUID.  For example,
 

     parameters = root=UUID=3b516bc5-05d2-4d8c-99ca-55558ca7d47a ro vmhalt=LOGOFF vmpoff=LOGOFF

Note that by default this needs to be changed in two places: one under the [debian] section and one under the [old] section.  The UUID to use is of course the UUID of the partition on the new minidisk which will become the new root partition.  You can determine the UUID to use by issuing the blkid command with no operands.
 

Edit /etc/initramfs-tools/conf.d/resume and specify the UUID of the new swap partition.  For example,
 

     RESUME=UUID=85e16bac-438d-463b-b056-0c8ff123d8d0

Again, the UUID to use is the UUID of the partition on the new minidisk which will become the new swap partition.
 

Edit /etc/fstab and change the UUIDs of the partitions to match their new counterparts.  Make sure that the UUID of the root partition listed in this file matches the UUID that you specified for the root partition in /etc/zipl.conf.

Step 8

OK, now we need to rebuild the initial RAM file system.
 

     debian2:/# update-initramfs -u -k $(uname -r)
     update-initramfs: Generating /boot/initrd.img-2.6.32-5-s390x
     Using config file '/etc/zipl.conf'
     Building bootmap in '/boot'
     Building menu 'menu'
     Adding #1: IPL section 'debian' (default)
     Adding #2: IPL section 'old'
     Preparing boot device: 0501.
     Done.
     debian2:/#

The use of "$(uname -r)" in the update-initramfs command assumes that DEBIAN1 and DEBIAN2 use identical default kernel images.  Make sure that the initial RAM file system image that you rebuild is the one for the default kernel that DEBIAN1 runs at boot time.  Note that the message "Preparing boot device: 0501." refers to the device number of the partition currently referred to as /boot, which is the one we want to update.  If it used device number 0201 we would be updating DEBIAN2's boot minidisk, not the minidisk which will eventually become DEBIAN1's boot minidisk.

Step 9

The minidisks have now been prepared for booting.  It's time to clean up.
 

     debian2:/# exit
     debian2:~# umount /mnt/var/lib/nfs/rpc_pipefs
     debian2:~# umount /mnt/run/shm
     debian2:~# umount /mnt/run/lock
     debian2:~# umount /mnt/run
     debian2:~# umount /mnt/dev/pts
     debian2:~# umount /mnt/dev
     debian2:~# umount /mnt/proc
     debian2:~# umount /mnt/sys
     debian2:~# umount /mnt/home
     debian2:~# umount /mnt/boot
     debian2:~# umount /mnt
     debian2:~# cd /sys/bus/ccw/devices/0.0.0503
     debian2:/sys/bus/ccw/devices/0.0.0503# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0503# echo 0 >online
     debian2:/sys/bus/ccw/devices/0.0.0503# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0503# cat use_diag
     1
     debian2:/sys/bus/ccw/devices/0.0.0503# echo 0 >use_diag
     debian2:/sys/bus/ccw/devices/0.0.0503# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0503# cd ../0.0.0502
     debian2:/sys/bus/ccw/devices/0.0.0502# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0502# echo 0 >online
     debian2:/sys/bus/ccw/devices/0.0.0502# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0502# cat use_diag
     1
     debian2:/sys/bus/ccw/devices/0.0.0502# echo 0 >use_diag
     debian2:/sys/bus/ccw/devices/0.0.0502# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0502# cd ../0.0.0501
     debian2:/sys/bus/ccw/devices/0.0.0501# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0501# echo 0 >online
     debian2:/sys/bus/ccw/devices/0.0.0501# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0501# cd ../0.0.0500
     debian2:/sys/bus/ccw/devices/0.0.0500# cat online
     1
     debian2:/sys/bus/ccw/devices/0.0.0500# echo 0 >online
     debian2:/sys/bus/ccw/devices/0.0.0500# cat online
     0
     debian2:/sys/bus/ccw/devices/0.0.0500# cat use_diag
     1
     debian2:/sys/bus/ccw/devices/0.0.0500# echo 0 >use_diag
     debian2:/sys/bus/ccw/devices/0.0.0500# cat use_diag
     0
     debian2:/sys/bus/ccw/devices/0.0.0500# cd
     debian2:~# vmcp DETACH VIRTUAL 0500-0503
     0500-0503 DETACHED
     debian2:~#

The umount commands are essentially the reverse of the mount commands issued at the beginning.  Make sure you unmount everything you mounted. 

All dynamically linked minidisks have now been detached from DEBIAN2.  Now use your CP directory maintenance procedures on VM to shuffle disks around for the DEBIAN1 virtual machine.  Transfer the 200-series minidisks of DEBIAN1 to another virtual machine.  These are your old cdl minidisks.  You want to hang on to these as a backout, just in case the migration didn't work right.  Now rename the 400-series minidisks to the 200-series minidisks.  (400 becomes 200, 401 becomes 201, etc.)  These are your new CMS reserved minidisks.

Step 10

Now fire up DEBIAN1.  If you did everything right, it will come up using the new CMS reserved minidisks and the DIAG driver for all minidisks except 0201, which will be using the ECKD (or FBA) driver.  Login as root and issue the command "cat /proc/dasd/devices".  You should see output similar to the following:
 

     debian1:~# cat /proc/dasd/devices
     0.0.0200(DIAG) at ( 94:     0) is dasda       : active at blocksize: 4096, 360000 blocks, 1406 MB
     0.0.0201(ECKD) at ( 94:     4) is dasdb       : active at blocksize: 4096, 13500 blocks, 52 MB
     0.0.0202(DIAG) at ( 94:     8) is dasdc       : active at blocksize: 4096, 90000 blocks, 351 MB
     0.0.0203(DIAG) at ( 94:    12) is dasdd       : active at blocksize: 4096, 90000 blocks, 351 MB
     debian1:~#

Once you're sure everything is working properly, you can delete the old cdl minidisks that you transferred to another userid. 

Conclusion

Now you can use a similar procedure on DEBIAN1 to convert DEBIAN2's minidisks to CMS reserved minidisks and switch it to the DIAG driver.

Someday, maybe the Debian installer will support the use of the DIAG driver during installation.  In the meantime, these migration instructions will allow you to make full use of this high-performance driver on all of your Debian for s390x virtual servers under z/VM.

That's all folks!  Have fun!  If anyone has any suggestions, comments, advice, tips, or any other form of feedback, please drop me a line at zlinuxman (at) fastmail (dot) com.

Return to my homepage