iscsiman Architecture

iscsiman manages iSCSI targets/initiators and OCFS2 cluster filesystems for shared block storage across PiVirt nodes.

Component Architecture

flowchart TB
    subgraph iscsiman[iscsiman]
        direction TB
        subgraph grpc["gRPC Server (Port 9002)"]
            direction TB
            target["Target Svc (LIO/tgt)"]
            initiator["Initiator Svc (open-iscsi)"]
            filesystem["Filesystem Svc (OCFS2/DLM)"]
        end
        subgraph cluster[Cluster Manager]
            direction TB
            dlm[DLM Lock Manager]
            o2cb[O2CB Cluster]
            membership["Membership/Quorum (Corosync/Pacemaker)"]
        end
    end
    block["Block Devices (LVM/MD/Raw)"]
    network["iSCSI Network (3260/TCP)"]
    storage["Shared Storage (SAN/NAS)"]
    
    iscsiman --> block
    iscsiman --> network
    iscsiman --> storage

Data Flow

iSCSI Target Creation

sequenceDiagram
    participant Client
    participant gRPC
    participant TargetMgr
    participant LIO
    participant BlockDevice

    Client->>gRPC: CreateTarget(iqn, config)
    gRPC->>TargetMgr: Validate config
    TargetMgr->>LIO: Create target
    LIO->>BlockDevice: Map LUN
    BlockDevice-->>LIO: LUN ready
    LIO-->>TargetMgr: Target created
    TargetMgr-->>gRPC: Target object
    gRPC-->>Client: Success

OCFS2 Volume Mount

sequenceDiagram
    participant Client
    participant gRPC
    participant ClusterMgr
    participant O2CB
    participant OCFS2
    participant DLM

    Client->>gRPC: MountVolume(uuid, path)
    gRPC->>ClusterMgr: Check quorum
    ClusterMgr->>O2CB: Verify cluster
    O2CB-->>ClusterMgr: Cluster ready
    ClusterMgr->>DLM: Acquire locks
    DLM-->>ClusterMgr: Locks acquired
    ClusterMgr->>OCFS2: Mount filesystem
    OCFS2-->>ClusterMgr: Mounted
    ClusterMgr-->>gRPC: Mount info
    gRPC-->>Client: Success

iSCSI Target Architecture

LIO (Linux-IO) Target

flowchart TB
    subgraph lio[LIO Target]
        subgraph tpg["Target Portal Group (TPG)"]
            portal["Portal (0.0.0.0:3260)"]
            lun0["LUN 0 (root)"]
            lun1["LUN 1 (data)"]
        end
        subgraph backstores[Backstores]
            block["Block (LVM/MD)"]
            file["File (file)"]
            pscsi["PSCSI (passthru)"]
            rbd["RBD (Ceph)"]
        end
    end

Target Configuration

target:
  backend: "lio"
  iqn_base: "iqn.2024-01.hu.pilab:iscsiman"

  portal:
    address: "0.0.0.0"
    port: 3260

  auth:
    method: "chap"
    incoming_user: "iscsi-user"
    incoming_password_ref: "vault:secret/data/iscsi/chap#incoming"

  alua:
    enabled: true
    default_tpg: 1

iSCSI Initiator Architecture

Session Management

flowchart TB
    subgraph session[Initiator Session]
        subgraph connection[Connection]
            portal["Portal (10.0.1.10:3260)"]
            chap["CHAP (auth)"]
            tcp["TCP (socket)"]
        end
        subgraph multipath[Multipath]
            path1["Path 1 (active)"]
            path2["Path 2 (active)"]
            path3["Path 3 (standby)"]
        end
    end

Multipath Configuration

initiator:
  multipath:
    enabled: true
    config: "/etc/multipath.conf"
    default_policy: "service-time"
    path_checker: "tur"
    path_grouping_policy: "group_by_prio"

OCFS2 Cluster Architecture

O2CB Cluster Stack

flowchart TB
    subgraph o2cb[O2CB Cluster]
        subgraph config[Cluster Configuration]
            node0["Node 0 (10.0.1.10:7777)"]
            node1["Node 1 (10.0.1.11:7777)"]
            node2["Node 2 (10.0.1.12:7777)"]
        end
        subgraph heartbeat[Heartbeat]
            region["Region (disk)"]
            global["Global (network)"]
            local["Local (local)"]
        end
    end

DLM (Distributed Lock Manager)

flowchart TB
    subgraph dlm[DLM Lock Space]
        subgraph resources[Lock Resources]
            metadata[Metadata Lock]
            data[Data Lock]
            journal[Journal Lock]
        end
        subgraph modes[Lock Modes]
            nl["NL (null)"]
            cr["CR (concurrent)"]
            cw["CW (concurrent write)"]
            pr["PR (protected read)"]
        end
    end

OCFS2 Configuration

cluster:
  name: "pivirt-cluster"
  o2cb:
    stack: "o2cb"
    nodes:
      - name: "node-01"
        ip: "10.0.1.10"
        number: 0
      - name: "node-02"
        ip: "10.0.1.11"
        number: 1

  dlm:
    lockspace: "pivirt-dlm"
    recovery_timeout: "30s"

  fencing:
    enabled: true
    agent: "ipmi"
    device:
      ip: "10.0.0.11"
      user: "admin"
      password_ref: "vault:secret/data/bmc/node-01#password"
      lanplus: true
    action: "reboot"
    timeout: "60s"

Storage Pool Architecture

flowchart TB
    subgraph storage[Storage Pool Manager]
        subgraph types[Pool Types]
            lvm["LVM (local)"]
            iscsi["iSCSI (shared)"]
            nfs["NFS (shared)"]
            rbd["RBD (Ceph)"]
        end
        subgraph management[Volume Management]
            create[Create]
            clone[Clone]
            snapshot[Snapshot]
            resize[Resize]
        end
    end

Performance Considerations

iSCSI Performance

# Check iSCSI throughput
fio --name=iscsi-test --ioengine=libaio --iodepth=16 \
  --rw=randread --bs=4k --direct=1 --size=1G \
  --filename=/dev/disk/by-path/ip-10.0.1.10:3260-iscsi-iqn.2024-01.hu.pilab:iscsiman.target-01-lun-1

# Check multipath
multipath -ll

OCFS2 Performance

# Check OCFS2 performance
fio --name=ocfs2-test --ioengine=libaio --iodepth=16 \
  --rw=randread --bs=4k --direct=1 --size=1G \
  --filename=/mnt/ocfs2/test.img

# Check DLM locks
dlm_tool lockdebug pivirt-dlm

Monitoring

Metrics

# View iscsiman metrics
curl http://localhost:9003/metrics

# Key metrics:
# iscsiman_volume_size_bytes{volume="data-01"} 107374182400
# iscsiman_volume_used_bytes{volume="data-01"} 53687091200
# iscsiman_pool_capacity_bytes{pool="default"} 1099511627776
# iscsiman_replication_lag_seconds{volume="data-01"} 5

Health Checks

# Check iscsiman health
curl http://localhost:9003/health

# Check gRPC health
grpcurl -plaintext localhost:9002 grpc.health.v1.Health/Check

Troubleshooting

iSCSI Target Issues

# Check target status
targetcli ls

# Check LIO config
cat /sys/kernel/config/target/iscsi/iqn.2024-01.hu.pilab:iscsiman.target-01/tpgt_1/np/0.0.0.0:3260

# Check sessions
iscsiadm -m session -P 3

OCFS2 Issues

# Check cluster status
o2cb cluster-status
o2cb node-status

# Check DLM
dlm_tool ls

# Check filesystem
mount.ocfs2 -d /dev/mapper/shared-vm--shared--01 /mnt/point
mounted.ocfs2 -f /mnt/point

Related

• iscsiman Guide
• Storage Guide
• iSCSI Target Setup
• OCFS2 Cluster Setup