Skip to content
PiVirt

iscsiman — iSCSI & OCFS2 Cluster Storage Manager

iscsiman manages iSCSI target/initiator configuration and OCFS2 cluster filesystems for shared storage across PiVirt nodes.

Technical Specifications

Section titled “Technical Specifications”
Property Value
Language Go 1.22+
Protocol gRPC (internal), ConnectRPC (via agent)
Storage iSCSI targets, OCFS2 volumes
Cluster Pacemaker/Corosync integration
Port 9002 (gRPC), 3260 (iSCSI)
Config /etc/iscsiman/config.yaml

Architecture

Section titled “Architecture”
flowchart TB
    subgraph Iscsiman["iscsiman Architecture"]
        subgraph Server["gRPC Server (Port 9002)"]
            TGT_SVC["TargetSvc"]
            INI_SVC["InitiatorSvc"]
            CLS_SVC["ClusterSvc"]
        end

        subgraph Core["Core Engine"]
            TGT_MGR["Target Manager"]
            INI_MGR["Initiator Manager"]
            OCFS_MGR["OCFS2 Manager"]
            PACE["Pacemaker Integration"]
        end

        Server --> Core
    end

Key Features

Section titled “Key Features”

iSCSI Target Management

Section titled “iSCSI Target Management”
  • Create/Delete targets: IQN-based target creation
  • LUN mapping: Map block devices to LUNs
  • CHAP authentication: Mutual CHAP for initiator auth
  • ALUA support: Asymmetric Logical Unit Access
  • Persistent reservations: SCSI PR for cluster fencing

iSCSI Initiator Management

Section titled “iSCSI Initiator Management”
  • Discovery: SendTargets, iSNS, static
  • Login/Logout: Session management
  • Multipath: DM-Multipath integration
  • Auto-connect: Persistent sessions across reboots

OCFS2 Cluster Filesystem

Section titled “OCFS2 Cluster Filesystem”
  • Volume creation: Format OCFS2 on shared block devices
  • Mount management: Cluster-aware mount/unmount
  • Node management: Add/remove nodes from cluster
  • Quorum: Configure quorum policies
  • Recovery: Journal replay, orphan dir recovery

Pacemaker/Corosync Integration

Section titled “Pacemaker/Corosync Integration”
  • Resource agents: iSCSI target, OCFS2 mount agents
  • Constraints: Colocation, ordering, location
  • Fencing: SBD, IPMI, fence-agents
  • Health monitoring: Resource health checks

Configuration

Section titled “Configuration”

Main Config (/etc/iscsiman/config.yaml)

Section titled “Main Config (/etc/iscsiman/config.yaml)”
# iscsiman configuration
server:
  grpc_addr: "0.0.0.0:9002"
  http_addr: "0.0.0.0:9003"  # Metrics, health
  tls:
    cert_file: "/etc/iscsiman/tls/server.crt"
    key_file: "/etc/iscsiman/tls/server.key"
    ca_file: "/etc/iscsiman/tls/ca.crt"


# iSCSI Target (tgt/targetcli)
target:
  backend: "tgt"  # tgt | targetcli | lio
  config_dir: "/etc/tgt/conf.d"
  default_iqn_base: "iqn.2024-01.hu.pilab"


  # CHAP authentication
  chap:
    enabled: true
    user: "pivirtd_initiator"
    password: "${ISCSI_CHAP_PASSWORD}"  # From env/secrets
    mutual_user: "pivirtd_target"
    mutual_password: "${ISCSI_MUTUAL_CHAP_PASSWORD}"


  # ALUA configuration
  alua:
    enabled: true
    default_tpg: 1


# iSCSI Initiator (open-iscsi)
initiator:
  config_dir: "/etc/iscsi"
  node_dir: "/etc/iscsi/nodes"
  iscsid_config: "/etc/iscsi/iscsid.conf"


  # Discovery
  discovery:
    sendtargets:
      - portal: "10.0.1.10:3260"
        iqn: "iqn.2024-01.hu.pilab:storage"
    static:
      - portal: "10.0.1.11:3260"
        iqn: "iqn.2024-01.hu.pilab:backup"


  # Multipath
  multipath:
    enabled: true
    config: "/etc/multipath.conf"
    policy: "queue_if_no_path"
    rr_min_io: 100


# OCFS2 Cluster Filesystem
ocfs2:
  # Cluster configuration
  cluster:
    name: "pivirt-cluster"
    config_file: "/etc/ocfs2/cluster.conf"
    heartbeat_device: "/dev/disk/by-id/scsi-360050764008181c60000000000000001p1"


  # Default mount options
  mount_options: "rw,noatime,data=ordered,errors=remount-ro"


  # Volume defaults
  defaults:
    block_size: "4K"
    cluster_size: "4K"
    journal_size: "256M"
    label_prefix: "pivirt-"


# Pacemaker/Corosync Integration
cluster:
  enabled: true
  crm_config: "/etc/corosync/corosync.conf"
  pacemaker_config: "/etc/pacemaker/cib.xml"


  # Resource defaults
  resource_defaults:
    resource_stickiness: 100
    migration_threshold: 3


  # Fencing
  fencing:
    enabled: true
    agent: "fence_sbd"
    devices:
      - name: "sbd-device"
        params:
          device: "/dev/disk/by-id/scsi-360050764008181c60000000000000002"


# Logging
logging:
  level: "info"
  format: "json"
  output: "journald"


# Metrics
metrics:
  enabled: true
  path: "/metrics"

API Reference

Section titled “API Reference”

Target Service

Section titled “Target Service”
service TargetService {
  // Create iSCSI target
  rpc CreateTarget(CreateTargetRequest) returns (Target);


  // Get target by IQN
  rpc GetTarget(GetTargetRequest) returns (Target);


  // List all targets
  rpc ListTargets(ListTargetsRequest) returns (stream Target);


  // Delete target
  rpc DeleteTarget(DeleteTargetRequest) returns (Empty);


  // Add LUN to target
  rpc AddLUN(AddLUNRequest) returns (LUN);


  // Remove LUN from target
  rpc RemoveLUN(RemoveLUNRequest) returns (Empty);


  // Set CHAP auth
  rpc SetCHAP(SetCHAPRequest) returns (Target);


  // Get target stats
  rpc GetTargetStats(GetTargetStatsRequest) returns (TargetStats);
}


message Target {
  string iqn = 1;
  string alias = 2;
  TargetState state = 3;
  repeated LUN luns = 4;
  CHAPConfig chap = 5;
  ALUAConfig alua = 6;
  google.protobuf.Timestamp created_at = 7;
}


message LUN {
  int32 lun = 1;
  string path = 2;           // Block device path
  string backing_file = 3;   // For file-backed
  int64 size_bytes = 4;
  bool readonly = 5;
  string scsi_id = 6;
  string scsi_serial = 7;
}

Initiator Service

Section titled “Initiator Service”
service InitiatorService {
  // Discover targets
  rpc Discover(DiscoverRequest) returns (stream DiscoveryResult);


  // Login to target
  rpc Login(LoginRequest) returns (Session);


  // Logout from target
  rpc Logout(LogoutRequest) returns (Empty);


  // List sessions
  rpc ListSessions(ListSessionsRequest) returns (stream Session);


  // Rescan sessions
  rpc Rescan(RescanRequest) returns (Empty);


  // Configure multipath
  rpc ConfigureMultipath(ConfigureMultipathRequest) returns (MultipathConfig);
}


message Session {
  string session_id = 1;
  string target_iqn = 2;
  string portal = 3;
  SessionState state = 4;
  repeated Connection connections = 5;
  MultipathInfo multipath = 6;
}

Cluster Service

Section titled “Cluster Service”
service ClusterService {
  // Create OCFS2 volume
  rpc CreateVolume(CreateVolumeRequest) returns (OCFS2Volume);


  // Get volume info
  rpc GetVolume(GetVolumeRequest) returns (OCFS2Volume);


  // List volumes
  rpc ListVolumes(ListVolumesRequest) returns (stream OCFS2Volume);


  // Mount volume
  rpc MountVolume(MountVolumeRequest) returns (MountInfo);


  // Unmount volume
  rpc UnmountVolume(UnmountVolumeRequest) returns (Empty);


  // Add node to cluster
  rpc AddNode(AddNodeRequest) returns (ClusterNode);


  // Remove node from cluster
  rpc RemoveNode(RemoveNodeRequest) returns (Empty);


  // Get cluster status
  rpc GetStatus(GetStatusRequest) returns (ClusterStatus);
}

Running iscsiman

Section titled “Running iscsiman”

Systemd Service

Section titled “Systemd Service”
/etc/systemd/system/iscsiman.service
[Unit]
Description=PiVirt iSCSI & Cluster Storage Manager
After=network.target tgt.service open-iscsi.service corosync.service pacemaker.service
Requires=tgt.service open-iscsi.service


[Service]
Type=notify
ExecStart=/usr/local/bin/iscsiman --config /etc/iscsiman/config.yaml
Restart=on-failure
RestartSec=5
LimitNOFILE=1048576
LimitMEMLOCK=infinity
CapabilityBoundingSet=CAP_SYS_ADMIN CAP_DAC_OVERRIDE CAP_NET_ADMIN CAP_SYS_RESOURCE


[Install]
WantedBy=multi-user.target

Prerequisites

Section titled “Prerequisites”
Terminal window
# Install iSCSI target (tgt)
# Ubuntu/Debian
apt-get install -y tgt


# RHEL/Fedora
dnf install -y targetcli


# Install iSCSI initiator
apt-get install -y open-iscsi multipath-tools


# Install OCFS2
apt-get install -y ocfs2-tools


# Install Pacemaker/Corosync
apt-get install -y pacemaker corosync fence-agents


# Configure kernel modules
modprobe iscsi_target
modprobe iscsi_tcp
modprobe ocfs2
modprobe ocfs2_dlmfs
modprobe ocfs2_nodemanager


# Enable services
systemctl enable --now tgt open-iscsi multipathd

Common Operations

Section titled “Common Operations”

For step-by-step operational workflows, see the iscsiman Guide.

Create iSCSI Target with LUNs

Section titled “Create iSCSI Target with LUNs”
Terminal window
# Using pvcli
pvcli storage target create \
  --iqn iqn.2024-01.hu.pilab:vm-storage \
  --alias "VM Storage Pool" \
  --pool iscsi-pool


# Add LUN
pvcli storage target lun add \
  --target iqn.2024-01.hu.pilab:vm-storage \
  --lun 1 \
  --device /dev/pivirtd-vg/vm-01-disk

Configure Initiator Connection

Section titled “Configure Initiator Connection”
Terminal window
# Discover targets
pvcli storage initiator discover --portal 10.0.1.10:3260


# Login
pvcli storage initiator login \
  --target iqn.2024-01.hu.pilab:vm-storage \
  --portal 10.0.1.10:3260


# Verify multipath
multipath -ll

Create OCFS2 Volume

Section titled “Create OCFS2 Volume”
Terminal window
# Create volume on shared block device
pvcli storage ocfs2 create \
  --name vm-shared \
  --device /dev/disk/by-id/scsi-360050764008181c60000000000000001 \
  --cluster pivirt-cluster


# Mount on all nodes
pvcli storage ocfs2 mount --name vm-shared --all-nodes

Troubleshooting

Section titled “Troubleshooting”

Target Not Accessible

Section titled “Target Not Accessible”
Terminal window
# Check target service
systemctl status tgt
tgt-admin --show


# Check CHAP
cat /etc/tgt/conf.d/*.conf | grep -i chap


# Check firewall
nft list ruleset | grep 3260

Initiator Connection Issues

Section titled “Initiator Connection Issues”
Terminal window
# Check iscsid
systemctl status open-iscsi


# Check sessions
iscsiadm -m session -P3


# Check multipath
multipath -ll


# Check iscsiadm discovery
iscsiadm -m discovery -t sendtargets -p 10.0.1.10

OCFS2 Mount Failures

Section titled “OCFS2 Mount Failures”
Terminal window
# Check cluster status
crm_mon -1


# Check OCFS2 cluster
mounted.ocfs2 -d


# Check DLM
ls /sys/kernel/config/dlm/cluster/


# Check heartbeat
cat /proc/ocfs2/cluster/

Pacemaker Issues

Section titled “Pacemaker Issues”
Terminal window
# Cluster status
crm_mon -1 -r


# Resource status
pcs status resources


# Debug resource
pcs resource debug-start <resource-name>


# Check constraints
pcs constraint show
Section titled “Related Documentation”

Storage Guide

Detailed configuration of directory, LVM, and iSCSI pools.
View Storage Guide

iSCSI Target Setup

Target creation, LUN mapping, and authentication.
View Target Guide

OCFS2 Clustered FS

Shared filesystem configuration with DLM and O2CB.
View OCFS2 Guide

Open: Community & Dev•Professional: Production Clustering•Enterprise: Mission-Critical & HA
© 2026 Progressive Innovation LAB. All rights reserved.Distributed virtualization platform licensed across Open, Professional, and Enterprise tiers.