♨️ 🕵🏽 ☪️ LXCおよびDockerコンテナ仮想化用のPacemakerベースのHA-Cluster ❔ 🤹 📮

この記事では、共有ストレージを使用したPacemaker、Corosync 2.x、およびCLVMに基づくアクティブ/アクティブクラスターのインストールと構成について説明します。このクラスターをLXCコンテナーとDockerコンテナーで動作するように適合させる方法を示します。クラスターを操作するためのコマンドについて説明します。そして、私が入ったレーキを思い出します。これは、次の詐欺師の運命を促進することを願っています。

サーバーのディストリビューションとして、CentOS 7 + epelとその中の現在のパッケージバージョンを使用します。 Pacemakerと連携するための主なツールは、PCS（pacemaker / corosync configuration system）です。

サーバーの準備

2つのノードの構成を使用しましたが、必要に応じてその数を増やすことができます。サーバーには、SASを介して接続された共有共有ストレージがあります。これが手元にない場合は、FCまたはiSCSIに接続されたストレージを使用できます。 2つのボリュームが必要です。1つは一般的なニーズ用で、もう1つはDocker用です。 1つのボリュームを2つのセクションに分割できます。

CentOS 7、epelリポジトリをインストールし、ネットワークを構成します。ネットワークインターフェイスにボンディングを使用し、SASにマルチパスを使用することが望ましいです。さまざまなVLANを使用するには、適切なブリッジbr0.VIDを構成し、LXCコンテナーをバインドします。詳細は説明しません-すべてが標準です。

LXCとDockerを使用するには、通常のfirewald を切断する必要があります。

# systemctl stop firewalld.service # systemctl disable firewalld.service # setenforce Permissive

また、セットアップと大胆な実験を容易にするために、selinuxをpermissiveモードにしました。後で、デバッグするときに、元に戻してください。

すべてのノードの/ etc / hostsに必要なアドレスと名前をすぐに追加します。

 #nodes, vlan 10 10.1.0.1 cluster-1 10.1.0.2 cluster-2 #nodes ipmi, vlan 314 10.1.15.1 ipmi-1 10.1.15.2 ipmi-2 #docker, vlan 12 10.1.2.10 docker 10.1.2.11 dregistry

動作させるには、STONITHメカニズム（「ヘッド内の他のノードを撮影する」）が必要になります。これにはipmiを使用します。 ipmitoolを使用して構成します。

 # ipmitool shell impitool> user set name 2 admin impitool> user set password 2 '  ' # <uid> < privilege level> <channel number>) impitool> user priv 2 4 1

adminユーザー（id = 2）を取得し、ネットワークインターフェイスに関連付けられたチャネル（channel = 1）で管理者権限（livel = 4）を付与します。

ipmiのネットワークを別のVLANに移動することが望ましいです。まず、それを分離できるようにします。次に、IPMI BMC（ベースボード管理コントローラー）がサーバーとネットワークインターフェースを共有する場合、接続の問題はありません。

 impitool> lan set 1 ipsrc static impitool> lan set 1 ipaddr 10.1.15.1 impitool> lan set 1 netmask 255.255.255.0 impitool> lan set 1 defgw ipaddr 10.1.15.254 impitool> lan set 1 vlan id 314 #  : impitool> lan set 1 access on impitool> lan set 1 auth ADMIN MD5 ipmitool> channel setaccess 1 2 callin=on ipmi=on link=on privilege=4

他のノードでも同様に、IPのみが異なります。

次のような接続を確認します。

 # ipmitool -I lan -U admin -P '  ' -H 10.1.15.1 bmc info

PacemakerとCLVMのインストールと基本セットアップ

Pacemakerが何であるかわからない場合は、まずそれについて読むことをお勧めします。ペースメーカーに関するロシア語の良い説明がここに書かれています。

すべてのノードで、epelリポジトリーからパッケージをインストールします。

 # yum install pacemaker pcs resource-agents fence-agents-all

すべてのノードで、haclusterクラスター管理者のパスワードを設定します。 PCSはこのユーザーの下で機能し、Webベースの管理インターフェイスも利用できます。

 echo CHANGEME | passwd --stdin hacluster

1つのノードでさらに操作が実行されます。

認証を構成します。

 # pcs cluster auth cluster-1 cluster-2 -u hacluster -p CHANGEME --force

2つのノードの「クラスター」クラスターを作成して起動します。

 # pcs cluster setup --force --name Cluster cluster-1 cluster-2 # pcs cluster start --all

結果を見てみましょう：

 # pcs cluster status Cluster Status: Last updated: Wed Jul 8 14:16:32 2015 Last change: Wed Jul 8 10:01:20 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 17 Resources configured (  ) PCSD Status: cluster-1: Online cluster-2: Online

注意点が1つあります。https_proxyが環境変数で指定されている場合、 pcsはノードのステータスについて嘘をつき、明らかにプロキシを使用しようとします。

起動時にpcsdデーモンを起動して登録します。

 # systemctl start pcsd # systemctl enable pcsd

その後、アドレス"https://ip_:2224"

クラスター管理Webインターフェースが利用可能になります

インターフェイスを使用すると、クラスターの状態を確認したり、パラメーターを追加または変更できます。些細なことですが、素晴らしい。

ノードは2つしかないため、クォーラムはありません。したがって、このポリシーを無効にする必要があります。

 # pcs property set no-quorum-policy=ignore

クラスターノードを自動起動するには、 ペースメーカーを起動に追加するだけです：

 # systemctl enable pacemaker

CLVMおよびGFS2が機能するには、DLM（Distributed Lock Manager）が必要です。スタンドアロンデーモンとしてのRHEL7（CentOS 7）のCLVMとDLMはどちらも存在せず、クラスターリソースです。同時に、DLONが動作するにはSTONITHが必要です。そうでない場合、対応するクラスターリソースは起動しません。カスタマイズ：

 # pcs property set stonith-enabled=true # pcs stonith create cluster-1.stonith fence_ipmilan ipaddr="ipmi-1" passwd="  ipmi" login="admin" action="reboot" method="cycle" pcmk_host_list=cluster-1 pcmk_host_check=static-list stonith-timeout=10s op monitor interval=10s # pcs stonith create cluster-2.stonith fence_ipmilan ipaddr="ipmi-2" passwd="  ipmi" login="admin" action="reboot" method="cycle" pcmk_host_list=cluster-2 pcmk_host_check=static-list stonith-timeout=10s op monitor interval=10s # pcs constraint location cluster-1.stonith avoids cluster-1=INFINITY # pcs constraint location cluster-2.stonith avoids cluster-2=INFINITY

なぜそうなのか、ここで詳しく説明しています。要するに、2つのstonithリソースを起動し、それぞれが独自のノードを担当し、それらが撃つ必要があるノードで作業することを禁止します。

追加のグローバルパラメータを設定します。

実験の場合、最初の失敗後にリソースの移行を構成すると便利です。

 # pcs resource defaults migration-threshold=1

障害の結果として別のノードに移行したリソースが戻らないように、ノードを復元した後、次を設定します。

 # pcs resource defaults resource-stickiness=100

「100」は、Pacemakerがリソースの動作を計算する基準となる特定の重みです。

大胆な実験の最中に、ノードが互いに撃たないように、リソース障害のポリシーを明示的に設定することをお勧めします。

 # pcs resource op defaults on-fail=restart

それ以外の場合、stonithは最も興味深い場所で動作し、デフォルトでは停止コマンドが失敗します。

各ノードにCLVMをインストールします。

 # yum install lvm2 lvm2-cluster

各ノードのクラスターで動作するようにLVMを構成します。

 # lvmconf --enable-cluster

クラスタ内でdlmおよびclvmdリソースを取得します。

 # pcs resource create dlm ocf:pacemaker:controld op monitor interval=30s on-fail=fence clone interleave=true ordered=true # pcs resource create clvmd ocf:heartbeat:clvm op monitor interval=30s on-fail=fence clone interleave=true ordered=true

これらはクラスターにとって重要なリソースであるため、障害が発生した場合は、stonith（ on-fail = fence ） ポリシーを明示的に設定します。リソースはクラスターのすべてのノードで実行されている必要があるため、クローンとして宣言されます。リソースを順番に起動し、並列ではありません（ ordered = true ）。リソースが他のクローンリソースに依存している場合、すべてのリソースサンプルがすべてのノードで実行されるのを待つのではなく、ローカルのサンプル（ interleave = true ）を使用します。最後の2つのパラメーターに注意してください。これらは、クラスター全体の動作に大きな影響を与える可能性があり、クローンリソースは引き続き使用できます。

リソースが起動される順序を設定します。clvmdは dlmの後にのみ起動されます。コマンドは、特定のノード上のリソースを示すサービス名* -cloneを既に使用しています。

 # pcs constraint order start dlm-clone then clvmd-clone # pcs constraint colocation add clvmd-clone with dlm-clone

また、 clvmd-cloneリソースとdlm-cloneを同じノードで実行する必要があります。 2つのノードの場合、これは不要なように見えますが、一般的な場合* *-リソースのクローンはノードの数より少ない場合があり、その場合、共同配置が重要になります。

リソースを作成するとすぐに、それらはすべてのノードで起動され、すべてが正常であれば、共有論理ボリュームとファイルシステムの作成を開始できます。 clvmdはメタデータの整合性を追跡し、すべてのノードに変更を通知するため、1つのノードで操作を実行します。

LVMで使用するためにセクションを初期化します。

 # pvcreate /dev/mapper/mpatha1 # pvcreate /dev/mapper/mpatha2

一般に、クラスター化されたLVMでの作業は、通常のLVMでの作業とほとんど変わりません。唯一の違いは、ボリュームグループ（VG）がクラスター化済みとしてマークされている場合、clvmdがそのメタデータを制御することです。ボリュームグループを作成します。

 # vgcreate --clustered y shared_vg /dev/mapper/mpatha1 # vgcreate --clustered y shared_vg-ex /dev/mapper/mpatha2

クラスターの一般的な構成が完了したら、リソースを追加します。 Pacemakerの観点からのリソースは、スクリプトで制御できるIPアドレスのサービス、プロセスです。リソーススクリプト自体はinitスクリプトに似ており、start、stop、monitorなどの一連の機能も実行します。従う基本原則は、1つのノードで実行されるリソースに必要なデータを、必要に応じてshared_vgグループの共有パーティションと任意のファイルシステムに配置することです。両方のノードで同時に必要なデータをGFS2に配置します。前者の場合、Pacemakerはデータの整合性を監視し、使用するファイルシステムを含む実行中のリソースの数と場所を制御します。 2番目のケースでは、GFS2の内部メカニズム。 shared_vg-exグループは、Dockerの論理パーティションの下に完全に付与されます。実際、Dockerは、単一ノードで排他モードでのみアクティブにできるシンプロビジョニングボリュームを作成します。また、このボリュームを別のグループに配置すると、さらに作業と構成を行うのに便利です。

クラスターでLXCを操作する

lxcパッケージに含まれているlxc- *ユーティリティを使用して作業します。私たちは置きます：

 # yum install lxc lxc-templates

将来のコンテナのデフォルトパラメータを設定します。

 # cat /etc/lxc/default.conf lxc.start.auto = 0 lxc.network.type = veth lxc.network.link = br0.10 lxc.network.flags = up # memory and swap lxc.cgroup.memory.limit_in_bytes = 256M lxc.cgroup.memory.memsw.limit_in_bytes = 256M

ネットワークタイプはvethになります-eth0はコンテナの内部にあり、コンテナの外部にはbr0.10ブリッジに接続されます。メモリからのみ使用する制限のうち、それらを示します。必要であれば、原則lxc.cgroup.state-object-name = valueに従って、カーネルでサポートされているものを登録できます。 lxc-cgroupを使用してオンザフライで変更することもできます。ファイルシステムでは、これらのパラメーターはパス/sys/fs/cgroup/TYPE/lxc/CT-NAME/object-name

沿って提示されます。制限に関する重要な点： memory.limit_in_bytesパラメーターは、 memory.memsw.limit_in_bytesの前に指定する必要があります。また、2番目のパラメーターはメモリとスワップの合計であり、最初のパラメーター以上でなければなりません。そうしないと、メモリ制限なしでマシンが起動します。

Pacemakerは、コンテナの配置、起動、停止を制御するため、コンテナの起動を自動的にオフにします。

各LXCコンテナは、shared_vgグループの論理ボリュームに存在します。デフォルトのVG名を設定します。

 # cat /etc/lxc/lxc.conf lxc.bdev.lvm.vg = shared_vg

このような配置により、クラスターの任意のノードでコンテナーを起動できます。コンテナ構成ファイルも共有する必要があるため、共通のファイルシステムを作成し、すべてのノードでその使用を構成します。

 # lvcreate -L 500M -n lxc_ct shared_vg # mkfs.gfs2 -p lock_dlm -j 2 -t Cluster:lxc_ct /dev/shared_vg/lxc_ct

ストレージが共有されているため、 lock_dlmロックプロトコルを選択します。ノードの数に応じて2つのログを開始します（ -j2 ）。ロックテーブルに名前を設定します。Clusterはクラスターの名前です。

 # pcs resource create fs-lxc_ct Filesystem fstype=gfs2 device=/dev/shared_vg/lxc_ct directory=/var/lib/lxc clone ordered=true interleave=true # pcs constraint order start clvmd-clone then fs-lxc_ct-clone

Filesystemなどの次のclone-resourceを開始します。デバイスとディレクトリのフィールドは必須で、マウントする場所と場所を記述しています。また、clvmdがないとファイルシステムがマウントされないため、リソースの起動順序を示します。その後、すべてのノードで、LXCがコンテナ設定を保存するディレクトリがマウントされます。

最初のコンテナーを作成します。

 # lxc-create -n lxc-racktables -t oracle -B lvm --fssize 2G --fstype ext4 --vgname shared_vg -- -R 6.6

ここで、 lxc-racktablesはコンテナの名前、 oracleは使用されるテンプレートです。 -Bは、使用されるストレージのタイプとオプションを決定します。 lxc-createは、テンプレートに従ってLVMパーティションを作成し、そこにベースシステムを展開します。「-」の後に、テンプレートのパラメーターが表示されます。私の場合はバージョンです。

この記事の執筆時点では、lvmのcentosパッケージのテンプレートは機能しませんでしたが、oracleも機能していました。

debパッケージに基づいてシステムを展開する必要がある場合は、最初にdebootstrapユーティリティをインストールする必要があります。準備されたシステムは最初に/var/cache/lxc/

デプロイされ、その後の起動ごとにlxc-createはシステムパッケージを現在のバージョンに更新します。必要なすべてのプリセットを使用して、独自のテンプレートを作成すると便利です。標準テンプレートは、 /usr/share/lxc/templates

ます。

特別な「 ダウンロード 」テンプレートを使用することもできます。このテンプレートは、準備済みのシステムアーカイブをリポジトリからダウンロードします。

コンテナの準備ができました。 lxc- *ユーティリティを使用してコンテナを管理できます。バックグラウンドで実行し、その状態を見て、停止します。

 # lxc-start -n lxc-racktables -d # lxc-info -n lxc-racktables Name: lxc-racktables State: RUNNING PID: 9364 CPU use: 0.04 seconds BlkIO use: 0 bytes Memory use: 1.19 MiB KMem use: 0 bytes Link: vethS7U8J1 TX bytes: 90 bytes RX bytes: 90 bytes Total bytes: 180 bytes # lxc-stop -n lxc-racktables

コンソールでlxc-consoleを使用するか、コンテナのlvmセクションをどこかにマウントすることにより、追加のコンテナパラメータを設定します。

これで、Pacemakerに制御を渡すことができます。しかし、最初に、 GitHubから新しいリソース管理ファイルを取得します。

 # wget -O /usr/lib/ocf/resource.d/heartbeat/lxc https://raw.githubusercontent.com/ClusterLabs/resource-agents/master/heartbeat/lxc # chmod +x /usr/lib/ocf/resource.d/heartbeat/lxc

/usr/lib/ocf/resource.d/

ディレクトリには、プロバイダ/タイプ階層のリソース管理ファイルが含まれています。 pcs resource list

コマンドを使用して、 pcs resource list

全体を表示できます。特定のリソースの説明を表示pcs resource describe <standard:provider:type|type>

例：

  # pcs resource describe ocf:heartbeat:lxc ocf:heartbeat:lxc - Manages LXC containers Allows LXC containers to be managed by the cluster. If the container is running "init" it will also perform an orderly shutdown. It is 'assumed' that the 'init' system will do an orderly shudown if presented with a 'kill -PWR' signal. On a 'sysvinit' this would require the container to have an inittab file containing "p0::powerfail:/sbin/init 0" I have absolutly no idea how this is done with 'upstart' or 'systemd', YMMV if your container is using one of them. Resource options: container (required): The unique name for this 'Container Instance' eg 'test1'. config (required): Absolute path to the file holding the specific configuration for this container eg '/etc/lxc/test1/config'. log: Absolute path to the container log file use_screen: Provides the option of capturing the 'root console' from the container and showing it on a separate screen. To see the screen output run 'screen -r {container name}' The default value is set to 'false', change to 'true' to activate this option

そこで、新しいリソースをクラスターに追加します。

 # pcs resource create lxc-racktables lxc container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config # pcs constraint order start fs-lxc_ct-clone then lxc-racktables

そして、開始順序を示します。

リソースはすぐに開始され、そのステータスはpcs status

コマンドで確認できます。起動に失敗した場合、考えられる理由があります。 pcs resource debug-start <resource id>

コマンドを使用すると、画面に結果が表示された状態でリソースを開始できます。

 # pcs resource debug-start lxc-racktables Operation start for lxc-racktables (ocf:heartbeat:lxc) returned 0 > stderr: DEBUG: State of lxc-racktables: State: STOPPED > stderr: INFO: Starting lxc-racktables > stderr: DEBUG: State of lxc-racktables: State: RUNNING > stderr: DEBUG: lxc-racktables start : 0

ただし、注意が必要です。リソース割り当てのクラスター設定は無視され、現在のノードで実行されます。また、リソースがすでに別のノードで実行されている場合は、驚くかもしれません。 "--full"

は、多くの追加情報を提供します。

Pacemakerコンテナを管理しますが、もちろん、それが現在動作しているノード上でのみPacemakerに注目して、すべてのlxc- *ユーティリティを使用して作業できます。

結果のコンテナリソースは、以下を実行することで別のノードに転送できます。

 # pcs resource move <resource id> [destination node]

この場合、コンテナは1つのノードで正しくシャットダウンし、別のノードで起動します。

残念ながら、LXCにはまともなライブマイグレーションツールはありませんが、その場合、マイグレーションを構成することも可能になります。これを行うには、GFS2に別の共通パーティションを作成し、ダンプを配置して、lxcリソーススクリプトを変更し、関数migrate_toおよびmigrate_fromを実行する必要があります。

CRIUプロジェクトを見ましたが、CentOS 7で作業できませんでした。

OpenVZコンテナーをLXCに転送します

新しい論理パーティションを作成し、そこでOpenVZコンテナー（オフになっている）からデータを転送します。

 # lvcreate -L 2G -n lxc-openvz shared_vg # mkfs.ext4 /dev/shared_vg/lxc-openvz # mount /dev/shared_vg/lxc-openvz /mnt/lxc-openvz # rsync -avh --numeric-ids -e 'ssh' openvz:/vz/private/<containerid>/ /mnt/lxc-openvz/

新しいコンテナ用の設定ファイルを作成し、lxc-racktablesの内容をコピーして変更します。

 # mkdir /var/lib/lxc/lxc-openvz # cp /var/lib/lxc/lxc-racktables/config /var/lib/lxc/lxc-openvz/

構成ファイルで、フィールドを変更する必要があります。

 lxc.rootfs = /dev/shared_vg/lxc-openvz lxc.utsname = openvz #lxc.network.hwaddr

必要に応じて、制限と目的のネットワークブリッジを構成します。また、コンテナでは、 eth0インターフェイスでネットワーク設定を上書きして変更し、 etc/sysconfig/network

ファイルを修正する必要があります。

原則として、この後、コンテナはすでに起動できますが、LXCとの互換性を高めるには、コンテンツを完成させる必要があります。例として、テンプレートを使用してcentosコンテナー（ /usr/share/lxc/templates/lxc-centos

）、つまりconfigure_centos

およびconfigure_centos_init

関数の内容を少し改良して作成しました。スクリプトetc/init/power-status-changed.conf

作成に特に注意してくださいetc/init/power-status-changed.conf

、それなしでは、コンテナは正しくシャットダウンできません。または、コンテナのinittabに "p0::powerfail:/sbin/init 0"

という形式のルールを含める必要があります（ディストリビューションに応じて）。

/etc/init/power-status-changed.conf

＃power-status-changed-SIGPWRのシャットダウン

＃

電源状態が変化したときに開始

exec / sbin / shutdown -h now“ SIGPWR received”

誰かが怠yourselfすぎて自分で理解できない場合（でも無駄に）、あなたは（自分の危険とリスクで）私を使うことができます。コンテナのMACアドレスは、構成ファイルで固定するのが最適です。

lxc-console

れたコンテナにはコンソールに問題がある可能性がありますlxc-console

を使用して取得することはできません。スクリプトを使用して、 agetty

（代替Linux getty）を使用してこの問題を解決しますagetty

はポータブルコンテナーに入っています。そして、フォームのプロセスを開始するinit

設定：

 /sbin/agetty -8 38400 /dev/console /sbin/agetty -8 38400 /dev/tty1

レシピとスクリプト/etc/init/

は、作成されたクリーンコンテナから借用され、 agetty

変換されagetty

。

/etc/init/start-ttys.conf

＃

＃このサービスは、設定された数のgettyを開始します。

＃

＃このファイルを直接編集しないでください。動作を変更する場合は、

＃ファイルstart-ttys.overrideを作成し、そこに変更を加えてください。

停止rc RUNLEVEL = [2345]で開始

env ACTIVE_CONSOLES = / dev / tty [1-6]

env X_TTY = / dev / tty1

仕事

台本

。 / etc / sysconfig / init

tty for $（echo $ ACTIVE_CONSOLES）; する

["$ RUNLEVEL" = "5" -a "$ tty" = "$ X_TTY"] &&続行

initctl start tty TTY = $ tty

やった

終了スクリプト

/etc/init/console.conf

＃コンソール-getty

＃

＃このサービスは、システムが存在する時点からコンソールでgettyを維持します

＃再びシャットダウンされるまで開始されます。

停止rc RUNLEVEL = [2345]で開始

ランレベルで停止[！2345]

envコンテナー

復活する

#exec / sbin / mingetty --nohangup --noclear / dev / console

exec / sbin / agetty -8 38400 / dev / console

/etc/init/tty.conf

＃tty-ゲッティ

＃

＃このサービスは、指定されたデバイスでgettyを維持します。

＃

＃このファイルを直接編集しないでください。動作を変更する場合は、

＃tty.overrideファイルを作成し、そこに変更を加えてください。

ランレベルで停止[S016]

復活する

インスタンス$ TTY

#exec / sbin / mingetty --nohangup $ TTY

exec / sbin / agetty -8 38,400 $ TTY

使用法 'tty TTY = / dev / ttyX-XはコンソールID'

CentOS 6.6でmingetty

れたコンテナでmingetty

を使用しようとしmingetty

たが、ログのエラーで動作しmingetty

た：

 # /sbin/mingetty --nohangup /dev/console console: no controlling tty: Operation not permitted

lxc：///ドライバーを使用してlibvrt経由でLXCをlibvrt

できますが、この方法は危険であり、RedHatはディストリビューションからサポートを削除すると脅しています。

Pacemakerのlibvrt

を介した制御には、 ocf:heartbeat:VirtualDomain

スクリプトリソースがあり、ドライバーに応じて任意のVMを制御できます。機能には、KVMのライブマイグレーションが含まれます。 Pacemakerを使用してKVMを制御することも似ていると思いますが、必要ありませんでした。

クラスターでDockerを使用する

Dockerで動作するようにPacemakerをセットアップすることは、LXCをセットアップすることに似ていますが、設計の違いもあります。

まず、Dockerをインストールします。これはRHEL / CentOS 7ディストリビューションに含まれているため、問題はありません。

 # yum install docker

DockerにLVMの操作方法を教えます。これを行うには、次の内容のファイル/etc/sysconfig/docker-storage-setup

を作成します。

 VG=shared_vg-ex

Dockerがプールを作成するボリュームグループを指定する場所。ここで追加のパラメーターを設定できます（ man docker-storage-setup

）。

docker-storage-setup

実行しdocker-storage-setup

。

 # docker-storage-setup Rounding up size to full physical extent 716.00 MiB Logical volume "docker-poolmeta" created. Logical volume "docker-pool" created. WARNING: Converting logical volume shared_vg-ex/docker-pool and shared_vg-ex/docker-poolmeta to pools data and metadata volumes. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Converted shared_vg-ex/docker-pool to thin pool. Logical volume "docker-pool" changed. # lvs | grep docker-pool docker-pool shared_vg-ex twi-aot--- 17.98g 14.41 2.54

Dockerはシンプロビジョニングされたボリュームを使用します。これにより、クラスターでの使用に制限が課されます。このようなボリュームは、同時に複数のノードでアクティブにすることはできません。 shared_vg-exグループのボリュームが自動的にアクティブ化されないようにLVMを構成します。これを行うには、（すべてのノードで） /etc/lvm/lvm.conf

ファイルで自動的にアクティブ化されるグループ（またはボリュームを個別に）を明示的に指定する必要があります。

  auto_activation_volume_list = [ "shared_vg" ]

このボリュームをPacemakerで制御しましょう。

 # pcs resource create lvm-docker-pool LVM volgrpname=shared_vg-ex exclusive=yes # pcs constraint order start clvmd-clone then lvm-docker-pool # pcs constraint colocation add lvm-docker-pool with clvmd-clone

これで、 lvm-docker-poolリソースが実行されているノードでshared_vg-exグループのボリュームがアクティブになります。

Dockerは、外部ネットワークからのNATコンテナーに専用IPを使用します。設定で修正しましょう：

 # cat /etc/sysconfig/docker-network DOCKER_NETWORK_OPTIONS="--ip=10.1.2.10 —fixed-cidr=172.17.0.0/16"

個別のブリッジを設定せず、デフォルトでdocker0を使用し、コンテナのネットワークを修正します。コンテナの便利なネットワークを教えてみましたが、理解できないエラーが発生しました。 Googleは私が一人ではないことを示唆したので、Dockerが自分で選んだネットワークを修正するだけで満足しました。 Dockerは、ブリッジがオフになってもブリッジを停止せず、IPアドレスを削除しませんが、ブリッジは物理インターフェースに接続されていないため、これは問題ではありません。他の構成では、これを考慮する必要があります。

Dockerがプロキシサーバー経由でインターネットにアクセスできるように、環境変数を構成します。これを行うには、次の内容を含むディレクトリとファイル/etc/systemd/system/docker.service.d/http-proxy.conf

を作成します。

 [Service] Environment="http_proxy=http://ip_proxy:port" "https_proxy=http://ip_proxy:port" "NO_PROXY=localhost,127.0.0.0/8,dregistry"

基本的なセットアップが完了したら、適切なリソースでクラスターを満たします。一連のリソースがDockerを担当するため、それらをグループにまとめるのが便利です。グループのすべてのリソースは1つのノードで起動され、グループ内の順序に従って順番に起動されます。ただし、グループのリソースの1つに障害が発生した場合、グループ全体が別のノードに移行しようとしていることに留意する必要があります。また、グループのリソースをオフにすると、後続のすべてのリソースもオフになります。グループの最初のリソースは、LVMによって作成されたボリュームになります。

 # pcs resource group add docker lvm-docker-pool

リソースの作成-Dockerによって発行されたIPアドレス：

 # pcs resource create dockerIP IPaddr2 --group docker --after lvm-docker-pool ip=10.1.2.10 cidr_netmask=24 nic=br0.12

LVMボリュームに加えて、Dockerはファイルシステムを使用してデータを保存します。したがって、Pacemakerの制御下で別のセクションを作成する必要があります。このデータは作業中のDokcerのみが必要とするため、リソースは正常です。

 # lvcreate -L 500M -n docker-db shared_vg # mkfs.xfs /dev/shared_vg/docker-db # pcs resource create fs-docker-db Filesystem fstype=xfs device=/dev/shared_vg/docker-db directory=/var/lib/docker --group docker --after dockerIP

これで、Dockerデーモン自体を追加できます。

 # pcs resource create dockerd systemd:docker --group docker --after fs-docker-db

グループのリソースを正常に起動した後、Dockerが落ち着いたノードで、そのステータスを調べ、すべてが正常であることを確認します。

ドッカー情報：

 # docker info Containers: 5 Images: 42 Storage Driver: devicemapper Pool Name: shared_vg--ex-docker--pool Pool Blocksize: 524.3 kB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 2.781 GB Data Space Total: 19.3 GB Data Space Available: 16.52 GB Metadata Space Used: 852 kB Metadata Space Total: 33.55 MB Metadata Space Available: 32.7 MB Udev Sync Supported: true Library Version: 1.02.93-RHEL7 (2015-01-28) Execution Driver: native-0.2 Kernel Version: 3.10.0-229.7.2.el7.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 4 Total Memory: 3.703 GiB Name: cluster-2

すでにDockerを通常の方法で使用できます。しかし、完全を期すために、Dockerレジストリクラスターを取得して配置しましょう。レジストリの場合、個別のIPと名前10.1.2.11（dregistry）を使用し、画像のファイルストレージは個別のセクションに配置されます。

 # lvcreate -L 10G -n docker-registry shared_vg # mkfs.ext4 /dev/shared_vg/docker-registry # mkdir /mnt/docker-registry # pcs resource create docker-registry Filesystem fstype=ext4 device=/dev/shared_vg/docker-registry directory=/mnt/docker-registry --group=docker –after=dockerd # pcs resource create registryIP IPaddr2 --group docker --after docker-registry ip=10.1.2.11 cidr_netmask=24 nic=br0.12

Dockerが実行されているノードでコンテナーcontainerを作成します。

 # docker create -p 10.1.2.11:80:5000 -e REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry -v /mnt/docker-registry:/var/lib/registry -h dregistry --name=dregistry registry:2

コンテナへのポート転送を設定します（10.1.2.11:80→5000）。プラグ可能なディレクトリ/mnt/docker-registry

。ホスト名とコンテナ名。

出力docker ps -a

には、作成されたコンテナが表示され、実行準備が整います。

Pacemakerに制御を渡します。開始するには、新しいリソーススクリプトをダウンロードします。

 # wget -O /usr/lib/ocf/resource.d/heartbeat/docker https://raw.githubusercontent.com/ClusterLabs/resource-agents/master/heartbeat/docker # chmod +x /usr/lib/ocf/resource.d/heartbeat/docker

すべてのノードでリソーススクリプトのIDを監視することが重要です。そうしないと、驚きが生じる可能性があります。 Dockerリソーススクリプト自体は、レジストリからダウンロードすることにより、指定されたパラメーターで必要なコンテナーを作成できます。したがって、共通のレジストリと個人リポジトリを持つクラスターノードで常に実行されているDockerを使用できます。また、Pacemakerは個々のコンテナのみを制御できますが、これはそれほど面白くなく、冗長です。 1つのDockerで何をすべきかはまだわかりません。

そのため、完成したコンテナの制御をPacemakerに移します。

 # pcs resource create dregistry docker reuse=true image="docker.io/registry:2" --group docker --after registryIP

再使用= trueは重要なパラメーターです。そうでない場合、コンテナーは停止後に削除されます。imageレジストリとタグを含むコンテナの完全な座標を指定する必要があります。リソーススクリプトは、dregistryという名前の完成したコンテナを選択して実行します。

ローカルレジストリをクラスターノードのDocker構成に登録します（/etc/sysconfig/docker

）。

 ADD_REGISTRY='--add-registry dregistry' INSECURE_REGISTRY='--insecure-registry dregistry'

HTTPSは必要ないため、ローカルレジストリに対してはHTTPSをオフにします。

その後、systemctl restart docker

それpcs resource restart dockerd

が存在するノードまたはクラスター内の任意のノードでDockerサービスを再起動します。そして、10.1.2.11（レジストリ）の個人レジストリの機能を使用できます。

次に、Dockerコンテナーの例を使用して、Pacemakerでテンプレートを操作する方法を示します。残念ながら、ユーティリティ機能はpcs

非常に限られています。彼女はテンプレートの作成方法をまったく知らず、制約のためにいくつかの関連付けを作成できますが、テンプレートを使用して作業することはpcs

便利ではありません。幸いなことに、xmlファイルでクラスター構成を直接編集する機能が役立ちます。

 # pcs cluster cib > /tmp/cluster.xml # ,   # pcs cluster cib-push /tmp/cluster.xml

Dockerリソースは次の要件を満たしている必要があります。

Dockerグループリソースと同じノード上にある
すべてのdockerグループリソースの後に実行する
コンテナはお互いのステータスに依存しない

もちろん、これは各コンテナの依存関係をでハングさせることで実現できますpcs constraint

が、設定は煩雑で読みにくいです。

まず、Nginxでコンテナーを使用する3つの実験的なコンテナーを作成します。コンテナは事前にダウンロードされ、ローカルレジストリにアップロードされます。

 # docker pull nginx:latest # docker push nginx:latest # pcs resource create doc-test3 docker reuse=false image="dregistry/nginx:latest" --disabled # pcs resource create doc-test2 docker reuse=true image="dregistry/nginx:latest" --disabled # pcs resource create doc-test docker reuse=true image="dregistry/nginx:latest" --disabled

リソースはオフに作成されます。そうでない場合、リソースは起動を試みてから、ノードでどれだけ幸運になります。

新しく追加したxmlで、リソースのプールを追加します。コロケーションの定義（セクション

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>     ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>  ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

     # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) 
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> 
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> 
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config 
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id> 
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90        ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]    clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]   ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]    Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]    /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]  -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

     Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full 
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id> 
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

 ): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> 
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): 
      

        
        
        
      

     <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         ( colocation set , id="docker-col"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( score="INFINITY"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ).    ( id="docker-col-0"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )  : 
      

        
        
        
      

        ( sequential="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )        ( require-all="false"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     )   ( role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ) 
      

        
        
        
      

      resource_ref     ,      .  role="Started"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

         ( id="docker-col-1"
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ),      docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     . 
      

        
        
        
      

            role   ,      ( ). 
      

        
        
        
      

     
      

        
        
        
      

     Ordering set ,    : 
      

        
        
        
      

     <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      ,     docker
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     ,       . 
      

        
        
        
      

     
      

        
        
        
      

        ,    /  .       .             ,    - .   pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
         : 
      

        
        
        
      

      # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

           .    LXC  (  ): 
      

        
        
        
      

     <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          .        : 
      

        
        
        
      

     <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

             : 
      

        
        
        
      

     # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          pcs
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
          . 
      

        
        
        
      

          LXC  ,   Docker-. 
      

        
        
        
      

     
      

        
        
        
      

       Pacemaker-    crmsh
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       opensuse.org , , ,    . 
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs status Cluster name: Cluster 
      

        
        
        
      

     Last updated: Thu Jul 16 12:29:33 2015 
      

        
        
        
      

     Last change: Thu Jul 16 10:23:40 2015 
      

        
        
        
      

     Stack: corosync 
      

        
        
        
      

     Current DC: cluster-1 (1) - partition with quorum 
      

        
        
        
      

     Version: 1.1.12-a14efad 
      

        
        
        
      

     2 Nodes configured 
      

        
        
        
      

     19 Resources configured 
      

        
        
        
      

     
      

        
        
        
      

     Online: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     
      

        
        
        
      

     Full list of resources: 
      

        
        
        
      

     
      

        
        
        
      

     cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 
      

        
        
        
      

     cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 
      

        
        
        
      

     Clone Set: dlm-clone [dlm] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: clvmd-clone [clvmd] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     Clone Set: fs-lxc_ct-clone [fs-lxc_ct] 
      

        
        
        
      

     Started: [ cluster-1 cluster-2 ] 
      

        
        
        
      

     lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 
      

        
        
        
      

     Resource Group: docker 
      

        
        
        
      

     lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 
      

        
        
        
      

     dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     dockerd (systemd:docker): Started cluster-2 
      

        
        
        
      

     docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 
      

        
        
        
      

     registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 
      

        
        
        
      

     dregistry (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test2 (ocf::heartbeat:docker): Started cluster-2 
      

        
        
        
      

     doc-test3 (ocf::heartbeat:docker): Stopped 
      

        
        
        
      

     
      

        
        
        
      

     PCSD Status: 
      

        
        
        
      

     cluster-1: Online 
      

        
        
        
      

     cluster-2: Online 
      

        
        
        
      

     
      

        
        
        
      

     Daemon Status: 
      

        
        
        
      

     corosync: active/disabled 
      

        
        
        
      

     pacemaker: active/enabled 
      

        
        
        
      

     pcsd: active/enabled 
      

        
        
        
      

     # pcs constraint Location Constraints: 
      

        
        
        
      

     Resource: cluster-1.stonith 
      

        
        
        
      

     Disabled on: cluster-1 (score:-INFINITY) 
      

        
        
        
      

     Resource: cluster-2.stonith 
      

        
        
        
      

     Disabled on: cluster-2 (score:-INFINITY) 
      

        
        
        
      

     Resource: doc-test 
      

        
        
        
      

     Enabled on: cluster-2 (score:INFINITY) (role: Started) 
      

        
        
        
      

     Ordering Constraints: 
      

        
        
        
      

     start dlm-clone then start clvmd-clone (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) 
      

        
        
        
      

     start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) 
      

        
        
        
      

     start clvmd-clone then start lvm-docker-pool (kind:Mandatory) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false 
      

        
        
        
      

     Colocation Constraints: 
      

        
        
        
      

     clvmd-clone with dlm-clone (score:INFINITY) 
      

        
        
        
      

     fs-lxc_ct-clone with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     lvm-docker-pool with clvmd-clone (score:INFINITY) 
      

        
        
        
      

     Resource Sets: 
      

        
        
        
      

     set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY 
      

        
        
        
      

     
      

        
        
        
      

       . 
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

        : 
      

        
        
        
      

     # pcs resource show <resource id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

         , : 
      

        
        
        
      

     # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
            ,  ,   update  . 
      

        
        
        
      

     
      

        
        
        
      

       
      

        
        
        
      

     # pcs resource restart <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        clone-    ,  ,  . 
      

        
        
        
      

     
      

        
        
        
      

            : 
      

        
        
        
      

     # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
       ,        Pacemaker    -     . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs resource cleanup [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        Pacemaker    . 
      

        
        
        
      

     
      

        
        
        
      

     / - : 
      

        
        
        
      

     # pcs resource disable [<resource id>] # pcs resource enable [<resource id>]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        /  ,   . 
      

        
        
        
      

     
      

        
        
        
      

         : 
      

        
        
        
      

     # pcs resource move <resource id> [destination node]
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      -    ,             .      ,     ,    .  ,  constraint   : 
      

        
        
        
      

      Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     c   .       . 
      

        
        
        
      

     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs constraint –full
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

      -   id,  : 
      

        
        
        
      

     # pcs constraint remove <constraint id>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

          : 
      

        
        
        
      

     # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash">
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

     C 
      

        
        
        
      

     Pacemaker,  Clusters from Scratch Configuration Explained Pacemaker:    Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU
): <rsc_colocation id="docker-col" score="INFINITY"> <resource_set id="docker-col-0" require-all="false" role="Started" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> <resource_set id="docker-col-1"> <resource_ref id="docker"/> </resource_set> </rsc_colocation> ( colocation set , id="docker-col" ) ( score="INFINITY" ). ( id="docker-col-0" ) : ( sequential="false" ) ( require-all="false" ) ( role="Started" ) resource_ref , . role="Started" . ( id="docker-col-1" ), docker . role , ( ). Ordering set , : <rsc_order id="order_doc"> <resource_set id="order_doc-0"> <resource_ref id="docker"/> </resource_set> <resource_set id="order_doc-1" require-all="false" sequential="false"> <resource_ref id="doc-test"/> <resource_ref id="doc-test2"/> <resource_ref id="doc-test3"/> </resource_set> </rsc_order> , docker , . , / . . , - . pcs : # pcs constraint --full | grep -i set Resource Sets: set docker (id:order_doc-0) set doc-test doc-test2 doc-test3 sequential=false require-all=false (id:order_doc-1) (id:order_doc) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false (id:docker-col-0) set docker (id:docker-col-1) setoptions score=INFINITY (id:docker-col) . LXC ( ): <template id="lxc-template" class="ocf" provider="heartbeat" type="lxc"> <meta_attributes id="lxc-template-meta_attributes"> <nvpair id="lxc-template-meta_attributes-allow-migrate" name="use_screen" value="false"/> </meta_attributes> <operations> <op id="lxc-template-monitor-30s" interval="30s" name="monitor" timeout="20s"/> <op id="lxc-template-start-0" interval="0" name="start" timeout="20s"/> <op id="lxc-template-stop-0" interval="0" name="start" timeout="90s"/> </operations> </template> . : <primitive id="lxc-racktables" template="lxc-template"> <instance_attributes id="lxc-racktables-instance_attributes"> <nvpair id="lxc-racktables-instance_attributes-container" name="container" value="lxc-racktables"/> <nvpair id="lxc-racktables-instance_attributes-config" name="config" value="/var/lib/lxc/lxc-racktables/config"/> </instance_attributes> </primitive> : # pcs resource show lxc-racktables Resource: lxc-racktables (template=lxc-template) Attributes: container=lxc-racktables config=/var/lib/lxc/lxc-racktables/config pcs . LXC , Docker-. Pacemaker- crmsh opensuse.org , , , . : # pcs status Cluster name: Cluster Last updated: Thu Jul 16 12:29:33 2015 Last change: Thu Jul 16 10:23:40 2015 Stack: corosync Current DC: cluster-1 (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 19 Resources configured Online: [ cluster-1 cluster-2 ] Full list of resources: cluster-1.stonith (stonith:fence_ipmilan): Started cluster-2 cluster-2.stonith (stonith:fence_ipmilan): Started cluster-1 Clone Set: dlm-clone [dlm] Started: [ cluster-1 cluster-2 ] Clone Set: clvmd-clone [clvmd] Started: [ cluster-1 cluster-2 ] Clone Set: fs-lxc_ct-clone [fs-lxc_ct] Started: [ cluster-1 cluster-2 ] lxc-racktables (ocf::heartbeat:lxc): Started cluster-1 Resource Group: docker lvm-docker-pool (ocf::heartbeat:LVM): Started cluster-2 dockerIP (ocf::heartbeat:IPaddr2): Started cluster-2 fs-docker-db (ocf::heartbeat:Filesystem): Started cluster-2 dockerd (systemd:docker): Started cluster-2 docker-registry (ocf::heartbeat:Filesystem): Started cluster-2 registryIP (ocf::heartbeat:IPaddr2): Started cluster-2 dregistry (ocf::heartbeat:docker): Started cluster-2 doc-test (ocf::heartbeat:docker): Started cluster-2 doc-test2 (ocf::heartbeat:docker): Started cluster-2 doc-test3 (ocf::heartbeat:docker): Stopped PCSD Status: cluster-1: Online cluster-2: Online Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled # pcs constraint Location Constraints: Resource: cluster-1.stonith Disabled on: cluster-1 (score:-INFINITY) Resource: cluster-2.stonith Disabled on: cluster-2 (score:-INFINITY) Resource: doc-test Enabled on: cluster-2 (score:INFINITY) (role: Started) Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) start clvmd-clone then start fs-lxc_ct-clone (kind:Mandatory) start fs-lxc_ct-clone then start lxc-racktables (kind:Mandatory) start clvmd-clone then start lvm-docker-pool (kind:Mandatory) Resource Sets: set docker set doc-test doc-test2 doc-test3 sequential=false require-all=false Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) fs-lxc_ct-clone with clvmd-clone (score:INFINITY) lvm-docker-pool with clvmd-clone (score:INFINITY) Resource Sets: set doc-test doc-test2 doc-test3 role=Started sequential=false require-all=false set docker setoptions score=INFINITY . : # pcs resource show <resource id> , : # pcs resource update <resource id> op start start-delay="3s" interval=0s timeout=90 , , update . # pcs resource restart <resource id> [node] clone- , , . : # pcs resource failcount show <resource id> [node] # pcs resource failcount reset <resource id> [node] , Pacemaker - . : # pcs resource cleanup [<resource id>] Pacemaker . / - : # pcs resource disable [<resource id>] # pcs resource enable [<resource id>] / , . : # pcs resource move <resource id> [destination node] - , . , , . , constraint : Resource: docker Enabled on: cluster-2 (score:INFINITY) (role: Started) (id:cli-prefer-docker) c . . : # pcs constraint –full - id, : # pcs constraint remove <constraint id> : # pcs cluster cib > /tmp/cluster.xml # pcs cluster cib-push /tmp/cluster.xml<source lang="bash"> C Pacemaker, Clusters from Scratch Configuration Explained Pacemaker: Creating LVM Volumes in a Cluster Linux Containers with libvirt-lxc (deprecated) High Availability Add-On Reference High Availability Add-On Administration Docker Docs Get Started with Docker Formatted Container Images on Red Hat Systems LXC 1.0. GitHub ClusterLabs CRIU

LXCおよびDockerコンテナ仮想化用のPacemakerベースのHA-Cluster

サーバーの準備

PacemakerとCLVMのインストールと基本セットアップ

クラスターでLXCを操作する

OpenVZコンテナーをLXCに転送します

クラスターでDockerを使用する

More articles: