节点对应的微服务无法重启

#1

目前运行了一个cita测试链，给的数据盘为100G，前阵子测试比较多，交易量多，然后发现因为数据量太多，磁盘满了导致节点程序停止。今天扩展了对应的磁盘后重启节点，仍然有问题：

具体表现为 start 之后，只有cita-forever 一个微服务，其他的对应的log里面有报错：

cita-auth.log

                        at src/libpanic_unwind/lib.rs:80
  13:     0x55dcec6d2e4c - std::panicking::try::hc3a9b5da4250385d
                        at src/libstd/panicking.rs:271
                         - std::panic::catch_unwind::hf27600bf8c37809a
                        at src/libstd/panic.rs:394
                         - std::rt::lang_start_internal::h409d4f2fe51133b0
                        at src/libstd/rt.rs:48
  14:     0x55dcec0e9b31 - main
  15:     0x7fe291679b96 - __libc_start_main
  16:     0x55dcec0b0929 - _start
  17:                0x0 - <unknown>

position:
Thread main panicked at failed to open url amqp://guest:guest@localhost/taidi-chain/1 : IoError(Other), /opt/.cargo/git/checkouts/cita-common-1cdbc5a3f2d672a3/50b22af/pubsub/src/lib.rs:58

This is a bug. Please report it at:

    https://github.com/citahub/cita/issues/new?labels=bug&template=bug_report.md

============================

2020-07-13 - 15:11:11 | cita_auth            - 140   | INFO  - CITA:auth
2020-07-13 - 15:11:11 | cita_auth            - 141   | INFO  - Version: 20.2.0-d8f65876
2020-07-13 - 15:11:11 | util::panic_hook     - 55    | ERROR -
============================
stack backtrace:
   0:     0x55c3866ca48d - backtrace::backtrace::trace::h1d058e17b1a6c38c
   1:     0x55c3866c9772 - <backtrace::capture::Backtrace as core::default::Default>::default::h2e5beecc95b1ca84
   2:     0x55c3866c8a14 - util::panic_hook::panic_hook::h5d3c8afd5da189bd
   3:     0x55c3866c8758 - core::ops::function::Fn::call::h8b252a0d2ec7c0c5
   4:     0x55c3868c297b - std::panicking::rust_panic_with_hook::h825f041245da8739
                        at src/libstd/panicking.rs:477
   5:     0x55c3868c2431 - std::panicking::continue_panic_fmt::hbe0378e33481e81b
                        at src/libstd/panicking.rs:380
   6:     0x55c3868c238b - std::panicking::begin_panic_fmt::hecb5f06c798651bd
                        at src/libstd/panicking.rs:335
   7:     0x55c3865af7b3 - pubsub::start_rabbitmq::h5dfb67ec25dba453
   8:     0x55c3862ae1ad - pubsub::start_pubsub::h8d03c0f93b076197
   9:     0x55c3862d966d - cita_auth::main::hf37da479a1c7209a
  10:     0x55c3862b4662 - std::rt::lang_start::{{closure}}::h2824d8927ce20bb9
  11:     0x55c3868c22c2 - std::rt::lang_start_internal::{{closure}}::ha04574f12d97cbe2
                        at src/libstd/rt.rs:49
                         - std::panicking::try::do_call::h7c2a8488f72db90c
                        at src/libstd/panicking.rs:292
  12:     0x55c3868c6ce9 - __rust_maybe_catch_panic
                        at src/libpanic_unwind/lib.rs:80
  13:     0x55c3868c2e4c - std::panicking::try::hc3a9b5da4250385d
                        at src/libstd/panicking.rs:271
                         - std::panic::catch_unwind::hf27600bf8c37809a
                        at src/libstd/panic.rs:394
                         - std::rt::lang_start_internal::h409d4f2fe51133b0
                        at src/libstd/rt.rs:48
  14:     0x55c3862d9b31 - main
  15:     0x7febf00ccb96 - __libc_start_main
  16:     0x55c3862a0929 - _start
  17:                0x0 - <unknown>

position:
Thread main panicked at failed to open url amqp://guest:guest@localhost/taidi-chain/1 : IoError(Other), /opt/.cargo/git/checkouts/cita-common-1cdbc5a3f2d672a3/50b22af/pubsub/src/lib.rs:58

This is a bug. Please report it at:

    https://github.com/citahub/cita/issues/new?labels=bug&template=bug_report.md

============================

cita-bft.log

============================

2020-07-13 - 15:11:11 | cita_bft             - 107   | INFO  - CITA:consensus:cita-bft
2020-07-13 - 15:11:11 | cita_bft             - 108   | INFO  - Version: 20.2.0-e5ea2911
2020-07-13 - 15:11:11 | util::panic_hook     - 55    | ERROR -
============================
stack backtrace:
   0:     0x56219f91e6ad - backtrace::backtrace::trace::h1d058e17b1a6c38c
   1:     0x56219f91d992 - backtrace::capture::Backtrace::new::h26cd411c0832f61b
   2:     0x56219f91cc34 - util::panic_hook::panic_hook::h5d3c8afd5da189bd
   3:     0x56219f91c978 - core::ops::function::Fn::call::h8b252a0d2ec7c0c5
   4:     0x56219fb0d12b - std::panicking::rust_panic_with_hook::h825f041245da8739
                        at src/libstd/panicking.rs:477
   5:     0x56219fb0cbe1 - std::panicking::continue_panic_fmt::hbe0378e33481e81b
                        at src/libstd/panicking.rs:380
   6:     0x56219fb0cb3b - std::panicking::begin_panic_fmt::hecb5f06c798651bd
                        at src/libstd/panicking.rs:335
   7:     0x56219f806423 - pubsub::start_rabbitmq::h5dfb67ec25dba453
   8:     0x56219f7de1ea - cita_bft::main::h5c4221faef63a567
   9:     0x56219f7e5792 - std::rt::lang_start::{{closure}}::h9f7482ec451de627
  10:     0x56219fb0ca72 - std::rt::lang_start_internal::{{closure}}::ha04574f12d97cbe2
                        at src/libstd/rt.rs:49
                         - std::panicking::try::do_call::h7c2a8488f72db90c
                        at src/libstd/panicking.rs:292
  11:     0x56219fb11479 - __rust_maybe_catch_panic
                        at src/libpanic_unwind/lib.rs:80
  12:     0x56219fb0d53c - std::panicking::try::hc3a9b5da4250385d
                        at src/libstd/panicking.rs:271
                         - std::panic::catch_unwind::hf27600bf8c37809a
                        at src/libstd/panic.rs:394
                         - std::rt::lang_start_internal::h409d4f2fe51133b0
                        at src/libstd/rt.rs:48
  13:     0x56219f7df881 - main
  14:     0x7fdb86f5ab96 - __libc_start_main
  15:     0x56219f7a49e9 - _start
  16:                0x0 - <unknown>

position:
Thread main panicked at failed to open url amqp://guest:guest@localhost/taidi-chain/1 : IoError(Other), /opt/.cargo/git/checkouts/cita-common-1cdbc5a3f2d672a3/50b22af/pubsub/src/lib.rs:58

This is a bug. Please report it at:

    https://github.com/citahub/cita/issues/new?labels=bug&template=bug_report.md

============================
#2

因为最终停止运行是系统盘数据满了，因此我觉得区块链的数据（在数据盘）没有损害。所以我完全停止节点，./bin/cita stop test-chain/1，停止并删除docker容器（cita_runxxxxxxx)，然后重新setup，start节点，然后一切正常:

/data/cita# ./bin/cita top  test-chain/1
root      2159     1  0 15:39 ?        00:00:00 cita-forever
root      2173  2159  1 15:39 ?        00:00:00 cita-auth -c auth.toml
root      2174  2159  0 15:39 ?        00:00:00 cita-bft -c consensus.toml -p privkey
root      2166  2159 26 15:39 ?        00:00:02 cita-chain -c chain.toml
root      2171  2159 10 15:39 ?        00:00:00 cita-executor -c executor.toml
root      2170  2159  0 15:39 ?        00:00:00 cita-jsonrpc -c jsonrpc.toml
root      2172  2159  0 15:39 ?        00:00:00 cita-network -c network.toml
1赞
cita无法启动
#3

可以考虑下如何做存储归档 :smile:

#4

上面的报错是rabbitmq没有正常运行导致的。
应该是磁盘满导致rabbitmq也收到了影响。
你后面的做法是对的，面对磁盘满这种AOE攻击，还是全部重启一下比较好。