Latest centos AMI stuck initializing on boot

[tarp (Tim Arp)]

Hi,
I'm testing out building my ECE cluster at AWS, I'm using the centos AMI from July 16th, ami-6b5a6314.
The EC2's are hung and I don't even get a network to login to. From the AWS system console is the following. The previous AMI, ami-26143a5c, works but with a limited number of instance types.

[e[32m  OK  e[0m] Started Network Manager Wait Online.

         Starting LSB: Bring up/down networking...

[e[1;31mFAILEDe[0m] Failed to start LSB: Bring up/down networking.

See 'systemctl status network.service' for details.

[e[32m  OK  e[0m] Reached target Network.

         Starting Postfix Mail Transport Agent...

         Starting /etc/rc.d/rc.local Compatibility...

         Starting Dynamic System Tuning Daemon...

         Starting Initial cloud-init job (metadata service crawler)...

[e[32m  OK  e[0m] Started /etc/rc.d/rc.local Compatibility.

[   12.088431] cloud-init[813]: Cloud-init v. 0.7.9 running 'init' at Wed, 25 Jul 2018 16:13:45 +0000. Up 12.05 seconds.

[   12.125395] cloud-init[813]: ci-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++

[   12.125761] cloud-init[813]: ci-info: +--------+------+-----------+-----------+-------+------------+

[   12.126020] cloud-init[813]: ci-info: | Device |  Up  |  Address  |    Mask   | Scope | Hw-Address |

[   12.126289] cloud-init[813]: ci-info: +--------+------+-----------+-----------+-------+------------+

[   12.126540] cloud-init[813]: ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |   .   |     .      |

[   12.126794] cloud-init[813]: ci-info: |  lo:   | True |     .     |     .     |   d   |     .      |

[   12.127035] cloud-init[813]: ci-info: +--------+------+-----------+-----------+-------+------------+

[   12.407118] cloud-init[813]: 2018-07-25 16:13:45,385 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', error(101, 'Network is unreachable'))]

[e[32m  OK  e[0m] Started Postfix Mail Transport Agent.

[e[32m  OK  e[0m] Started Dynamic System Tuning Daemon.

[   13.407268] cloud-init[813]: 2018-07-25 16:13:46,389 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: request error [('Connection aborted.', error(101, 'Network is unreachable'))]

[   14.410487] cloud-init[813]: 2018-07-25 16:13:47,392 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: request error [('Connection aborted.', error(101, 'Network is unreachable'))]

I can provide the full log, but I can't post it here to this discussion.

[Alex_Piggott]

Hi - sorry for the delay in responding to this - is it still an issue?

[tarp (Tim Arp)]

Yes, currently the previous AMI works fine but for limited EC2 types. I switch to the new one and it fails. I have a test build being done with Terraform to build up an environment, so I can easily reproduce this.
--Tim